Message Brokers

1. Why

  • robustness and scalability require decoupled components
  • message brokers act as asynchronous intermediaries for interservice coms
  • instead of direct, synchronous calls, applications exchange messages through the broker
  • see Event Driven Programming

2. Benefits of Indirection

  • Decoupling: Producers and consumers operate independently, enabling asynchronous communication and enhancing fault tolerance.
  • Increased Performance & Responsiveness: Asynchronous messaging prevents producers from blocking, optimizing performance even under heavy load.
  • Simplified Development & Deployment: Location transparency abstracts service discovery and routing complexities.
  • Load Levelling: Message queues act as buffers, absorbing traffic spikes and preventing cascading failures.
  • Reliable Delivery: Brokers offer various message persistence and acknowledgement options, ensuring message delivery even in failure scenarios.

3. Characteristics

  • Message Ordering: Guaranteed within a partition/queue, not across multiple. Impacts scalability and consistency.
  • Delivery: "At-least-once" ensures delivery (potential duplicates). "Exactly-once" guarantees single processing (more complex). Choose based on use case needs.
  • Durability: In-memory queues offer speed but risk data loss. Disk-based persistence prioritizes reliability over speed.
  • Message Flow: Push-based brokers deliver messages proactively. Pull-based brokers require consumer action.
  • Routing: Content, topic, or header-based routing mechanisms allow for granular control over message delivery and consumption.

4. Tools

4.1. RabbitMQ

  • Mature and versatile message broker.
  • Robust routing capabilities with diverse exchange types and protocol support.
  • Prioritizes flexible message delivery over raw throughput.

4.2. Kafka

  • High-throughput data streaming for event-driven architectures and real-time analytics.
  • Guaranteed order within partitions and robust fault tolerance.
  • Ideal for mission-critical applications.

4.3. Apache Pulsar

  • Cloud-native platform combining queuing and streaming.
  • Features multi-tenancy, geo-replication, and flexible subscription semantics.
  • Versatile choice for distributed applications.
Tags::programming:tool:data: