message queue
Message Queues are asynchronous communication middleware that decouple producers (senders) from consumers (receivers) using persistent or transient queues — enabling parallel processing, load leveling, and fault tolerance by allowing components to operate at different speeds without blocking each other, forming the backbone of distributed system architectures.
Core Concepts
- Producer: Sends messages to the queue.
- Queue/Topic: Buffer that stores messages until consumed.
- Consumer: Reads and processes messages from the queue.
- Broker: Server that manages queues and routes messages.
Message Queue vs. Direct Communication
| Aspect | Direct (RPC/HTTP) | Message Queue |
|---|---|---|
| Coupling | Tight (caller waits) | Loose (fire and forget) |
| Failure handling | Caller must retry | Queue retains message |
| Speed mismatch | Caller blocked by slow receiver | Queue buffers overflow |
| Scalability | 1:1 or load balanced | 1:N fan-out, N:1 fan-in |
Popular Systems
| System | Type | Throughput | Latency | Persistence |
|---|---|---|---|---|
| Apache Kafka | Distributed log | Millions msg/sec | 2-10 ms | Persistent (disk) |
| RabbitMQ | Traditional broker | 100K msg/sec | < 1 ms | Optional |
| Redis Streams | In-memory log | Millions msg/sec | < 0.5 ms | AOF/RDB |
| Amazon SQS | Managed queue | Unlimited (scaled) | 1-10 ms | Persistent |
| ZeroMQ | Brokerless library | Millions msg/sec | < 0.1 ms | None |
| NATS | Cloud-native | Millions msg/sec | < 1 ms | JetStream |
Patterns for Parallel Processing
Work Queue (Competing Consumers)
- Multiple consumers pull from same queue → parallel processing.
- Load automatically balanced — faster consumers process more messages.
- Example: 100 image resize tasks queued → 10 workers process in parallel.
Fan-Out (Pub/Sub)
- Producer publishes to topic → all subscribers receive a copy.
- Example: New user signup → email service, analytics service, CRM all notified.
Request-Reply
- Producer sends request with reply-to queue → consumer sends result to reply queue.
- Enables async RPC with queue-based routing.
Delivery Guarantees
| Level | Meaning | Implementation |
|---|---|---|
| At-most-once | May lose messages | Fire and forget |
| At-least-once | May duplicate messages | Ack + retry |
| Exactly-once | No loss, no duplicates | Transactional (Kafka) |
Message queues are essential infrastructure for building reliable, scalable distributed systems — by decoupling components and buffering communication, they enable parallel processing at scale while providing fault tolerance that synchronous communication cannot offer.