The choice between Kafka and RabbitMQ depends on what a message represents and what should happen after it is processed. A task usually represents pending work that some worker must complete. An event records something that happened and may interest several services, including one that starts consuming later.
Routing, retention, and recovery make that difference concrete. The most common comparison contrasts RabbitMQ classic or quorum queues with Kafka’s partitioned log; RabbitMQ also offers a Streams model with retention and replay.
RabbitMQ and task queues
RabbitMQ receives publications through an exchange and routes them to queues according to its rules and bindings. That separation allows work to be directed by keys, patterns, or destinations, and copies to be distributed to several queues when appropriate.
A set of workers can compete for deliveries from the same queue. With manual acknowledgement, the consumer sends an acknowledgement after completing processing. The broker can then remove the message from that queue; copies in other queues have their own delivery and acknowledgement lifecycle.
If a consumer loses its connection while it has unacknowledged messages, those messages may be delivered again. Rejection, requeueing, expiration deadlines, and routing to a queue for failed messages change that behavior. The recovery policy must distinguish a temporary error from a task that cannot be completed.
This model fits email delivery, report generation, and other background work. Flexible routing allows responsibilities to be separated and policies to be applied per queue. A payment flow can also use it, provided its effects are coordinated idempotently.
For a topology of tasks and workers, the model can be easy to follow. Operational simplicity and latency nevertheless depend on the queue type, replication, backlog, and deployment. A queue that keeps growing or an unlimited retry cycle still needs capacity control.
Kafka and the event log
Kafka organizes records into topics divided into partitions. Producers append records to the log, and each partition preserves an order by offset. A key can direct related events to a partition under the partitioning strategy in use.
In a conventional consumer group, partitions are distributed among consumers. The group records its offsets to track its progress. Different groups maintain their progress independently, so analytics, notifications, and projections can read the same stream without competing for a single delivery.
Reading or committing an offset does not delete the record. Retention is configured by time or size, and some topics use compaction by key. Replay only reaches data that still exists. A compacted log retains information according to its policy, but not necessarily every historical event for an entity.
Kafka fits when several services need a retained stream, need to rebuild projections, or need to introduce new consumers without depending on the progress of earlier ones. History must be sized for the recovery it is intended to support; Kafka does not automatically act as a permanent archive.
Ordering is per partition. Groups, rebalances, and concurrent processing require additional attention: read order alone does not guarantee the completion order of effects. In RabbitMQ, multiple consumers, priorities, or redeliveries can also change the order observed when processing a queue.
Two consumption models
The diagram shows a queue served by workers and a topic read by two independent groups. A RabbitMQ publication may have other routes and queues; this example represents one task destination.
--- config: look: handDrawn --- flowchart LR subgraph rabbit["RabbitMQ · Task queue"] direction TB producerR["Producer"] --> exchange["Exchange<br/>Routing rules"] exchange --> queue[("Queue")] queue --> workers["Competing workers<br/>Process and acknowledge"] workers -.->|"Delivery ack"| queue end subgraph kafka["Kafka · Event log"] direction TB producerK["Producer"] --> topic[("Partitioned topic<br/>Configured retention")] topic --> groupA["Group A<br/>Independent offsets"] topic --> groupB["Group B<br/>Independent offsets"] end rabbit ~~~ kafka
| Aspect | RabbitMQ queues | Kafka log |
|---|---|---|
| Distribution | Exchanges route to one or more queues. | Producers publish to partitioned topics. |
| Consumption | Workers compete for deliveries from a queue. | Independent groups read the same stream. |
| Progress | The ack acknowledges a delivery in its queue. | The commit records the group’s offset. |
| History | Acknowledged messages are no longer pending in that queue. | Records remain according to retention. |
| Recovery | Redelivery of pending work. | Resume offsets or reread retained data. |
Redelivering a pending task and replaying history serve different needs. If a log is needed within RabbitMQ, the Streams model must be evaluated under its own retention and consumption contract.
Delivery and idempotency
Both RabbitMQ and Kafka allow at-least-once delivery flows to be configured. In that model, a record may be processed more than once during recovery. The guarantee depends on acknowledgements, retries, persistence, retention, and the failure conditions covered; it is not an unconditional promise of no losses.
In RabbitMQ, publisher confirms tell the producer that the broker has assumed responsibility for the publication according to the queue and its configuration. They are independent of the consumer’s ack and do not prove that the work has finished. An unroutable publication can receive a confirm; detecting and handling messages with no destination requires an appropriate policy, such as publishing with mandatory and handling returns.
Durability also depends on queue and message configuration and on replication. If a worker completes the effect and crashes before sending the ack, delivery may repeat. Acknowledging before completing the effect can leave work undone after a failure.
In Kafka, processing and then committing the offset leaves a similar window: if the process crashes between those actions, the record may be processed again. Committing the offset before completing the work can advance the group even though the effect has not been performed.
The acknowledgement to the producer has a different scope. With acks=all, the write waits for acknowledgements from the replicas that form the in-sync set, or ISR, at that time. min.insync.replicas sets the minimum threshold for admitting that write. Replication, leader election, and recovery policies complete the durability and availability contract.
Consumers therefore need a stable event or operation identity and a way to recognize effects already performed. If the effect is in a database, deduplication and the local change can be stored in the same transaction. An external charge also requires coordination and idempotency under the provider’s guarantees.
Kafka offers exactly-once mechanisms within a defined scope. Producer idempotence prevents duplicates from certain protocol retries; it does not automatically identify two equivalent business requests. Transactions can coordinate records produced in Kafka and consumed offsets, with readers configured to see committed data through read_committed. That mechanism does not make a charge or a write to an external database atomic with the offset.
Performance and operations
Kafka uses batching, sequential writes, compression, and partitions to distribute load. RabbitMQ maintains routing and delivery state for its queues. Both consume resources to store, transmit, and replicate data; the cost depends on the model and configuration.
A comparison of 4,000 to 10,000 messages per second in RabbitMQ with more than one million messages per second in Kafka needs to describe the workload and measurement conditions. Those figures alone establish neither typical rates nor product limits nor a constant 100-fold advantage.
A useful benchmark must specify:
- Message size, batching, and compression.
- Persistence, replicas, and acknowledgement policy.
- Number of producers, consumers, queues, or partitions.
- Hardware, network, and distribution of destinations or keys.
- Target latency, tolerated backlog, and behavior during failures.
Relevant capacity includes completing consumer effects and recovering under load. Publishing quickly while pending work grows does not demonstrate that the entire flow has that capacity.
How to choose
RabbitMQ often fits when tasks must be assigned, messages routed to specific destinations, and their delivery lifecycle controlled. Kafka often fits when a stream must be retained and reread, and several groups need to maintain independent progress. Operations and latency must be validated for the real workload in both cases.
Both can coexist when there are different needs: a retained event in Kafka can trigger a task directed to RabbitMQ. The bridge adds another failure point and another delivery that can repeat. The event identity must be preserved, and consumers must tolerate the task being published again.
When an operation also modifies a database, publication requires resolving the dual write. The transactional outbox described in SD: 2PC vs Saga stores the local change and pending event in the same transaction; the relay and consumers maintain their own recovery and idempotency rules.
What to verify
The central tests stop a consumer after it produces the effect and before it acknowledges, lose an acknowledgement to the producer, and repeat publications. They must also cover unroutable messages, node failures, messages that cannot be processed, and prolonged consumer delays.
For Kafka, add rebalances, concurrent consumers, and recovery after retention expires. For RabbitMQ, verify requeueing, limits on unacknowledged messages, and the destination of failed tasks.
Useful signals include queue depth and age, unacknowledged messages, lag by group and partition, end-to-end latency, retries, and duplicate effects. The contract is verified when it is possible to explain which work remains pending, which history remains available, and how each uncertain effect is recovered.