Message Brokers: Kafka & RabbitMQ

Kafka is a distributed append-only log: a topic is split into partitions, each an ordered log, and consumers in a group share the partitions and record progress as an offset per partition. RabbitMQ is a message queue: messages are routed into queues, acknowledged one by one and deleted. Kafka vocabulary Partition: the unit of ordering and parallelism. The key's hash picks the partition, so messages with one key stay in order. Consumer group: each partition is read by exactly one member; members beyond the partition count sit idle. Offset and commit: a commit means "everything before offset N is done"; after a crash or rebalance the group resumes from there. Retention: messages stay for a configured time or size whether read or not, so a new consumer can replay history. Delivery guarantees At-most-once: commit before processing. At-least-once: commit after (the one you normally want). Exactly-once holds only inside Kafka (idempotent producer plus transactions); for side effects outside it you need an idempotent consumer. Kafka vs RabbitMQ Kafka: high throughput, replay, ordering per partition, consumers pull. Fits event streams, logs, CDC. RabbitMQ: routing through exchanges, per-message ack and redelivery, priorities, TTL and dead-letter exchanges built in. Fits task queues. RabbitMQ streams and Kafka 4.x share groups narrow the gap, but the default holds: an event log goes to Kafka, a work queue to RabbitMQ.

A producer publishes to an exchange, not to a queue, and the exchange copies the message into every queue whose binding matches. Direct matches the routing key exactly, topic matches it against patterns, fanout ignores it and feeds every bound queue, and headers matches on message headers. Binding: links a queue to an exchange with a binding key (QueueBind); every matching queue gets its own copy of the message. Direct: the routing key equals the binding key, for commands and work queues. The default exchange "" binds every queue under its own name, so publishing with a queue's name as the key reaches that queue. Topic: keys are dot-separated words; * matches one word, # zero or more. order.*.created gets order.eu.created, order.# gets every order event. Fanout: a broadcast. Each subscriber needs its own queue, because consumers sharing one queue split the messages. Headers: matches header values (x-match set to all or any); rarely needed. No match: the message is dropped silently, unless it was published with mandatory (the broker returns it) or the exchange has an alternate exchange.

ZooKeeper stored the cluster's metadata (brokers, topics, partition leaders, configs, ACLs) and elected the controller. KRaft moves that metadata into Kafka itself: a small quorum of controller nodes keeps it as a Raft-replicated log, and Kafka 4.0 removed ZooKeeper mode entirely. What ZooKeeper did: brokers kept a session there, so an expired session meant a dead broker, and it elected the one controller broker that assigned partition leaders. Consumer offsets had long since moved to the __consumer_offsets topic. KRaft: three or five controllers (process.roles=controller) replicate the metadata in the internal __cluster_metadata topic; brokers fetch it like any other log. What it gives: one system to run and secure instead of two; fast controller failover, because standby controllers already hold the metadata, while a ZooKeeper-era controller reloaded it for minutes on a large cluster; and many more partitions per cluster. Migration: KRaft has been production-ready since 3.3. A ZooKeeper cluster must migrate on a 3.x bridge release (the last is 3.9) before it can upgrade to 4.0. Go clients are unaffected: they only ever talk to brokers.

Kafka turns every workload into large sequential I/O: each partition is an append-only file, reads and writes go through the OS page cache, records travel in compressed batches, and the broker sends log data to consumers with sendfile, without copying it through the JVM. Append-only log: a write appends to the active segment, a read scans forward from an offset. Sequential I/O is fast even on spinning disks, and there is no per-message B-tree to update, only a sparse offset index. Page cache: no message cache in the JVM heap and no fsync per write; the OS caches the files, so consumers near the tail read from memory, the cache survives a broker restart and the GC sees a small heap. Durability comes from replication. Batching: the producer groups records per partition (linger.ms, 5 ms by default since Kafka 4.0, and batch.size) and compresses whole batches (lz4, zstd); the broker stores them as they arrived, so one request moves thousands of records. Zero-copy: the file format is the wire format, so sendfile moves bytes from the page cache to the socket without passing through user space. TLS turns this off, since encryption runs in user space. Cheap reads: the broker keeps one offset per group and partition, no per-message state, and reading deletes nothing, so one more consumer group costs little.

Kafka deletes data by age or size, never because it was read: retention.ms (7 days by default) and retention.bytes remove whole old segments. Log compaction (cleanup.policy=compact) instead keeps at least the latest record for every key, so the topic works as a table of current values. Delete policy Retention deletes whole segments (segment.bytes 1 GiB, segment.ms 7 days), and only once a segment's newest record is past retention. A topic with a steady trickle of writes rolls its active segment only after segment.ms, so old records in it can outlive retention.ms by up to that much. retention.bytes is a per-partition limit, unlimited by default. A consumer that falls behind retention loses data, and auto.offset.reset decides where it resumes: alert on lag measured in time, long before retention. Compaction A background cleaner keeps only the newest record per key in old segments. Offsets do not change, so consumers see gaps; the active segment is not compacted, so recent duplicates remain. Keyless writes are rejected. A null value is a tombstone that deletes the key. It is removed itself after delete.retention.ms (1 day by default), so a consumer that rebuilds state from the start more slowly than that can miss a deletion. It fits current state keyed by id: profiles, CDC changelogs, Kafka Streams state, __consumer_offsets itself. A new service reads it from the start and gets the latest value of every key. cleanup.policy=compact,delete combines both.

Committed offsets live in the compacted internal topic __consumer_offsets, one record per group, topic and partition. auto.offset.reset applies only when a partition has no valid committed offset: latest (the Java default) jumps to the end, earliest starts at the oldest record, none fails. When the reset applies: a brand-new group; a committed offset that retention has already deleted; a group that stayed empty longer than offsets.retention.minutes (7 days), whose offsets expired, so on its return it silently skips to the end or rereads everything. Client defaults differ: a new Java group misses everything written before it started, while kafka-go defaults StartOffset to kafka.FirstOffset and reads the whole history. Set it explicitly. Replaying a group from a time: each partition has a time index, so Kafka can find the first offset at or after a timestamp. Stop the group's consumers, then run kafka-consumer-groups.sh --bootstrap-server kafka:9092 --group billing --topic orders --reset-offsets --to-datetime 2026-10-20T09:00:00.000 --execute (--dry-run first); --to-earliest and --shift-by -100 work the same way. A one-off job: a reader outside any group sets its own position (SetOffsetAt in kafka-go) and leaves the group's offsets alone. Replays must be harmless: reprocessing a day of events is safe only if the handlers are idempotent.

Kafka orders messages only within a partition, and keyless messages are spread across all twelve, so two events of one order land on different partitions and race. And each partition is read by one member of the group, so members 13 to 24 got nothing. **Key by order_id**: the partition is hash(key) % partitions, so every event of one order goes to one partition and is read in the order it was written. segmentio/kafka-go defaults Writer.Balancer to RoundRobin, which ignores the key: set &kafka.Hash{}, or &kafka.Murmur2Balancer{} to match Java producers. Adding partitions later changes hash(key) % n for existing keys and splits one order's history across old and new partitions. Over-provision up front. To go wider than the partition count, process in parallel inside one consumer with a worker per key hash, so per-key order survives. Rebalance: when a member joins, leaves or misses its heartbeats, partitions move and uncommitted work is redelivered to the new owner. Cooperative rebalancing and the Kafka 4.0 consumer protocol (KIP-848, Java client) avoid stopping the whole group.

With acks=1 the leader confirms a write before any follower has copied it, so when that leader dies the new one never had the record. A write survives losing a broker with acks=all, replication factor 3 and min.insync.replicas=2: it is confirmed only once at least two replicas hold it. ISR: the in-sync replicas, the leader plus the followers that caught up within replica.lag.time.max.ms (30 s). Only an ISR member can become leader, unless unclean.leader.election.enable is turned on (it is off by default). **acks**: 0 waits for nothing, 1 for the leader's own write, all (-1) for every replica currently in the ISR. **Why acks=all alone is not enough**: when the ISR has shrunk to the leader, all means one copy. min.insync.replicas=2 makes the leader refuse acks=all writes with NOT_ENOUGH_REPLICAS while fewer than two replicas are in sync (acks=1 ignores it): writes stop instead of data being lost later. RF 3 with min ISR 2 keeps accepting writes with one broker down for a crash or a rolling upgrade; RF 2 with min ISR 2 stops writes on every restart. Defaults to check: the Java producer has used acks=all since Kafka 3.0, but the broker's min.insync.replicas is 1, and the zero value of kafka-go's Writer.RequiredAcks is RequireNone, which is fire-and-forget. Memory, not disk: an acknowledged record sits in each replica's page cache, so place replicas in racks or zones that do not fail together.

It stops the producer's own retries from writing a batch twice: the broker gives the producer an id, each batch carries a per-partition sequence number, and the leader discards a batch it already has. It does not stop a send your code repeats, a restarted producer or a consumer that processes twice. The duplicate it fixes: the batch is written, the response is lost, the producer retries. The leader sees a sequence number it already has and answers success without writing again. Order across retries: with up to 5 requests in flight, a plain producer can write batch 3 and then the retry of a failed batch 2. With idempotence the leader rejects a batch that is not next in sequence, so the producer resends in order; without it, order took max.in.flight.requests.per.connection=1. Requirements: acks=all and retries. The Java producer has it on by default since Kafka 3.0. Not covered: a second send after a timeout (a new record, a new sequence number), a restart (a new producer id, unless a transactional.id links the sessions), and duplicates on the consumer side. Those need an idempotent consumer. In Go: franz-go is idempotent by default, confluent-kafka-go needs "enable.idempotence": true, and kafka-go's Writer has no such mode. Transactions (transactional.id) build on it to write several partitions and the consumer's offsets atomically.

WriteMessages blocks until its batch is sent, and the writer sends a batch when it reaches BatchSize (100 messages by default) or when BatchTimeout (1 second by default) expires. One message per request never fills a batch, so every call waits out the full second. Not the network: the broker answers in milliseconds; the time goes on waiting for messages that never arrive. Concurrent calls for the same partition share a batch, so the delay shrinks under load and is worst at low traffic. **BatchTimeout**: a few milliseconds (5 * time.Millisecond, like linger.ms in the Java client) brings a lone message down to about one round trip, and batches still form under load. **BatchSize**: a batch also goes out once it holds 100 messages, which only happens under load. A bulk job should pass all its messages to one WriteMessages call: they then wait for one flush per partition, not one per message. **Async: true**: WriteMessages returns immediately and errors reach only the Completion callback; without one, a failed write disappears. Acceptable for metrics or click logs, not for an order event. Also wrong here: RequiredAcks is unset, and its zero value is RequireNone, so the broker does not even confirm the write. Set kafka.RequireAll.

For a new service, franz-go: pure Go, actively developed, and it covers almost the whole protocol, idempotence and transactions included. kafka-go has the simplest API but neither feature, confluent-kafka-go is complete but brings cgo, and sarama is the oldest and the most verbose. **twmb/franz-go**: idempotent writes and acks=all by default, transactions, cooperative rebalancing, admin (kadm) and schema registry (sr) packages. The cost is a large option-based API to learn. **segmentio/kafka-go**: a small Writer/Reader API. No idempotent or transactional producer, and defaults unlike Java's: RequireNone acks, a 1 s BatchTimeout, FirstOffset for a new group. **confluentinc/confluent-kafka-go**: a wrapper over librdkafka, the C library behind most non-Java clients, so it is complete and battle-tested. The cost is cgo: CGO_ENABLED=1, prebuilt librdkafka only for common platforms (Alpine needs -tags musl), slower builds, harder cross-compiling, and crashes inside C that a Go stack trace does not explain. **IBM/sarama** (formerly Shopify/sarama): pure Go and long in production, but a consumer group means implementing ConsumerGroupHandler (Setup, Cleanup, ConsumeClaim) and tuning a large Config. Whichever you pick: set acks, the start offset and the commit mode explicitly, hide the client behind a small interface of your own, and test against a real broker in a container.

Committing before the work is done makes the consumer at-most-once: the group already recorded those offsets as processed, so whatever was queued or in flight died with the pod and will never be redelivered. Commit only after the effect is durable. Nothing fails anywhere, which is why such gaps are found only by reconciliation. Committing after processing gives at-least-once: a crash costs a redelivery, which an idempotent handler absorbs. An offset is a position, not a set: committing 100 while 95 is still running marks 95 as done. With a worker pool, commit per partition only up to the lowest offset still in flight. In kafka-go, ReadMessage with a GroupID commits for you; FetchMessage plus CommitMessages lets you choose when. A rebalance works like a crash: a member that misses its session (SessionTimeout in kafka-go) loses its partitions mid-batch.

At-least-once delivery plus an idempotent consumer. Exactly-once delivery over a network is impossible (a lost message and a lost acknowledgement look the same to the sender), so you make the effect happen once instead. Kafka's exactly-once semantics (idempotent producer plus transactions) covers read-process-write inside Kafka only. Your database and the card processor are outside it. Store a dedupe key (the message id or a business key) under a unique constraint in the same transaction as the effect. A redelivery hits the constraint, the handler returns success, and the offset moves on. Prefer naturally idempotent writes: SET status = 'paid' repeats safely, balance = balance - 10 does not. For an effect you cannot roll back (an email, a card capture), pass the message id to the provider as its idempotency key. Keep dedupe rows longer than the longest possible redelivery window.

Separate permanent errors from transient ones, retry only the transient ones with bounded backoff, and park the rest in a dead-letter topic with the error and the source offset, then commit and move on. One message that will never succeed must not block a partition. Permanent vs transient: a payload that fails to decode or validate fails every time; a database timeout may not. Retrying the first kind only burns time. Retry topics: for long waits, republish to orders.retry.1m or orders.retry.10m instead of sleeping, so the main partition keeps flowing. Dead-letter topic (DLQ): keep key, value and headers with the error text, the original topic, partition and offset, and the attempt count. Alert on its growth and have a replay tool: a DLQ nobody reads is data loss with extra steps. Ordering: parking one event lets later events of the same key overtake it. When order matters, park the key: send its following messages to the DLQ too until the first is fixed. If the DLQ write itself fails, do not commit; stop rather than lose the message.

The queue must be durable, the message published as persistent (DeliveryMode: 2), and the publisher must wait for a confirm, because only a confirm means the broker has taken the message. To survive losing the node itself, the queue must also be a quorum queue, replicated across nodes with Raft. Durable and persistent: a non-durable queue is deleted on restart, and a classic queue drops transient messages on recovery even when it is durable (a quorum queue persists every message). The exchange must be durable too, or its bindings vanish. Publisher confirms: in confirm mode (ch.Confirm(false)) the broker acks a message once its queues have accepted it: on disk for a persistent message in a classic queue, on a majority of replicas in a quorum queue. Without confirms, a nil from Publish only means the bytes left your process. Nacks and unroutable messages: a nack means publish again. A message that matched no queue is acked too and dropped, unless it was published with mandatory and you handle the returns. Quorum queues: a classic queue lives on one node (classic mirroring was removed in RabbitMQ 4.0). A quorum queue (x-queue-type: quorum) is always durable, keeps replicas on three or five nodes, survives losing a minority of them, and drops or dead-letters a poison message after its delivery limit (20 by default since 4.0). The cost: persistence and confirms lower throughput, so do not wait for each confirm in turn; collect them in batches.

Consume is called with autoAck set to true, so the broker treats each message as done the moment it writes it to the socket, and deletes it. Whatever the worker was rendering, and everything buffered in msgs, dies with the pod. Use manual acks and Ack only after the work succeeds. Why durability did not help: durable queues and persistent messages protect messages that are in the queue. With auto-ack a delivered message is no longer in the queue at all. More than one message: auto-ack has no prefetch limit, so the broker pushes the backlog into the client's memory as fast as the network allows; a crash loses all of it. The fix: autoAck: false, ch.Qos(10, 0, false) to cap the unacknowledged deliveries one worker holds, and d.Ack(false) after render returns. When a channel or connection closes with unacked messages, RabbitMQ requeues them for another worker. The second bug: on a render error the loop only logs, so that job is lost too. Nack(false, false) sends it to the queue's dead-letter exchange, or drops it if the queue has none; requeueing it forever would be a redelivery loop. At-least-once now: a pod killed after rendering but before Ack gets the message redelivered with d.Redelivered set, so the handler must be idempotent. A quorum queue also takes back a delivery left unacked past consumer_timeout (30 minutes by default); since RabbitMQ 4.3 classic queues do not.

Consumers on one queue compete: each message goes to exactly one of them, handed out in turn up to each consumer's prefetch limit. The queue itself stays FIFO, but parallel processing and redeliveries break the order in which work finishes; a single active consumer, or a queue per key, restores it. Dispatch: round-robin among consumers with free prefetch slots (Qos). With unlimited prefetch a slow consumer gets as many messages as a fast one, and the first to subscribe can take the whole backlog. Prefetch 1 is the fairest and the slowest; the RabbitMQ docs suggest 100 to 300 for fast handlers, far less for long jobs. A work queue, not a broadcast: for every subscriber to see every message, each one declares its own queue bound to a fanout or topic exchange. How order breaks: two consumers work on messages 1 and 2 at once and 2 finishes first; a nacked message, or one held by a consumer that crashed, is redelivered after later messages were already processed. Single active consumer (x-single-active-consumer: true): all instances subscribe, the queue delivers to one at a time and fails over to the next when it disconnects. Order holds, at the throughput of a single consumer. Per-key order with parallelism: several queues, each with a single active consumer, routed by a hash of the key (the x-consistent-hash exchange plugin). That rebuilds Kafka partitions by hand; if per-key order is the core requirement, Kafka is the simpler fit.

A thin event (order_id and a version) only notifies, so every consumer calls the orders service back, which adds coupling, load and races. A fat event carries the state consumers need, so they work on their own, at the price of bigger messages and a payload that becomes a public contract. Thin events (notification) Every consumer makes a synchronous call to the owner: the owner's outage stalls all of them, and a burst of events becomes a burst of GETs against it. The call returns the current state, not the state at the time of the event: two quick updates can both read the second version, and the consumer cannot tell what changed. They fit when the payload is large or sensitive, or when the consumer needs fresh data anyway. Fat events (event-carried state transfer) The consumer keeps its own copy (a read model) and never calls back; replaying the topic rebuilds it, and a compacted topic keyed by order_id holds the latest state of every order. The payload is now a contract: every field you publish is a field you must keep, and personal data spreads into every consumer's database and into Kafka for the whole retention period. Messages get bigger: fine at kilobytes, wrong for files (send a link instead). What makes either work A version that grows with every change at the source: the consumer applies an event only if it is newer than what it has, so redeliveries and reordering cannot roll state back. Full state rather than deltas (status changed to shipped): a lost or reordered full-state event is corrected by the next one, a lost delta stays wrong. Names from the business (OrderShipped), not table diffs, so the owner can change its schema without breaking consumers.

Make only compatible changes, in an order that keeps every running version working: add fields with defaults, and never rename, retype or reuse one. A schema registry enforces it: each new schema is registered before use, and the registry rejects it if it breaks the subject's compatibility rule. Compatible and breaking changes Safe: a new field with a default (Avro) or a new field number (Protobuf). Old consumers ignore it, new consumers see the default in old messages. Breaking: renaming a field, changing its type, making an optional field required, or deleting one that consumers still read. Protobuf field numbers follow the same reserved rule as in gRPC. JSON without a schema gets no checks. Go's encoding/json ignores unknown fields, and a consumer that turns on DisallowUnknownFields makes every added field an outage; a contract test in CI is the minimum. Deploy order **BACKWARD (the Confluent registry's default): the new schema can read data written with the previous one, so consumers are upgraded first, then producers. FORWARD: the previous schema can read data written with the new one, so producers go first. FULL** is both, and either side can deploy first. The _TRANSITIVE variants check against every earlier version, not only the last, which matters when a topic is replayed from the beginning. What the registry does It keeps schema versions per subject (by default <topic>-value); the serializer prefixes each message with a magic byte and a 4-byte schema id, so a consumer fetches the writer's exact schema once and caches it. An incompatible change fails at registration, in CI or at producer startup, instead of in a consumer at 3 a.m. In Go: the schemaregistry package of confluent-kafka-go, or franz-go's sr. A truly breaking change goes to a new topic (orders.v2): the producer writes both for a while, consumers move over, then v1 is retired.