Explain Kafka's topic, partition, consumer group and offset, and how Kafka differs from RabbitMQ.

Kafka is a distributed append-only log: a topic is split into partitions, each an ordered log, and consumers in a group share the partitions and record progress as an offset per partition.

What did ZooKeeper do for Kafka, and what changed when Kafka 4.0 switched to KRaft?

ZooKeeper stored the cluster's metadata (brokers, topics, partition leaders, configs, ACLs) and elected the controller.

Why is Kafka so fast on ordinary disks, and what do the append-only log, the page cache, batching and zero-copy each contribute?

Kafka turns every workload into large sequential I/O: each partition is an append-only file, reads and writes go through the OS page cache, records travel in compressed batches, and the broker sends log data to consumers with `sendfile`, without copying it through the JVM.

How long does Kafka keep messages, and what is the difference between time- or size-based retention and log compaction?

Kafka deletes data by age or size, never because it was read: `retention.ms` (7 days by default) and `retention.bytes` remove whole old segments.

Events for one order arrive out of order, and growing the consumer group from 12 to 24 members did not reduce lag. 12 partitions, messages sent without a key. Why both?

Kafka orders messages only within a partition, and keyless messages are spread across all twelve, so two events of one order land on different partitions and race.

What does enable.idempotence on a Kafka producer prevent, how does it keep order across retries, and which duplicates does it not stop?

It stops the producer's own retries from writing a batch twice: the broker gives the producer an id, each batch carries a per-partition sequence number, and the leader discards a batch it already has.

Which Go Kafka client would you choose (franz-go, segmentio/kafka-go, confluent-kafka-go, sarama), and what does each one cost you?

For a new service, `franz-go`: pure Go, actively developed, and it covers almost the whole protocol, idempotence and transactions included.

This consumer commits each offset, then hands the message to a worker pool. After a pod eviction, hundreds of events were silently lost. Where did they go?

Committing before the work is done makes the consumer at-most-once: the group already recorded those offsets as processed, so whatever was queued or in flight died with the pod and will never be redelivered.

After a broker failover the payments consumer replays a minute of messages and some customers pay twice. Product wants exactly-once delivery. What do you build?

At-least-once delivery plus an idempotent consumer.

One malformed message makes the consumer fail and retry forever, and its whole partition stops moving. How do you handle poison messages?

Separate permanent errors from transient ones, retry only the transient ones with bounded backoff, and park the rest in a dead-letter topic with the error and the source offset, then commit and move on.

What must be true for a RabbitMQ message to survive a broker restart, and what do publisher confirms and quorum queues add?

The queue must be durable, the message published as persistent (`DeliveryMode: 2`), and the publisher must wait for a confirm, because only a confirm means the broker has taken the message.

Several consumers read one RabbitMQ queue. Who gets which message, what happens to ordering, and when do you need a single active consumer?

Consumers on one queue compete: each message goes to exactly one of them, handed out in turn up to each consumer's prefetch limit.

Should an OrderUpdated event carry the whole order or only its id, and what does each choice cost the consumers?

A thin event (`order_id` and a version) only notifies, so every consumer calls the orders service back, which adds coupling, load and races.

Producers and consumers of an event deploy on their own schedules. How do you change the event's schema safely, and what does a schema registry check?

Make only compatible changes, in an order that keeps every running version working: add fields with defaults, and never rename, retype or reuse one.

Irbisa · Cheatsheet

Message Brokers: Kafka & RabbitMQ

Q: Where does Kafka keep a consumer group's committed offsets, what does auto.offset.reset do, and how do you replay a topic from a given time?

Committed offsets live in the compacted internal topic `__consumer_offsets`, one record per group, topic and partition.

Q: A broker died and some acknowledged messages are gone. How do acks, the replication factor, min.insync.replicas and the ISR decide whether a write survives?

With `acks=1` the leader confirms a write before any follower has copied it, so when that leader dies the new one never had the record.

Q: Each HTTP request publishes one event through this kafka.Writer and takes just over a second. Why, and what do BatchTimeout, BatchSize and Async trade off?

`WriteMessages` blocks until its batch is sent, and the writer sends a batch when it reaches `BatchSize` (100 messages by default) or when `BatchTimeout` (1 second by default) expires.

Partitions, consumer groups and offsets, replication and acks, delivery guarantees, idempotent producers, retention and compaction, RabbitMQ exchanges and acks, event schemas

Middle Developer · 19 questions

01
Explain Kafka's topic, partition, consumer group and offset, and how Kafka differs from RabbitMQ.
Kafka is a distributed append-only log: a topic is split into partitions, each an ordered log, and consumers in a group share the partitions and record progress as an offset per partition.

What exchange types does RabbitMQ have, and how do bindings and routing keys decide which queues receive a message?

A producer publishes to an exchange, not to a queue, and the exchange copies the message into every queue whose binding matches.

func setup(ch *amqp.Channel) error {	// durable: the exchange, the queue and the binding survive a broker restart	if err := ch.ExchangeDeclare("orders", amqp.ExchangeTopic, true, false, false, false, nil); err != nil {		return err	}	q, err := ch.QueueDeclare("billing", true, false, false, false, nil)…

03
What did ZooKeeper do for Kafka, and what changed when Kafka 4.0 switched to KRaft?
ZooKeeper stored the cluster's metadata (brokers, topics, partition leaders, configs, ACLs) and elected the controller.
04
Why is Kafka so fast on ordinary disks, and what do the append-only log, the page cache, batching and zero-copy each contribute?
Kafka turns every workload into large sequential I/O: each partition is an append-only file, reads and writes go through the OS page cache, records travel in compressed batches, and the broker sends log data to consumers with sendfile, without copying it through the JVM.
05
How long does Kafka keep messages, and what is the difference between time- or size-based retention and log compaction?
Kafka deletes data by age or size, never because it was read: retention.ms (7 days by default) and retention.bytes remove whole old segments.

Where does Kafka keep a consumer group's committed offsets, what does `auto.offset.reset` do, and how do you replay a topic from a given time?

Committed offsets live in the compacted internal topic __consumer_offsets, one record per group, topic and partition.

// A group reader: StartOffset applies only to partitions with no committed offset.func newBillingReader(brokers []string) *kafka.Reader {	return kafka.NewReader(kafka.ReaderConfig{		Brokers:     brokers,		GroupID:     "billing",		Topic:       "orders",…

07
Events for one order arrive out of order, and growing the consumer group from 12 to 24 members did not reduce lag. 12 partitions, messages sent without a key. Why both?
Kafka orders messages only within a partition, and keyless messages are spread across all twelve, so two events of one order land on different partitions and race.
```
// wrong: &kafka.Writer{Addr: ..., Topic: "orders"} with keyless messages;// the default balancer is RoundRobin, so one order's events scatter. // right: key by the entity whose order matters, with a balancer that hashes itvar w = &kafka.Writer{	Addr:         kafka.TCP("kafka-1:9092", "kafka-2:9092"),…
```

A broker died and some acknowledged messages are gone. How do `acks`, the replication factor, `min.insync.replicas` and the ISR decide whether a write survives?

With acks=1 the leader confirms a write before any follower has copied it, so when that leader dies the new one never had the record.

// Topic: three copies, and writes are refused while fewer than two are in sync.func createPayments(ctx context.Context, c *kafka.Client) error {	res, err := c.CreateTopics(ctx, &kafka.CreateTopicsRequest{Topics: []kafka.TopicConfig{{		Topic:             "payments",		NumPartitions:     12,		ReplicationFactor: 3,…

09
What does enable.idempotence on a Kafka producer prevent, how does it keep order across retries, and which duplicates does it not stop?
It stops the producer's own retries from writing a batch twice: the broker gives the producer an id, each batch carries a per-partition sequence number, and the leader discards a batch it already has.

Each HTTP request publishes one event through this `kafka.Writer` and takes just over a second. Why, and what do `BatchTimeout`, `BatchSize` and `Async` trade off?

var w = &kafka.Writer{	Addr:     kafka.TCP("kafka-1:9092", "kafka-2:9092"),	Topic:    "orders",	Balancer: &kafka.Hash{},} // Called once per POST /orders, after the order is saved.func publishCreated(ctx context.Context, o Order) error {	return w.WriteMessages(ctx, kafka.Message{		Key:   []byte(o.ID),		Value: o.JSON(),	})}

WriteMessages blocks until its batch is sent, and the writer sends a batch when it reaches BatchSize (100 messages by default) or when BatchTimeout (1 second by default) expires.

11
Which Go Kafka client would you choose (franz-go, segmentio/kafka-go, confluent-kafka-go, sarama), and what does each one cost you?
For a new service, franz-go: pure Go, actively developed, and it covers almost the whole protocol, idempotence and transactions included.
12
This consumer commits each offset, then hands the message to a worker pool. After a pod eviction, hundreds of events were silently lost. Where did they go?
```
func consume(ctx context.Context, r *kafka.Reader, jobs chan<- kafka.Message) error {	for {		m, err := r.FetchMessage(ctx)		if err != nil {			return err		}		if err := r.CommitMessages(ctx, m); err != nil {			return err		}		jobs <- m // 16 goroutines read jobs and call handle(ctx, m)	}}
```
Committing before the work is done makes the consumer at-most-once: the group already recorded those offsets as processed, so whatever was queued or in flight died with the pod and will never be redelivered.
13
After a broker failover the payments consumer replays a minute of messages and some customers pay twice. Product wants exactly-once delivery. What do you build?
At-least-once delivery plus an idempotent consumer.
```
func handle(ctx context.Context, db *sql.DB, r *kafka.Reader, m kafka.Message) error {	tx, err := db.BeginTx(ctx, nil)	if err != nil {		return err	}	defer tx.Rollback() // no-op after Commit…
```
14
One malformed message makes the consumer fail and retry forever, and its whole partition stops moving. How do you handle poison messages?
Separate permanent errors from transient ones, retry only the transient ones with bounded backoff, and park the rest in a dead-letter topic with the error and the source offset, then commit and move on.
```
func process(ctx context.Context, r *kafka.Reader, dlq *kafka.Writer, m kafka.Message) error {	var err error	for attempt := 0; attempt < 3; attempt++ {		if err = handle(ctx, m); err == nil {			return r.CommitMessages(ctx, m)		}…
```
15
What must be true for a RabbitMQ message to survive a broker restart, and what do publisher confirms and quorum queues add?
The queue must be durable, the message published as persistent (DeliveryMode: 2), and the publisher must wait for a confirm, because only a confirm means the broker has taken the message.
```
// Once at startup: a replicated queue and a channel in confirm mode.func setup(ch *amqp.Channel) error {	_, err := ch.QueueDeclare("payments", true, false, false, false, amqp.Table{		amqp.QueueTypeArg: amqp.QueueTypeQuorum, // replicated with Raft, always durable	})	if err != nil {…
```

Find the bug: when a worker pod is killed mid-job, the RabbitMQ messages it was processing are gone, although the queue is durable and the messages persistent.

func work(ctx context.Context, ch *amqp.Channel) error {	msgs, err := ch.Consume("thumbnails", "", true, false, false, false, nil)	if err != nil {		return err	}	for d := range msgs {		if err := render(ctx, d.Body); err != nil {			log.Printf("render failed: %v", err)		}	}	return nil}

Consume is called with autoAck set to true, so the broker treats each message as done the moment it writes it to the socket, and deletes it.

17
Several consumers read one RabbitMQ queue. Who gets which message, what happens to ordering, and when do you need a single active consumer?
Consumers on one queue compete: each message goes to exactly one of them, handed out in turn up to each consumer's prefetch limit.
18
Should an OrderUpdated event carry the whole order or only its id, and what does each choice cost the consumers?
A thin event (order_id and a version) only notifies, so every consumer calls the orders service back, which adds coupling, load and races.
```
type OrderUpdated struct {	OrderID    string `json:"order_id"`	Version    int64  `json:"version"` // grows with every change at the source	Status     string `json:"status"`	TotalCents int64  `json:"total_cents"`}…
```
19
Producers and consumers of an event deploy on their own schedules. How do you change the event's schema safely, and what does a schema registry check?
Make only compatible changes, in an order that keeps every running version working: add fields with defaults, and never rename, retype or reuse one.