Irbisa · cheatsheetSeptember 19, 2026

gRPC, Messaging & Services

gRPC and protobuf evolution, deadlines, Kafka semantics, outbox, retries

Senior Developer16 itemscompressed for a skim

01
A cleanup PR deletes an unused bool is_trial = 4; from order.proto, and a new int64 trial_days = 4; takes the freed number. Staging is green; production starts reporting nonsense trial lengths. What rule was broken?
Easy
A protobuf field is identified on the wire by its tag number, never by its name, so the new field silently inherited the retired one's bytes from every peer still running the old schema — a deleted number must be reserved, not recycled.
```
// order.proto, after the v2.3 cleanup:
//
//   message Order {
//     reserved 4;             // was bool is_trial, deleted in v2.3
//     reserved "is_trial";    // and nobody may take the name back either
//
…
```
02
UpdateUser is meant to be a partial update, but every caller that leaves notify_email out of the request turns the flag off. The field is declared bool notify_email = 3;. What is missing from the schema?
Easy
Plain proto3 scalars have no field presence — an omitted field and an explicit false decode to the same zero value — so declare it optional, which generates a *bool, or make the caller send a FieldMask.
```
// user.proto
//   message UpdateUserRequest {
//     string id = 1;
//     optional bool notify_email = 3;              // presence -> *bool in Go
//     google.protobuf.FieldMask update_mask = 9;
//   }
…
```
03
A handler returns fmt.Errorf("order %s not found", id). The client logs rpc error: code = Unknown desc = order 42 not found, retries it as a transient failure, and the text surfaces in a customer-facing 500. What should the server return instead?
Easy
Any error that is not already a *status.Status crosses the wire as codes.Unknown carrying your raw text — return status.Error(codes.NotFound, ...) and classify it on the client with status.FromError.
```
var ErrNoOrder = errors.New("order not found")

func (s *server) GetOrder(ctx context.Context, req *pb.GetOrderRequest) (*pb.Order, error) {
	o, err := s.db.Load(ctx, req.GetId())
	switch {
	case errors.Is(err, ErrNoOrder):
…
```
04
A worker pool calls grpc.NewClient(target, ...) at the top of every job and closes the connection when the job ends. Under load the service burns CPU on TLS handshakes and netstat shows thousands of sockets in TIME_WAIT. What is the correct lifetime for a ClientConn?
Easy
A grpc.ClientConn is a long-lived, goroutine-safe channel — one per target for the life of the process.
```
// ❌ a ClientConn per call: DNS, TCP and the TLS handshake every time, a fresh
// resolver and balancer, and a socket left in TIME_WAIT for each one.
func chargeBad(ctx context.Context, req *pb.ChargeRequest) error {
	conn, err := grpc.NewClient(target, grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		return err
…
```
05
The gateway gives a request a 2 s deadline, yet a trace shows orders still calling pricing three seconds after the gateway already answered DeadlineExceeded. Every service wraps its outbound calls in context.WithTimeout(context.Background(), 2*time.Second). What is wrong with that line?
Medium
context.Background() starts a new deadline chain, so the hop never learns that the caller has given up — derive the outbound context from the inbound one and spend less than the budget that is left.
```
func (s *Orders) Place(ctx context.Context, req *pb.PlaceRequest) (*pb.PlaceResponse, error) {
	// the caller's remaining budget arrived as the grpc-timeout header
	dl, ok := ctx.Deadline()
	if !ok {
		dl = time.Now().Add(2 * time.Second) // no caller budget: impose one
	}
…
```
06
An interceptor puts the caller's tenant into ctx and every unary RPC sees it, but the one server-streaming method panics because the tenant is missing — and ss.Context() = newCtx does not compile. What is the fix?
Medium
Streaming RPCs go through a separate interceptor chain, so register a grpc.StreamServerInterceptor as well; and since grpc.ServerStream exposes Context() read-only, hand the handler a wrapper that overrides it.
```
// grpc.ServerStream hands out its context read-only, so a stream interceptor
// that needs to add a value passes the handler a wrapper instead.
type ctxStream struct {
	grpc.ServerStream
	ctx context.Context
}
…
```
07
Ten replicas sit behind a Kubernetes ClusterIP Service. REST traffic spreads evenly across them; gRPC traffic pins about 90% of calls to one pod. Why does the same Service behave differently, and what are the fixes?
Medium
kube-proxy balances connections, not requests, and gRPC multiplexes every call onto one long-lived HTTP/2 connection — balance in the client (a headless Service plus round_robin) or terminate HTTP/2 in an L7 proxy.
```
func dial() (*grpc.ClientConn, error) {
	return grpc.NewClient(
		// headless Service (clusterIP: None): DNS returns every pod IP, not one
		// virtual IP that kube-proxy pins a single connection to.
		"dns:///orders.prod.svc.cluster.local:8080",
		grpc.WithTransportCredentials(insecure.NewCredentials()),
…
```
08
A worker pushes to a gRPC service every few minutes. The first call after a quiet period fails with Unavailable: connection reset by peer and the immediate retry always succeeds. The service sits behind a cloud load balancer. What is happening?
Medium
An idle TCP connection is being reaped by a middlebox without either end being told, so your RPC is what discovers it — enable HTTP/2 keepalive pings so the connection is either kept warm or found dead before a call is riding on it.
```
func dial() (*grpc.ClientConn, error) {
	return grpc.NewClient(target,
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithKeepaliveParams(keepalive.ClientParameters{
			Time:                30 * time.Second, // default is "never"; below 10s is clamped to 10s
			Timeout:             10 * time.Second, // no ping ack within this -> transport is dead
…
```
09
A retry policy is added to the payments client and support starts seeing double charges whenever the provider slows down. ChargeCard fails with Unavailable when the provider times out. Is the retry wrong, or the method?
Medium
Both, in the order that matters: a transport retry is only safe on an idempotent method, and Unavailable says the RPC failed, not that the server did nothing — so either the write dedupes on a caller-supplied idempotency key, or ChargeCard stays out of the retry policy.
```
// Attached with grpc.WithDefaultServiceConfig: the policy is per method, and
// Charge is deliberately not in it — a write with side effects is not something
// the transport may replay on its own.
const svcConfig = `{
  "methodConfig": [{
    "name": [{"service": "payments.v1.Payments", "method": "GetCharge"}],
…
```
10
The new service is gRPC-only, but the web front end cannot call it from the browser and a mobile release from last year still speaks the old JSON API. Do you write and maintain a second HTTP server?
Medium
No — generate the JSON edge from the same .proto with grpc-gateway, and keep the version in the proto package (orders.v1), so one implementation serves both transports and a breaking change ships as orders.v2 beside v1.
```
// orders/v1/orders.proto — the version lives in the package path, so a breaking
// change ships as orders.v2 alongside v1 instead of mutating it.
//
//   package orders.v1;
//   service Orders {
//     rpc GetOrder(GetOrderRequest) returns (Order) {
…
```
11
After a broker failover the payments consumer replays the last minute of messages and a few customers are charged twice. Product asks you to switch on exactly-once delivery. What do you actually build?
Medium
At-least-once delivery plus an idempotent consumer — exactly-once delivery is not achievable across a network, so the guarantee has to live in the effect rather than in the transport.
```
// At-least-once in, effect-once out: the dedupe row and the effect commit in one
// transaction, so a redelivery can never apply the second without the first.
func handle(ctx context.Context, db *sql.DB, m Message) error {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return err
…
```
12
A consumer commits the offset as soon as it fetches a message and hands the work to a pool of goroutines. After a pod eviction, several hundred events turn out never to have been processed, and nothing was logged. Where did they go?
Medium
Committing before the work is durable makes the consumer at-most-once: the broker already considers those offsets delivered, so the eviction dropped everything in flight and nothing will ever redeliver it.
```
func consume(ctx context.Context) error {
	r := kafka.NewReader(kafka.ReaderConfig{
		Brokers: brokers, GroupID: "billing", Topic: "orders",
		// ReadMessage would auto-commit as it hands the message over; FetchMessage
		// does not, which is the only way to decide when "done" means done.
	})
…
```
13
order.shipped sometimes arrives before order.created for the same order, and doubling the consumer group from 12 to 24 members left the lag exactly where it was. One topic, 12 partitions, messages produced without a key. Name both mistakes.
Hard
Ordering in Kafka holds only within a partition and an unkeyed message is spread across all of them, so the two events raced; and a partition is owned by exactly one member of a group, so members 13–24 were assigned nothing at all.
```
func produce(ctx context.Context) error {
	// ❌ no key, and kafka-go's default balancer is RoundRobin: the two events for
	//    one order land on different partitions, and nothing orders them again.
	bad := &kafka.Writer{Addr: kafka.TCP(brokers...), Topic: "orders"}
	if err := bad.WriteMessages(ctx,
		kafka.Message{Value: created},
…
```
14
CreateOrder commits the row and then publishes OrderCreated. About once a day an order exists with no event, and after a broker blip a handful of events exist with no order. Which pattern removes both cases at once?
Hard
Two writes to two systems cannot be made atomic by reordering them — the fix is the transactional outbox: insert the event into an outbox table inside the same database transaction as the order, and let a separate relay publish from there.
```
func CreateOrder(ctx context.Context, tx *sql.Tx, o Order) error {
	if _, err := tx.ExecContext(ctx,
		`INSERT INTO orders (id, total) VALUES ($1, $2)`, o.ID, o.Total); err != nil {
		return err
	}
	payload, err := protojson.Marshal(event(o))
…
```
15
Checkout must reserve stock, charge the card and book a courier, each owned by a different service. An architect proposes wrapping the three in one distributed transaction. What do you build instead, and what does it cost you?
Hard
A saga: a sequence of local transactions, each with a compensating action, driven by an orchestrator that persists its own state — you keep atomicity in the business sense and give up isolation, and that trade has to be designed for explicitly.
```
type Step struct {
	Name string
	Do   func(context.Context, *Saga) error
	// Semantic undo, not a rollback: idempotent, and retried until it succeeds,
	// because there is nothing else to fall back on.
	Compensate func(context.Context, *Saga) error
…
```
16
One dependency's p99 goes from 400 ms to 6 s and four services that never call it directly fall over with it. Every client retries three times and every timeout is 30 s. Walk through the amplification, and the controls that stop it.
Hard
Retries multiply load exactly when the system has the least capacity, and a 30 s timeout turns every stuck call into 30 s of held goroutines, connections and buffers — the controls are a deadline budget, a circuit breaker, and a bulkhead with a rate limiter in front of it.
```
type Guard struct {
	lim  *rate.Limiter // rate.NewLimiter(500, 50): 500 rps, burst 50
	slot chan struct{} // bulkhead: make(chan struct{}, 20) calls in flight, no more
}

func (g *Guard) Unary(ctx context.Context, req any, _ *grpc.UnaryServerInfo, h grpc.UnaryHandler) (any, error) {
…
```

gRPC, Messaging & Services

A cleanup PR deletes an unused `bool is_trial = 4;` from `order.proto`, and a new `int64 trial_days = 4;` takes the freed number. Staging is green; production starts reporting nonsense trial lengths. What rule was broken?

`UpdateUser` is meant to be a partial update, but every caller that leaves `notify_email` out of the request turns the flag off. The field is declared `bool notify_email = 3;`. What is missing from the schema?

A handler returns `fmt.Errorf("order %s not found", id)`. The client logs `rpc error: code = Unknown desc = order 42 not found`, retries it as a transient failure, and the text surfaces in a customer-facing 500. What should the server return instead?

A worker pool calls `grpc.NewClient(target, ...)` at the top of every job and closes the connection when the job ends. Under load the service burns CPU on TLS handshakes and `netstat` shows thousands of sockets in `TIME_WAIT`. What is the correct lifetime for a `ClientConn`?

The gateway gives a request a 2 s deadline, yet a trace shows `orders` still calling `pricing` three seconds after the gateway already answered `DeadlineExceeded`. Every service wraps its outbound calls in `context.WithTimeout(context.Background(), 2*time.Second)`. What is wrong with that line?

An interceptor puts the caller's tenant into `ctx` and every unary RPC sees it, but the one server-streaming method panics because the tenant is missing — and `ss.Context() = newCtx` does not compile. What is the fix?

Ten replicas sit behind a Kubernetes `ClusterIP` Service. REST traffic spreads evenly across them; gRPC traffic pins about 90% of calls to one pod. Why does the same Service behave differently, and what are the fixes?

A worker pushes to a gRPC service every few minutes. The first call after a quiet period fails with `Unavailable: connection reset by peer` and the immediate retry always succeeds. The service sits behind a cloud load balancer. What is happening?

A retry policy is added to the payments client and support starts seeing double charges whenever the provider slows down. `ChargeCard` fails with `Unavailable` when the provider times out. Is the retry wrong, or the method?

The new service is gRPC-only, but the web front end cannot call it from the browser and a mobile release from last year still speaks the old JSON API. Do you write and maintain a second HTTP server?

After a broker failover the payments consumer replays the last minute of messages and a few customers are charged twice. Product asks you to switch on exactly-once delivery. What do you actually build?

A consumer commits the offset as soon as it fetches a message and hands the work to a pool of goroutines. After a pod eviction, several hundred events turn out never to have been processed, and nothing was logged. Where did they go?

`order.shipped` sometimes arrives before `order.created` for the same order, and doubling the consumer group from 12 to 24 members left the lag exactly where it was. One topic, 12 partitions, messages produced without a key. Name both mistakes.

`CreateOrder` commits the row and then publishes `OrderCreated`. About once a day an order exists with no event, and after a broker blip a handful of events exist with no order. Which pattern removes both cases at once?

Checkout must reserve stock, charge the card and book a courier, each owned by a different service. An architect proposes wrapping the three in one distributed transaction. What do you build instead, and what does it cost you?

One dependency's p99 goes from 400 ms to 6 s and four services that never call it directly fall over with it. Every client retries three times and every timeout is 30 s. Walk through the amplification, and the controls that stop it.

A cleanup PR deletes an unused bool is_trial = 4; from order.proto, and a new int64 trial_days = 4; takes the freed number. Staging is green; production starts reporting nonsense trial lengths. What rule was broken?

UpdateUser is meant to be a partial update, but every caller that leaves notify_email out of the request turns the flag off. The field is declared bool notify_email = 3;. What is missing from the schema?

A handler returns fmt.Errorf("order %s not found", id). The client logs rpc error: code = Unknown desc = order 42 not found, retries it as a transient failure, and the text surfaces in a customer-facing 500. What should the server return instead?

A worker pool calls grpc.NewClient(target, ...) at the top of every job and closes the connection when the job ends. Under load the service burns CPU on TLS handshakes and netstat shows thousands of sockets in TIME_WAIT. What is the correct lifetime for a ClientConn?

The gateway gives a request a 2 s deadline, yet a trace shows orders still calling pricing three seconds after the gateway already answered DeadlineExceeded. Every service wraps its outbound calls in context.WithTimeout(context.Background(), 2*time.Second). What is wrong with that line?

An interceptor puts the caller's tenant into ctx and every unary RPC sees it, but the one server-streaming method panics because the tenant is missing — and ss.Context() = newCtx does not compile. What is the fix?

Ten replicas sit behind a Kubernetes ClusterIP Service. REST traffic spreads evenly across them; gRPC traffic pins about 90% of calls to one pod. Why does the same Service behave differently, and what are the fixes?

A worker pushes to a gRPC service every few minutes. The first call after a quiet period fails with Unavailable: connection reset by peer and the immediate retry always succeeds. The service sits behind a cloud load balancer. What is happening?

A retry policy is added to the payments client and support starts seeing double charges whenever the provider slows down. ChargeCard fails with Unavailable when the provider times out. Is the retry wrong, or the method?

The new service is gRPC-only, but the web front end cannot call it from the browser and a mobile release from last year still speaks the old JSON API. Do you write and maintain a second HTTP server?

After a broker failover the payments consumer replays the last minute of messages and a few customers are charged twice. Product asks you to switch on exactly-once delivery. What do you actually build?

A consumer commits the offset as soon as it fetches a message and hands the work to a pool of goroutines. After a pod eviction, several hundred events turn out never to have been processed, and nothing was logged. Where did they go?

order.shipped sometimes arrives before order.created for the same order, and doubling the consumer group from 12 to 24 members left the lag exactly where it was. One topic, 12 partitions, messages produced without a key. Name both mistakes.

CreateOrder commits the row and then publishes OrderCreated. About once a day an order exists with no event, and after a broker blip a handful of events exist with no order. Which pattern removes both cases at once?

Checkout must reserve stock, charge the card and book a courier, each owned by a different service. An architect proposes wrapping the three in one distributed transaction. What do you build instead, and what does it cost you?

One dependency's p99 goes from 400 ms to 6 s and four services that never call it directly fall over with it. Every client retries three times and every timeout is 30 s. Walk through the amplification, and the controls that stop it.

A cleanup PR deletes an unused `bool is_trial = 4;` from `order.proto`, and a new `int64 trial_days = 4;` takes the freed number. Staging is green; production starts reporting nonsense trial lengths. What rule was broken?

`UpdateUser` is meant to be a partial update, but every caller that leaves `notify_email` out of the request turns the flag off. The field is declared `bool notify_email = 3;`. What is missing from the schema?

A handler returns `fmt.Errorf("order %s not found", id)`. The client logs `rpc error: code = Unknown desc = order 42 not found`, retries it as a transient failure, and the text surfaces in a customer-facing 500. What should the server return instead?

A worker pool calls `grpc.NewClient(target, ...)` at the top of every job and closes the connection when the job ends. Under load the service burns CPU on TLS handshakes and `netstat` shows thousands of sockets in `TIME_WAIT`. What is the correct lifetime for a `ClientConn`?

The gateway gives a request a 2 s deadline, yet a trace shows `orders` still calling `pricing` three seconds after the gateway already answered `DeadlineExceeded`. Every service wraps its outbound calls in `context.WithTimeout(context.Background(), 2*time.Second)`. What is wrong with that line?

An interceptor puts the caller's tenant into `ctx` and every unary RPC sees it, but the one server-streaming method panics because the tenant is missing — and `ss.Context() = newCtx` does not compile. What is the fix?

Ten replicas sit behind a Kubernetes `ClusterIP` Service. REST traffic spreads evenly across them; gRPC traffic pins about 90% of calls to one pod. Why does the same Service behave differently, and what are the fixes?

A worker pushes to a gRPC service every few minutes. The first call after a quiet period fails with `Unavailable: connection reset by peer` and the immediate retry always succeeds. The service sits behind a cloud load balancer. What is happening?

A retry policy is added to the payments client and support starts seeing double charges whenever the provider slows down. `ChargeCard` fails with `Unavailable` when the provider times out. Is the retry wrong, or the method?

`order.shipped` sometimes arrives before `order.created` for the same order, and doubling the consumer group from 12 to 24 members left the lag exactly where it was. One topic, 12 partitions, messages produced without a key. Name both mistakes.

`CreateOrder` commits the row and then publishes `OrderCreated`. About once a day an order exists with no event, and after a broker blip a handful of events exist with no order. Which pattern removes both cases at once?