What are gRPC and Protocol Buffers, and when would you choose gRPC over REST with JSON?

gRPC is an RPC framework over HTTP/2 that uses Protocol Buffers (a typed, binary schema in a `.proto` file) to generate client and server code.

What are the trade-offs between a monolith and microservices, and which would you choose for a five-person startup?

A monolith is one deployable unit; microservices are independently deployed services, each owning its data.

When should services talk synchronously over REST or gRPC, and when through a message broker? What does the asynchronous option cost?

Call synchronously when the caller needs the answer to continue (a price, a permission check); publish an event when it only reports that something happened.

What do an API gateway, service discovery and a service mesh do, and which concerns do they take out of your Go code?

An API gateway is the single entry point for external clients (routing, auth, rate limits, TLS), service discovery tells a service where healthy instances of another one are, and a service mesh adds a proxy next to every service for mTLS, retries, timeouts, balancing and telemetry.

Which .proto changes are safe once clients are in production, why must a deleted field number be reserved, and what does proto3 optional add?

On the wire a field is identified only by its number, so adding or renaming fields is safe on the binary wire, while changing a number or a type, or reusing a deleted number, silently breaks old peers.

What is a gRPC interceptor, and why does an auth interceptor that works for every unary method never run for streaming ones?

An interceptor is gRPC middleware: a function wrapped around every call for auth, logging, metrics or recovery.

How does gRPC's built-in retry work, which status codes belong in retryableStatusCodes, and why must the method be idempotent before you enable it?

gRPC retries a failed call by itself when the service config has a `retryPolicy` for the method: up to `maxAttempts` tries with backoff, only for the status codes you list.

Ten replicas sit behind a Kubernetes ClusterIP Service. REST traffic spreads evenly, but gRPC sends about 90% of calls to one pod. Why, and how do you fix it?

kube-proxy balances connections, not requests, and gRPC sends every call as a stream on one long-lived HTTP/2 connection, so the pod that got the connection gets the traffic.

A dependency's p99 jumps from 400 ms to 6 s and four services behind a shared caller fall over. Clients retry 3x with 30 s timeouts. What amplified it, and what stops it?

Retries multiply load exactly when capacity is lowest (three attempts at each of three hops is up to 27 times the traffic), and 30 s timeouts keep every stuck request holding a goroutine and a pooled connection until the shared caller runs out of goroutines and connections for everyone.

Checkout must reserve stock, charge the card and book a courier, each in a different service. Why not a distributed transaction, and how does a saga work instead?

A saga is a sequence of local transactions, each with a compensating action; if a step fails, the finished ones are compensated in reverse order.

Irbisa · Cheatsheet

gRPC & Microservices

Q: This handler returns a fmt.Errorf error for a missing order. Which status code does the gRPC client get, and how should the handler report "not found"?

The client gets `codes.Unknown` with your raw text, `order 42: sql: no rows in result set`.

Q: How long should a grpc.ClientConn live (one per call or one per process), and what does opening a new connection for every call cost?

One per target for the life of the process: create it with `grpc.NewClient` at startup, share it between goroutines, close it at shutdown.

gRPC and protobuf evolution, deadlines, sync vs async calls, outbox and sagas, retries and circuit breakers

Senior Developer · 20 questions

01
What are gRPC and Protocol Buffers, and when would you choose gRPC over REST with JSON?
gRPC is an RPC framework over HTTP/2 that uses Protocol Buffers (a typed, binary schema in a .proto file) to generate client and server code.
```
// orders/v1/orders.proto////   syntax = "proto3";//   package orders.v1;//   option go_package = "example.com/gen/orders/v1;ordersv1";//…
```
02
What are the trade-offs between a monolith and microservices, and which would you choose for a five-person startup?
A monolith is one deployable unit; microservices are independently deployed services, each owning its data.
03
When should services talk synchronously over REST or gRPC, and when through a message broker? What does the asynchronous option cost?
Call synchronously when the caller needs the answer to continue (a price, a permission check); publish an event when it only reports that something happened.

What are the four kinds of gRPC calls, and when is streaming the right choice over a unary call?

Unary (one request, one response), server streaming (one request, many responses), client streaming (many requests, one response) and bidirectional (both sides send independently on one stream).

//   service Catalog {//     rpc GetItem(GetItemRequest) returns (Item);            // unary//     rpc ListItems(ListRequest) returns (stream Item);      // server streaming//     rpc Upload(stream Item) returns (UploadSummary);       // client streaming//     rpc Sync(stream SyncEvent) returns (stream SyncEvent); // bidirectional//   }…

This handler returns a `fmt.Errorf` error for a missing order. Which status code does the gRPC client get, and how should the handler report "not found"?

func (s *server) GetOrder(ctx context.Context, req *pb.GetOrderRequest) (*pb.Order, error) {	o, err := s.repo.Find(ctx, req.GetId())	if errors.Is(err, sql.ErrNoRows) {		return nil, fmt.Errorf("order %s: %w", req.GetId(), err)	}	if err != nil {		return nil, err	}	return o, nil}

The client gets codes.Unknown with your raw text, order 42: sql: no rows in result set.

How long should a `grpc.ClientConn` live (one per call or one per process), and what does opening a new connection for every call cost?

One per target for the life of the process: create it with grpc.NewClient at startup, share it between goroutines, close it at shutdown.

type Payments struct{ client pb.PaymentsClient } // Built once in main; conn.Close is returned for shutdown.func NewPayments(target string) (*Payments, func() error, error) {	conn, err := grpc.NewClient(target,		grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})),…

07
What do an API gateway, service discovery and a service mesh do, and which concerns do they take out of your Go code?
An API gateway is the single entry point for external clients (routing, auth, rate limits, TLS), service discovery tells a service where healthy instances of another one are, and a service mesh adds a proxy next to every service for mTLS, retries, timeouts, balancing and telemetry.
08
Which .proto changes are safe once clients are in production, why must a deleted field number be reserved, and what does proto3 optional add?
On the wire a field is identified only by its number, so adding or renaming fields is safe on the binary wire, while changing a number or a type, or reusing a deleted number, silently breaks old peers.
```
// user.proto////   message UpdateUserRequest {//     reserved 4;                       // was bool is_trial//     reserved "is_trial";              // and the name cannot come back either//     string id = 1;…
```

How does a gRPC deadline differ from an HTTP client timeout, and what does the server see when the client's deadline passes?

An HTTP client timeout is only a local timer: the server is told nothing about the budget and notices only when the connection is cut.

func quote(ctx context.Context, c pb.PricingClient, sku string) (*pb.Quote, error) {	// no default deadline in gRPC: without this a stuck server holds the call forever	ctx, cancel := context.WithTimeout(ctx, 300*time.Millisecond)	defer cancel()	q, err := c.Quote(ctx, &pb.QuoteRequest{Sku: sku}) // remaining time sent as grpc-timeout	if status.Code(err) == codes.DeadlineExceeded {…

10
What is a gRPC interceptor, and why does an auth interceptor that works for every unary method never run for streaming ones?
An interceptor is gRPC middleware: a function wrapped around every call for auth, logging, metrics or recovery.
```
func UnaryAuth(ctx context.Context, req any, _ *grpc.UnaryServerInfo, h grpc.UnaryHandler) (any, error) {	ctx, err := authenticate(ctx)	if err != nil {		return nil, err	}	return h(ctx, req)…
```

Find the bug: one client's request makes the orders pod restart, and every other in-flight call fails with `Unavailable`.

func (s *server) ListOrders(ctx context.Context, req *pb.ListOrdersRequest) (*pb.ListOrdersResponse, error) {	limit := int(req.Page.Size)	if limit <= 0 || limit > 100 {		limit = 20	}	orders, next, err := s.repo.List(ctx, req.GetCustomerId(), limit, req.Page.Token)	if err != nil {		return nil, status.Error(codes.Internal, "list orders failed")	}	return &pb.ListOrdersResponse{Orders: orders, NextPageToken: next}, nil}

req.Page is a message field, so it is a nil pointer when the client leaves it unset, and req.Page.Size panics.

This client returns the first in-stock item from a server stream. No-match searches log `search failed: EOF`, and goroutines pile up. What are the two bugs?

func findFirst(ctx context.Context, c pb.CatalogClient, q string) (*pb.Item, error) {	stream, err := c.Search(ctx, &pb.SearchRequest{Query: q})	if err != nil {		return nil, err	}	for {		item, err := stream.Recv()		if err != nil {			log.Printf("search failed: %v", err)			return nil, err		}		if item.GetInStock() {			return item, nil		}	}}

io.EOF from Recv is the normal end of a stream, not a failure; and returning early without cancelling the stream's context leaves the stream alive.

13
How does gRPC's built-in retry work, which status codes belong in retryableStatusCodes, and why must the method be idempotent before you enable it?
gRPC retries a failed call by itself when the service config has a retryPolicy for the method: up to maxAttempts tries with backoff, only for the status codes you list.
```
// Passed with grpc.WithDefaultServiceConfig. Charge is listed only because it carries// an idempotency key; a write without one must never be retried by the transport.const svcConfig = `{  "methodConfig": [{    "name": [{"service": "payments.v1.Payments", "method": "Charge"}],    "retryPolicy": {…
```

What does a circuit breaker do, what are its three states, and when should it trip?

A circuit breaker wraps calls to one dependency and stops calling it once failures cross a threshold, failing fast instead of waiting on timeouts.

// github.com/sony/gobreaker/v2var pricingCB = gobreaker.NewCircuitBreaker[*pb.Quote](gobreaker.Settings{	Name:        "pricing",	MaxRequests: 3,                // trial calls while half-open	Interval:    30 * time.Second, // closed: reset the counts every 30 s	Timeout:     10 * time.Second, // open -> half-open after 10 s…

Find the bug: a week after a deploy, this circuit breaker keeps opening for 30 s at a time, though the dependency fails one call in a thousand.

func (b *Breaker) Do(fn func() error) error {	b.mu.Lock()	if b.failures >= 5 && time.Since(b.lastFail) < 30*time.Second {		b.mu.Unlock()		return ErrOpen	}	b.mu.Unlock()	err := fn()	b.mu.Lock()	if err != nil {		b.failures++		b.lastFail = time.Now()	}	b.mu.Unlock()	return err}

failures is never reset on success, so it counts every error since the process started.

16
Ten replicas sit behind a Kubernetes ClusterIP Service. REST traffic spreads evenly, but gRPC sends about 90% of calls to one pod. Why, and how do you fix it?
kube-proxy balances connections, not requests, and gRPC sends every call as a stream on one long-lived HTTP/2 connection, so the pod that got the connection gets the traffic.
```
func dial() (*grpc.ClientConn, error) {	return grpc.NewClient(		// headless Service: DNS returns every pod IP, not one virtual IP		"dns:///orders.prod.svc.cluster.local:8080",		grpc.WithTransportCredentials(insecure.NewCredentials()),		// the default policy is pick_first: one pod gets everything…
```
17
A dependency's p99 jumps from 400 ms to 6 s and four services behind a shared caller fall over. Clients retry 3x with 30 s timeouts. What amplified it, and what stops it?
Retries multiply load exactly when capacity is lowest (three attempts at each of three hops is up to 27 times the traffic), and 30 s timeouts keep every stuck request holding a goroutine and a pooled connection until the shared caller runs out of goroutines and connections for everyone.
```
// One Bulkhead per dependency: at most cap(slot) calls in flight to it.type Bulkhead struct{ slot chan struct{} } func NewBulkhead(n int) *Bulkhead { return &Bulkhead{slot: make(chan struct{}, n)} } func (b *Bulkhead) Unary(ctx context.Context, method string, req, reply any,…
```

Some orders never get their `OrderCreated` event, and publishing before the commit instead gives events for orders that rolled back. Which pattern fixes both?

Two writes to two systems cannot be made atomic by reordering them.

func CreateOrder(ctx context.Context, tx *sql.Tx, o Order, payload []byte) error {	_, err := tx.ExecContext(ctx, `INSERT INTO orders (id, total) VALUES ($1, $2)`, o.ID, o.Total)	if err == nil { // same transaction: the event exists if and only if the order does		_, err = tx.ExecContext(ctx,			`INSERT INTO outbox (aggregate_id, type, payload) VALUES ($1, $2, $3)`,			o.ID, "OrderCreated", payload)…

19
Checkout must reserve stock, charge the card and book a courier, each in a different service. Why not a distributed transaction, and how does a saga work instead?
A saga is a sequence of local transactions, each with a compensating action; if a step fails, the finished ones are compensated in reverse order.

How do you build a distributed lock on Redis, and why is a lock without a fencing token unsafe?

Acquire with SET key token NX PX ttl; release with a script that deletes the key only if it still holds your token.

func acquire(ctx context.Context, rdb *redis.Client, key string, ttl time.Duration) (string, bool, error) {	token := uuid.NewString() // random: only the owner can release it	ok, err := rdb.SetNX(ctx, "lock:"+key, token, ttl).Result()	return token, ok, err}…

gRPC & Microservices

What are gRPC and Protocol Buffers, and when would you choose gRPC over REST with JSON?

What are the trade-offs between a monolith and microservices, and which would you choose for a five-person startup?

When should services talk synchronously over REST or gRPC, and when through a message broker? What does the asynchronous option cost?

What are the four kinds of gRPC calls, and when is streaming the right choice over a unary call?

This handler returns a `fmt.Errorf` error for a missing order. Which status code does the gRPC client get, and how should the handler report "not found"?

How long should a `grpc.ClientConn` live (one per call or one per process), and what does opening a new connection for every call cost?

What do an API gateway, service discovery and a service mesh do, and which concerns do they take out of your Go code?

Which `.proto` changes are safe once clients are in production, why must a deleted field number be `reserved`, and what does proto3 `optional` add?

How does a gRPC deadline differ from an HTTP client timeout, and what does the server see when the client's deadline passes?

What is a gRPC interceptor, and why does an auth interceptor that works for every unary method never run for streaming ones?

Find the bug: one client's request makes the orders pod restart, and every other in-flight call fails with `Unavailable`.

This client returns the first in-stock item from a server stream. No-match searches log `search failed: EOF`, and goroutines pile up. What are the two bugs?

How does gRPC's built-in retry work, which status codes belong in `retryableStatusCodes`, and why must the method be idempotent before you enable it?

What does a circuit breaker do, what are its three states, and when should it trip?

Find the bug: a week after a deploy, this circuit breaker keeps opening for 30 s at a time, though the dependency fails one call in a thousand.

Ten replicas sit behind a Kubernetes `ClusterIP` Service. REST traffic spreads evenly, but gRPC sends about 90% of calls to one pod. Why, and how do you fix it?

A dependency's p99 jumps from 400 ms to 6 s and four services behind a shared caller fall over. Clients retry 3x with 30 s timeouts. What amplified it, and what stops it?

Some orders never get their `OrderCreated` event, and publishing before the commit instead gives events for orders that rolled back. Which pattern fixes both?

Checkout must reserve stock, charge the card and book a courier, each in a different service. Why not a distributed transaction, and how does a saga work instead?

How do you build a distributed lock on Redis, and why is a lock without a fencing token unsafe?

What are gRPC and Protocol Buffers, and when would you choose gRPC over REST with JSON?

What are the trade-offs between a monolith and microservices, and which would you choose for a five-person startup?

When should services talk synchronously over REST or gRPC, and when through a message broker? What does the asynchronous option cost?

What are the four kinds of gRPC calls, and when is streaming the right choice over a unary call?

This handler returns a fmt.Errorf error for a missing order. Which status code does the gRPC client get, and how should the handler report "not found"?

How long should a grpc.ClientConn live (one per call or one per process), and what does opening a new connection for every call cost?

What do an API gateway, service discovery and a service mesh do, and which concerns do they take out of your Go code?

Which .proto changes are safe once clients are in production, why must a deleted field number be reserved, and what does proto3 optional add?

How does a gRPC deadline differ from an HTTP client timeout, and what does the server see when the client's deadline passes?

What is a gRPC interceptor, and why does an auth interceptor that works for every unary method never run for streaming ones?

Find the bug: one client's request makes the orders pod restart, and every other in-flight call fails with Unavailable.

This client returns the first in-stock item from a server stream. No-match searches log search failed: EOF, and goroutines pile up. What are the two bugs?

How does gRPC's built-in retry work, which status codes belong in retryableStatusCodes, and why must the method be idempotent before you enable it?

What does a circuit breaker do, what are its three states, and when should it trip?

Find the bug: a week after a deploy, this circuit breaker keeps opening for 30 s at a time, though the dependency fails one call in a thousand.

Ten replicas sit behind a Kubernetes ClusterIP Service. REST traffic spreads evenly, but gRPC sends about 90% of calls to one pod. Why, and how do you fix it?

A dependency's p99 jumps from 400 ms to 6 s and four services behind a shared caller fall over. Clients retry 3x with 30 s timeouts. What amplified it, and what stops it?

Some orders never get their OrderCreated event, and publishing before the commit instead gives events for orders that rolled back. Which pattern fixes both?

Checkout must reserve stock, charge the card and book a courier, each in a different service. Why not a distributed transaction, and how does a saga work instead?

How do you build a distributed lock on Redis, and why is a lock without a fencing token unsafe?

This handler returns a `fmt.Errorf` error for a missing order. Which status code does the gRPC client get, and how should the handler report "not found"?

How long should a `grpc.ClientConn` live (one per call or one per process), and what does opening a new connection for every call cost?

Which `.proto` changes are safe once clients are in production, why must a deleted field number be `reserved`, and what does proto3 `optional` add?

Find the bug: one client's request makes the orders pod restart, and every other in-flight call fails with `Unavailable`.

This client returns the first in-stock item from a server stream. No-match searches log `search failed: EOF`, and goroutines pile up. What are the two bugs?

How does gRPC's built-in retry work, which status codes belong in `retryableStatusCodes`, and why must the method be idempotent before you enable it?

Ten replicas sit behind a Kubernetes `ClusterIP` Service. REST traffic spreads evenly, but gRPC sends about 90% of calls to one pod. Why, and how do you fix it?

Some orders never get their `OrderCreated` event, and publishing before the commit instead gives events for orders that rolled back. Which pattern fixes both?