gRPC & Microservices

gRPC is an RPC framework over HTTP/2 that uses Protocol Buffers (a typed, binary schema in a .proto file) to generate client and server code. Choose it for calls between your own services; keep REST/JSON for browsers and public APIs. Contract first: the .proto is the single source of truth. protoc-gen-go and protoc-gen-go-grpc (or buf generate) produce typed messages and a client stub, so a mismatch breaks the build, not production. Cheaper on the wire: the binary encoding is several times smaller than JSON and faster to parse, and HTTP/2 runs many calls over one connection. Built in: deadlines, status codes, metadata and streaming, which REST leaves to convention. The cost: payloads are not human-readable (grpcurl instead of curl), and a browser cannot call gRPC directly: it needs gRPC-Web behind a proxy, Connect, or a JSON edge generated from the same .proto with grpc-gateway.

A monolith is one deployable unit; microservices are independently deployed services, each owning its data. They buy team autonomy and independent scaling, paid for with network failures, distributed data and heavy operations, so a five-person team starts with a modular monolith. What microservices give you Teams build, deploy and roll back on their own schedule. The hot part scales alone, each service picks its own storage, and a failure can be contained behind timeouts and circuit breakers. What they cost Every call can fail or be slow, so deadlines, retries, idempotency and breakers become mandatory. No transaction spans services: outbox, sagas and eventual consistency replace a JOIN and a BEGIN. Tracing, service discovery, a pipeline per service, contract versioning and on-call for many deployables. How to decide Split when team boundaries and deploy contention start to hurt, not because it is fashionable; services follow the team structure (Conway's law). In Go, a modular monolith means a package per domain under internal/, modules talking through interfaces, a schema per module in one database. A module with a clean boundary can later move out as a service. Avoid the distributed monolith: services that must be deployed together or share tables pay every cost and get none of the benefits.

Call synchronously when the caller needs the answer to continue (a price, a permission check); publish an event when it only reports that something happened. Async buys decoupling and buffering at the cost of eventual consistency. Sync: simple to follow and debug, but availability multiplies: a request through five services at 99.9% each is about 99.5% available, and latencies add up. Async: the broker absorbs spikes and outages, and the producer does not know its consumers, so adding a new one changes nothing upstream. What async costs: the result is not visible at once (the UI shows "processing"); messages arrive at least once and can arrive out of order, so consumers must be idempotent; tracing needs the trace id in message headers; the broker is one more system to run. Commands vs events: ChargeCard is a command to one owner and is often a sync call; OrderPlaced is a fact others react to and belongs on a topic. A request that must answer now but triggers slow work: save it, publish through an outbox and return 202 Accepted.

Unary (one request, one response), server streaming (one request, many responses), client streaming (many requests, one response) and bidirectional (both sides send independently on one stream). Stream when data is large, open-ended or live; otherwise stay unary. Server streaming: a large export, a feed of price updates, a watch on changes: the client starts reading before the server has finished. Client streaming: an upload or batched telemetry; the server answers once, after the client calls CloseAndRecv. Bidirectional: chat, collaborative editing, a long-lived control channel; reading and writing usually run in separate goroutines. The cost: a stream stays on one backend for its whole life, so balancing is per stream, not per message. After a disconnect the application must resume it itself, and a deadline covers the whole stream. Unary calls are easier to retry, cache and balance.

The client gets codes.Unknown with your raw text, order 42: sql: no rows in result set. Any error that is not a gRPC status crosses the wire that way. Return status.Errorf(codes.NotFound, ...) so the caller can branch on the code. Only the code, the message and optional details travel. Go's error chain stays on the server, so errors.Is(err, sql.ErrNoRows) on the client is always false. The code is the contract: NotFound means do not retry, Unavailable means try again later, InvalidArgument means fix the request. The raw message leaks internals such as SQL or hostnames. Log the detail and return codes.Internal with a safe message. On the client, status.Code(err) reads the code: codes.OK for nil, codes.Unknown for an error that is not a status.

One per target for the life of the process: create it with grpc.NewClient at startup, share it between goroutines, close it at shutdown. A connection per call pays for DNS, TCP and TLS every time and throws away HTTP/2 multiplexing. A ClientConn is not a socket: it owns a name resolver, a load-balancing policy and the connections under it, and reconnects with backoff by itself. It is safe for concurrent use. Parallel RPCs are streams on the same HTTP/2 connection, so one ClientConn carries thousands of calls at once. grpc.NewClient does no I/O; it connects on the first RPC. grpc.Dial, DialContext and WithBlock are deprecated. What belongs to each call is a deadline, not a connection.

An API gateway is the single entry point for external clients (routing, auth, rate limits, TLS), service discovery tells a service where healthy instances of another one are, and a service mesh adds a proxy next to every service for mTLS, retries, timeouts, balancing and telemetry. Gateway: Kong, Envoy Gateway or a cloud gateway; it can also aggregate calls and translate REST to gRPC. It is the edge, not a place for business logic. Discovery: in Kubernetes, Service DNS names and the endpoints API; outside it, Consul or etcd. Client-side discovery (gRPC's dns:/// resolver, xDS) picks an instance itself; server-side puts a load balancer in between. Mesh: Istio or Linkerd: service identity and mTLS without certificates in your code, uniform retries and timeouts, per-request L7 balancing, metrics and traces for every hop. Istio's ambient mode replaces the per-pod sidecar with a per-node L4 proxy and adds an optional waypoint proxy for L7. What stays in your code: deadlines and ctx propagation, idempotency, which errors are safe to retry, trace context, business fallbacks. A mesh cannot know whether Charge may be replayed. The cost: another moving part, extra latency on each hop, and retry or timeout settings that can drift from what the code assumes.

On the wire a field is identified only by its number, so adding or renaming fields is safe on the binary wire, while changing a number or a type, or reusing a deleted number, silently breaks old peers. reserved blocks the reuse; optional brings back field presence. Compatible and breaking changes Safe: a new field with a new number, a renamed field (binary wire only: it changes the JSON name), a new RPC or enum value. Old readers skip unknown fields; new readers see zero values for missing ones. Breaking: renumbering or retyping a field, reusing a number, renaming the package or service (it changes the method path /orders.v1.Orders/GetOrder). Reuse is the silent one: bool and int64 are both varints, so an old is_trial: true decodes as trial_days: 1 with no error. Add reserved 4; reserved "is_trial"; in the same commit that deletes the field. Run buf breaking in CI; ship a truly breaking change as a new package, orders.v2, next to v1. Field presence A plain proto3 scalar does not record whether it was set: an absent bool and an explicit false both decode as false, so an update cannot tell "leave it" from "clear it". optional bool notify_email = 3; generates a *bool in Go, where nil means not sent. For partial updates of many fields, a google.protobuf.FieldMask lists the paths the caller means to write. Protobuf Editions (edition = "2023") make explicit presence the default.

An HTTP client timeout is only a local timer: the server is told nothing about the budget and notices only when the connection is cut. A gRPC deadline travels with the call in the grpc-timeout header, so the handler's ctx carries the same deadline and is cancelled when it passes. With HTTP the server sees a cancelled r.Context() only after the client has aborted; with gRPC it knows the remaining time up front, so it can refuse early or pass the budget downstream. The client gets codes.DeadlineExceeded. On the server ctx.Done() closes and ctx.Err() returns context.DeadlineExceeded, but the handler keeps running unless it passes ctx to the database and to its own outbound calls. codes.Canceled is the other case: the client cancelled or disconnected before the deadline. gRPC sets no deadline by default, so a call without one can wait forever on a stuck server. Every client call needs context.WithTimeout. Pass the handler's ctx to outbound RPCs and the remaining time propagates hop by hop; a fresh context.Background() breaks the chain. If too little time is left to finish, fail at once instead of starting work nobody will read.

An interceptor is gRPC middleware: a function wrapped around every call for auth, logging, metrics or recovery. Unary and streaming calls go through two separate chains, so an interceptor registered only with grpc.ChainUnaryInterceptor never sees a stream. The server has grpc.UnaryServerInterceptor and grpc.StreamServerInterceptor, and the client has a matching pair. Register both, or a whole call type runs without auth. Request headers arrive as metadata: metadata.FromIncomingContext(ctx) on the server, metadata.AppendToOutgoingContext on the client. Keys are lower-case. grpc.ServerStream exposes its context read-only, so a stream interceptor that adds a value passes the handler a wrapper whose Context() returns the new one. A stream interceptor runs once per stream, not per message; per-message work goes in an overridden RecvMsg/SendMsg. Use ChainUnaryInterceptor; setting grpc.UnaryInterceptor twice panics. Tracing with OpenTelemetry is now a stats handler: grpc.StatsHandler(otelgrpc.NewServerHandler()).

req.Page is a message field, so it is a nil pointer when the client leaves it unset, and req.Page.Size panics. grpc-go does not recover panics in handlers, so one such request crashes the whole process and every call in flight on it. Why it is nil: in generated Go code every message-typed field is a pointer (Page *Page), and a field the client did not send stays nil. Scalars are safe: an unset customer_id is just "". Why the pod dies: net/http recovers a handler panic, logs it and drops only that connection. grpc-go runs the handler without recover, so the panic exits the process and the other callers get Unavailable. The fix: read through the generated getters. GetPage() on a nil request returns nil, and GetSize() on a nil *Page returns 0, so req.GetPage().GetSize() never panics and an unset page falls back to 20. Defence in depth: add a recovery interceptor (recovery.UnaryServerInterceptor from go-grpc-middleware v2) that logs the stack and returns codes.Internal, so the next missed check fails one call instead of the pod. How to catch it: the protogetter linter in golangci-lint flags direct field reads on proto messages, and a handler test with &pb.ListOrdersRequest{} reproduces the crash in one line.

io.EOF from Recv is the normal end of a stream, not a failure; and returning early without cancelling the stream's context leaves the stream alive. Check for io.EOF, and derive a cancellable context with defer cancel(). A stream ends with io.EOF when the server's handler returned nil. Any other error carries the status the server sent. grpc-go releases a client stream only when its context is cancelled, Recv returns an error, or the ClientConn is closed. Returning after the first match does none of these, so a goroutine and the HTTP/2 stream stay behind on every call. Cancelling also tells the server to stop: its stream.Context() is cancelled and its next Send fails, instead of streaming results nobody reads.

gRPC retries a failed call by itself when the service config has a retryPolicy for the method: up to maxAttempts tries with backoff, only for the status codes you list. Enable it only for idempotent methods, because Unavailable does not prove the server did nothing. **retryPolicy vs transparent retry**: transparent retry needs no config and resends only a call that never reached the handler. retryPolicy is explicit, set per method, and also retries after the server has started working. **maxAttempts**: the client clamps it to 5. The call's deadline covers all attempts together, so DeadlineExceeded is pointless to list. **retryableStatusCodes**: for writes only UNAVAILABLE; reads may add codes such as RESOURCE_EXHAUSTED. Never INVALID_ARGUMENT or NOT_FOUND: the same request fails the same way. **retryThrottling: a token bucket per server: failures drain it, and below half retries stop, so an overloaded server is not buried. hedgingPolicy**: sends the next attempt after a delay without waiting for a failure; for latency-critical idempotent reads, and not together with retryPolicy on one method. Idempotency first: the charge may have gone through and only the reply been lost, so the caller sends one idempotency key per logical charge and the server dedupes on it.

A circuit breaker wraps calls to one dependency and stops calling it once failures cross a threshold, failing fast instead of waiting on timeouts. It goes from closed to open on failures, to half-open after a cool-down to let a few trial calls through, and back to closed if they succeed. The states Closed: calls pass through and their outcomes are counted. Open: every call fails at once (gobreaker.ErrOpenState); the caller serves a fallback such as a cached value, a default or a degraded response. Half-open: after Timeout, up to MaxRequests trial calls go through; success closes the breaker, a failure reopens it. When it should trip On a failure ratio over a minimum number of requests, not on one error: "over 50% of at least 20 calls" ignores noise at low traffic. Count only failures that say the dependency is unhealthy: Unavailable, DeadlineExceeded, ResourceExhausted. NotFound or InvalidArgument means the dependency is working. One breaker per dependency, or per host; never one shared by everything. Why it matters It gives the struggling dependency room to recover and stops your goroutines and connections piling up on calls that would fail anyway. It complements deadlines and retry budgets; a mesh can do the same outside the code (Istio outlier detection).

failures is never reset on success, so it counts every error since the process started. After the fifth one, however far apart, each single failure opens the breaker for 30 s and rejects calls to a healthy dependency. Why: failures >= 5 is meant as "five failures in a row", but nothing ever lowers the counter. In a test loop with one failure per 1000 calls that runs inside the 30 s window, this breaker rejected every call after the 5000th; the fixed one rejected none. In production it is a 30 s outage after every later failure. The fix: reset failures to 0 on every success. The breaker now trips only on five consecutive failures, which is what a dead dependency looks like, not on background noise. A better trigger: production breakers use a failure ratio over a rolling window with a minimum request count; gobreaker clears its counts every Interval while closed and decides in ReadyToTrip. Still simplified: after the cool-down this breaker lets every waiting caller through at once. A real half-open state admits a few trial calls (MaxRequests in gobreaker) and closes only when they succeed. How to catch it: a unit test that mixes outcomes: four failures, a success, then one failure must leave the breaker closed. Inject the clock (now func() time.Time) to test the cool-down without sleeping.

kube-proxy balances connections, not requests, and gRPC sends every call as a stream on one long-lived HTTP/2 connection, so the pod that got the connection gets the traffic. Balance per request instead: in the client or in an L7 proxy. REST only looks balanced because HTTP/1.1 clients keep a pool of connections and keep replacing them. In the client: a headless Service (clusterIP: None) makes DNS return every pod IP. Dial dns:///name:port and set the round_robin policy in the service config; the default pick_first uses one address. In a proxy: a service mesh (Istio, Linkerd) or Envoy terminates HTTP/2 and balances each request, with no client changes. New pods: the client re-resolves DNS only when connections change, so a scaled-up fleet gets no traffic. Set MaxConnectionAge on the server: it sends GOAWAY periodically and clients reconnect to the current set of pods.

Retries multiply load exactly when capacity is lowest (three attempts at each of three hops is up to 27 times the traffic), and 30 s timeouts keep every stuck request holding a goroutine and a pooled connection until the shared caller runs out of goroutines and connections for everyone. One deadline: set at the edge and propagated through ctx; a retry spends what is left of it, not a fresh 30 s. Retry at one layer only, under a budget: for example retries capped at 10% of requests (gRPC retryThrottling), with exponential backoff and jitter. Circuit breaker per dependency, to fail fast while it is down. Bulkhead: a bounded number of concurrent calls per dependency (a buffered channel used as a semaphore, in a client interceptor), so one slow dependency cannot take every goroutine and connection. Load shedding: on the server, a rate.Limiter Allow() check or a concurrency cap; when full, reject at once with ResourceExhausted or 503 instead of queueing; a queued request whose caller already gave up is pure waste.

Two writes to two systems cannot be made atomic by reordering them. Use the transactional outbox: insert the event into an outbox table in the same database transaction as the order, and let a separate relay publish it to the broker. Commit-then-publish loses the event if the process dies or Kafka is down in between; publish-then-commit emits an event for an order that rolled back. With the outbox the event exists exactly when the order does, because both are one local transaction. The relay selects unpublished rows with FOR UPDATE SKIP LOCKED, publishes them, then marks them. A crash between the two republishes, so delivery is at-least-once and consumers dedupe on the outbox id. Or let change data capture be the relay: Debezium reads the database WAL and publishes outbox rows without polling. Key messages by the aggregate id to keep per-order ordering, and delete published rows with a cleanup job.

A saga is a sequence of local transactions, each with a compensating action; if a step fails, the finished ones are compensated in reverse order. It replaces two-phase commit, which holds locks across services, at the price of isolation. Why not two-phase commit Every participant holds its locks until the slowest one votes, and a coordinator that crashes mid-commit leaves them all blocked. Kafka and external APIs such as a card processor cannot take part at all. Orchestration vs choreography Orchestration: a coordinator calls each step and its compensation: a state machine persisted in a database, or a workflow engine such as Temporal. The flow lives in one place and is easy to monitor. Choreography: each service reacts to the previous event (StockReserved triggers the charge). Nothing central, but the flow is spread across services and hard to follow beyond three or four steps. What it costs Others see intermediate states, so model them explicitly: PENDING, PAYMENT_FAILED. Compensations are new transactions (a refund, a stock release) and must be idempotent and retried until they succeed. Order steps so the hard-to-undo ones come last, and know the pivot step: after it the saga can no longer roll back and must retry forward until it succeeds. Step messages arrive at least once, so every step is idempotent, and step events go out through an outbox.

Acquire with SET key token NX PX ttl; release with a script that deletes the key only if it still holds your token. The lock can expire while its holder is paused, so two holders act at once; a fencing token (growing with every acquisition, checked by the resource) rejects the stale one. Why it breaks: a GC pause, a slow call or a VM freeze longer than the TTL: the lock expires, another worker takes it, and the first one wakes up and writes anyway. Nothing on the lock side fixes this; only the resource can refuse the old holder. Fencing: the storage keeps the highest token it has seen and rejects writes that carry a lower one. Where tokens come from: etcd and ZooKeeper give a monotonically growing revision with every lock. Redis has none built in; an INCR counter bumped in the same Lua script that takes the lock is the usual workaround, and Redlock across several nodes does not add fencing either. Releasing: a plain DEL can delete the next owner's lock after yours expired; compare the token and delete in one Lua script. Simpler options: if the resource is Postgres, pg_try_advisory_lock or SELECT ... FOR UPDATE SKIP LOCKED avoids a second system. Leader election has the same problem: use etcd's concurrency.Election or a Kubernetes Lease, and still fence the writes.