Go in Production

A container is an ordinary Linux process that the kernel isolates: namespaces give it its own view of the system, and cgroups limit how much CPU and memory it may use. A virtual machine boots a whole guest kernel on virtual hardware; all containers on a host share the host's kernel. Namespaces decide what it sees: its own root filesystem unpacked from the image (mnt), its own process tree in which it is PID 1 (pid), its own interfaces, IP and ports (net), hostname (uts) and user ids (user). cgroups decide how much it gets: cpu.max (the quota that Kubernetes limits.cpu sets and Go 1.25 reads for GOMAXPROCS), memory.max (past it the OOM killer, exit code 137), and limits on processes and I/O. Image vs container: the image is read-only filesystem layers; the container is a process on top of them with a thin writable layer that dies with it. The trade-off: a container starts in milliseconds and costs megabytes; a VM boots in seconds and reserves gigabytes. The price is the shared kernel: one kernel bug reaches everything on the host, so untrusted tenants get VMs, microVMs (Firecracker) or gVisor. On macOS, Docker Desktop runs a small Linux VM, because Linux containers need a Linux kernel.

A Pod runs one or more containers together; a Deployment keeps N Pods of one version running and rolls them to the next; a Service gives those Pods one stable name and address and sends traffic only to the Ready ones; a ConfigMap holds configuration that becomes env vars or files. From image to traffic: CI pushes an image with an immutable tag; kubectl apply (or Helm, Argo CD) submits the Deployment, which creates a ReplicaSet, which creates the Pods. The scheduler picks a node with room for each Pod's requests, and that node's kubelet pulls the image and starts it. Ready means traffic: once the readiness probe passes, the Pod's IP joins the Service's EndpointSlices and kube-proxy routes the Service's address to it. A Pod that is not Ready gets nothing. From outside: an Ingress or a Gateway API route (or a LoadBalancer Service) brings external requests to the Service. Pods are disposable: every restart gets a new IP, so callers use the Service name (http://orders inside the namespace, orders.shop.svc.cluster.local from anywhere). Neighbours: Secret (a ConfigMap for credentials), Job and CronJob (run to completion), HorizontalPodAutoscaler (changes replicas).

Read it from the environment once at startup into one typed Config struct, with defaults for safe values and validation for the rest. A missing or malformed required value stops the process with a clear error before it serves a single request. 12-factor: config is whatever differs between deploys (addresses, credentials, limits), and it lives in the environment, not in the code or the image. The same image then runs in staging and production. Env or flags: environment variables are what Docker, Kubernetes and CI set easily; flags suit CLI tools. Services often accept both, with the flag winning. Fail fast: exit with code 1 and list every problem at once (errors.Join). A pod in CrashLoopBackOff with a clear log line beats a service that fails on its first request at 3 am. **Parse once in main**: pass Config down. Packages that call os.Getenv themselves hide their inputs and are hard to test. Libraries: caarlos0/env or kelseyhightower/envconfig map variables to struct tags; Viper adds files and remote sources at the cost of size and magic. Never log the whole config: it holds secrets.

SIGTERM is the polite "please stop" that docker stop and Kubernetes send, SIGINT is Ctrl+C, and both can be caught with signal.NotifyContext. SIGKILL cannot be caught or ignored: the kernel ends the process at once, and no defer runs. Go's default: without signal.Notify, SIGTERM and SIGINT end the program immediately: no deferred calls, no Shutdown, in-flight requests are cut. Catching: signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM) returns a context that is cancelled on the first signal. Call stop() once it fires, so a second Ctrl+C kills the process the default way. Who sends what: docker stop sends SIGTERM, waits 10 s, then SIGKILL. Kubernetes runs the preStop hook, sends SIGTERM, and after terminationGracePeriodSeconds (30 s by default) sends SIGKILL. Exit codes: a process killed by a signal reports 128 + its number: 137 for SIGKILL (an OOM kill looks the same), 143 for an unhandled SIGTERM. A Go panic exits with 2.

Without user namespaces, UID 0 in a container is UID 0 on the host, held back only by dropped capabilities and seccomp, so a compromised root process is one kernel or runtime bug from the node. A Go binary needs no privileges: set a numeric USER in the image and enforce it with runAsNonRoot. **On scratch**: USER 65534:65534 (nobody). With no /etc/passwd, only a numeric UID works. On distroless: use the :nonroot tag (UID 65532), and write USER 65532:65532 rather than USER nonroot. With runAsNonRoot: true the kubelet refuses a named user (image has non-numeric user (nonroot), cannot verify user is non-root) unless the pod sets runAsUser. Ports: listen on 8080, not 80: ports below 1024 need CAP_NET_BIND_SERVICE on many runtimes, and the Service can still expose 80. The rest of the hardening: readOnlyRootFilesystem: true with an emptyDir for temp files, allowPrivilegeEscalation: false, and all capabilities dropped.

The script is PID 1 and the Go binary is its child, so the kubelet's SIGTERM goes to sh, which neither forwards it nor, as PID 1 without a handler, even dies of it. Go never sees the signal, and after 30 s everything gets SIGKILL. PID 1 is special: the kernel delivers a signal to a namespace's init process only if it installed a handler for it. sh waiting on a child has none, so SIGTERM is dropped. Go always installs handlers, so as PID 1 it does receive it. The fix: end the script with exec /app/server "$@". exec replaces the shell with the binary in the same PID 1, and the signal lands in signal.NotifyContext. Shell form has the same trap: ENTRYPOINT /app/server --port=8080 runs under /bin/sh -c; some shells exec a lone command, some do not. Use the exec form ENTRYPOINT ["/app/server", "--port=8080"], which on distroless is the only one that works at all: there is no /bin/sh. Symptoms: every rollout waits the full grace period per pod, the exit code is 137, and in-flight requests are cut because the drain never ran. Child processes: if the service spawns subprocesses, add a tiny init (tini, or docker run --init) that forwards signals and reaps zombies.

Every layer of an image is readable by anyone who can pull it, and deleting a file or overriding a variable in a later layer does not remove it from the earlier one. Build-time secrets go through BuildKit secret mounts; runtime secrets come from the platform as env vars or mounted files. How they leak: ENV DB_PASSWORD=... sits in the image config (docker inspect), an ARG used by a RUN shows up in docker history, a copied .env stays in its layer after rm. And the image is pushed to a registry, cached in CI and pulled onto every node. Build time: usually a token for private modules. RUN --mount=type=secret,id=netrc,target=/root/.netrc go mod download exposes it to that one command and writes it into no layer. Runtime: a Kubernetes Secret, ideally synced from Vault or a cloud secret manager, injected as an env var or a file. The same image then runs in every environment. Env or file: env is simplest, but child processes inherit it, /proc/<pid>/environ shows it, and it changes only on restart. A mounted file can have tighter permissions and is updated in place when the secret rotates (not with subPath). In Go: load once at startup, keep it out of logs and error messages, and never pass it as a command-line flag: ps and /debug/pprof/cmdline show flags.

-s drops the ELF symbol table and -w the DWARF debug info, which typically cuts the binary by about 30%. Panic traces and pprof keep working because they rely on Go's own pclntab, which stays; what you lose is Delve, readable core dumps and native tools like perf. Kept: pclntab maps program counters to functions, files and lines. Panic traces, runtime.Callers, pprof profiles (symbolised inside the process) and the build info that go version -m reads all survive. Lost: Delve cannot set breakpoints or show variables without DWARF, a core dump is close to unreadable, and perf and other ELF-symbol tools show raw addresses. Go 1.22+: -s alone implies -w; -s -w=0 keeps DWARF without the symbol table. **-trimpath is a different job**: it replaces absolute build paths (/home/runner/work/...) with module paths in traces and makes the build reproducible; it does not shrink anything noticeably. The trade-off: a few MB per image barely matters for pulls. Teams that debug production with Delve or analyse core dumps skip -w, or keep an unstripped build of the same commit as a CI artifact.

ListenAndServe returns ErrServerClosed the moment Shutdown closes the listener, not when the drain ends. So run closes the DB pool and main exits while handlers are still running: in-flight requests fail with sql: database is closed or die with the process. Why: only Shutdown returning means "drained", and here nothing waits for it: it runs in a goroutine whose result is dropped. The fix: send Shutdown's result to a channel and block on it after ListenAndServe returns; close the DB only after that. Timeout: if the 25 s budget runs out, Shutdown returns context.DeadlineExceeded with handlers still running; srv.Close() cuts them, then the DB closes. The ignored error: a bare srv.Shutdown(sctx) hides a drain that timed out; errcheck in golangci-lint flags it. Budget: 25 s plus any preStop sleep must stay below terminationGracePeriodSeconds, or SIGKILL arrives mid-drain.

The pod has 30 s in total, but the preStop sleep alone takes 30 s and Shutdown is allowed another 25 s. The kubelet sends SIGKILL (exit code 137) while requests are still draining, so they are cut and the shutdown log line never appears. One clock: terminationGracePeriodSeconds starts when the pod becomes Terminating and covers the preStop hook and the drain after SIGTERM. It is not restarted after preStop; if the hook uses it all, the container gets only a couple of seconds after SIGTERM. The budget: preStop + the Shutdown timeout + a margin must fit in the grace period: 10 + 25 + 10 = 45 s. How it creeps in: the default 30 s is enough for a plain drain. The bug appears when someone adds a preStop sleep or raises the shutdown timeout and forgets the grace period. **preStop length**: it only has to outlast the endpoint removal reaching the balancer, usually 5-15 s. The native sleep handler (beta since Kubernetes 1.30, GA in 1.34) also works on distroless, which has no sleep binary.

Serve it on a separate admin listener that no Service, Ingress or load balancer exposes; loopback plus kubectl port-forward is the simplest. Left open to the internet, it leaks the command line and the shape of your code and gives anyone a cheap way to stall the process. What leaks: /debug/pprof/cmdline returns os.Args, secrets passed as flags included. goroutine?debug=2 dumps every stack with function names and file paths, and heap and CPU profiles reveal the code layout and dependencies. What it costs: goroutine?debug=2 stops the world while it formats every stack; with 100,000 goroutines that is a visible pause, and nothing stops someone from requesting it in a loop. The usual trap: import _ "net/http/pprof" registers its handlers on http.DefaultServeMux, so any server started with a nil handler, or a router that falls back to the default mux, serves them publicly. The admin port: a second http.Server with its own mux (pprof, /metrics) on 127.0.0.1 or on a port the Service does not list; a NetworkPolicy limits who can reach it inside the cluster. If it must be remote: put it behind auth middleware, or let a continuous profiler (Pyroscope, Parca, a vendor agent) collect profiles so people never hit the endpoint directly.

The container's memory reached its cgroup limit and the kernel's OOM killer sent SIGKILL: no panic, no deferred calls, no last log line. Go's own fatal error: runtime: out of memory is rare in a container, because the kernel kills the process before an allocation can fail. Confirm it: kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137. An eviction for node memory pressure shows Evicted instead. What is counted: the cgroup charges the Go heap and stacks, plus memory Go does not manage: cgo allocations and files in a medium: Memory emptyDir. Compare container_memory_working_set_bytes with the limit, not just the heap. Why Go grew past it: with GOGC=100 the GC lets the heap reach about twice the live data and knows nothing about the container limit; a burst of concurrent requests or a bigger live set pushes the next heap goal over it. What to do: set GOMEMLIMIT below the limit (see the runtime topic), bound concurrency so a burst cannot hold unbounded buffers, and take a heap profile before the next kill to rule out a leak. Other exit codes: 2 is a Go panic or fatal runtime error (read it with kubectl logs --previous), 1 is usually log.Fatal or os.Exit(1), 143 an unhandled SIGTERM.

Every 15 s the HPA computes ceil(replicas × current / target) per metric and scales to the largest result, within minReplicas and maxReplicas. CPU works only when CPU is the bottleneck: an API waiting on its database, or a consumer behind on a topic, can be saturated at 20% CPU. The arithmetic: 4 pods at 140% against a 70% target ask for ceil(4 × 140/70) = 8; a ratio within 10% of 1.0 is ignored. Scale-up is immediate, scale-down takes the highest recommendation of the last 5 minutes, so a dip does not remove pods the next spike needs. **Utilisation is relative to requests.cpu: a pod without a CPU request gives the HPA nothing to compute, and the CPU of pods that are still starting is set aside. I/O-bound APIs: goroutines waiting on Postgres burn no CPU, so latency climbs, CPU stays flat and the HPA does nothing. Scale on in-flight requests or requests per second per pod, served to the HPA from Prometheus by a metrics adapter. More pods can make it worse: when the database is the bottleneck, every new pod adds a connection pool, and scaling on latency adds pods exactly then. Kafka consumers: the signal is consumer lag, which KEDA reads directly. Consumers beyond the partition count sit idle, and KEDA's Kafka scaler stops at that count by default. Memory is no better**: a Go pod's memory falls slowly after load spreads out (the runtime returns memory to the OS gradually), so a memory-based HPA scales up and seldom back down.

Cheap checks first, then tests, then the artifact: go mod tidy -diff, go vet and the linter in seconds; then go test -race, integration tests and govulncheck in parallel; then one image per commit, tagged with its SHA. Restored module and build caches and sharded tests keep it fast. Fail fast: a lint failure should cost 30 seconds, not wait behind a ten-minute integration suite; the image job waits for all the others. What each tool checks is in the tooling topic. Build once, promote: the image built from the commit goes to staging and then to production; rebuilding per environment ships a binary nobody tested. Module cache: GOMODCACHE saves the downloads. actions/setup-go restores it, but since v6 its default cache key is go.mod; point cache-dependency-path at go.sum. Build cache: GOCACHE holds compiled packages and test results, so unchanged packages are neither rebuilt nor retested (-race and -coverprofile runs are cached too; -count=1 and -shuffle disable it). setup-go saves it only when the key changes, so it ages between dependency updates. Docker builds: BuildKit's --mount=type=cache is not part of the exported layer cache, so on a fresh runner it starts empty. Building the binary with the restored GOCACHE and copying it into the image avoids that. Sharding: go test already runs packages in parallel on one machine. When one suite dominates, split the package list across runners, and keep integration tests behind a build tag.

Rolling replaces pods a few at a time, blue-green brings up a full second copy and switches all traffic at once, and canary sends a small share of traffic to the new version and widens it while metrics stay healthy. All three run old and new versions side by side, and the service must survive that. The strategies Rolling: the Kubernetes Deployment default (maxSurge and maxUnavailable at 25%). Cheap and automatic; a bad version reaches every pod unless readiness or a human stops it. Blue-green: two full environments, and the switch is a Service selector or load balancer change. Rollback is the same flip back, at the price of double capacity during the switch. Canary: 1–5% of traffic first, compared on error rate and latency, then promoted step by step. Needs traffic splitting (Argo Rollouts, Flagger, a mesh or ingress weights) and metrics you trust. What the service needs Two versions at once: APIs, message formats and the schema must work for N and N-1 together: add a field before using it, keep the old one until nobody sends it, expand/contract migrations. Clean entry and exit: readiness that turns green only when the instance can serve, and a graceful drain on SIGTERM. Idempotent consumers: during the switch a message can be half-handled by an old pod and redelivered to a new one. Feature flags: separate deploy from release: ship the code dark, turn it on later, turn it off without a rollback.

In production, as a separate step before the rollout (a CI job or a Kubernetes Job from the same image), so one process applies them once, with its own timeout and DDL rights. On startup is fine for a single instance, but with replicas it races and ties a failed migration to a crash loop. On startup Ten pods start together and all try to migrate. golang-migrate and goose can take a Postgres advisory lock so only one runs, but the other nine wait while their probes tick. A long migration outlives the startup or liveness probe: the kubelet kills the pod mid-ALTER, the next pod starts over, and golang-migrate leaves the version marked dirty until someone runs force. The app's database user needs DDL rights it otherwise should not have. As a deploy step A Helm pre-upgrade hook, an Argo CD PreSync hook or a pipeline stage runs app migrate or the tool's CLI, and the rollout starts only if it succeeded. The new schema then runs under the old code for a while, so every migration must be backward compatible: expand before the code that needs it, contract releases later (see the databases topic).

Take the schedule out of the replicas, or make them agree on who runs: a Kubernetes CronJob starts one pod per run, or every replica tries to claim the run in the database and only one succeeds. Either way the job must be idempotent, because no scheduler promises exactly once. **CronJob**: the job becomes a command of the same image (app send-report), scheduled on wall-clock time with timeZone; concurrencyPolicy: Forbid stops runs from overlapping. Kubernetes documents that it creates a Job "approximately once" per slot: rarely two, rarely none. Claim the period: every replica ticks each minute and inserts (job, hour) with ON CONFLICT DO NOTHING; the one whose insert went through runs the job. Claiming the hour, not just holding a lock while running, is what stops a late replica from running the same hour again after the first one finished. A crash mid-run leaves the hour claimed and unfinished. Record started_at and done_at, and let a later tick take over a claim older than the job's timeout with no done_at. Leader election: one replica holds a Kubernetes Lease (client-go's leaderelection) and runs every schedule; a deposed leader may still be finishing, so the work stays idempotent (see the distributed lock question). river on Postgres schedules periodic jobs from its elected leader. Idempotent work: record "report 13:00 sent" in the same transaction, or pass that key to the email provider, so a second run is a no-op. **time.Ticker is the wrong clock anyway**: it counts from pod start, and every restart resets it.

A rolling update removes more old pods only after new ones are Ready and stay Ready for minReadySeconds, so readiness is the rollout's brake. It works only if a broken release cannot become Ready: check config and dependencies once during startup, then keep readiness local. How the rollout uses readiness With maxSurge: 2 and maxUnavailable: 0, Kubernetes starts two new pods and removes an old one only when a new one is available. New pods that never get Ready stall the rollout there, while the old pods keep serving. minReadySeconds catches a version that passes the first probe and crashes seconds later. progressDeadlineSeconds marks the rollout ProgressDeadlineExceeded but does not roll back. kubectl rollout status then exits non-zero and the pipeline runs kubectl rollout undo; Argo Rollouts can abort on metrics as well. What makes a bad release fail early A readiness handler that always returns 200 lets a version with a wrong DATABASE_URL or a missing secret reach every pod and fail on real traffic. Run the risky checks during startup: validate config, open the pool and run one query, verify the schema version the code expects. Fail by exiting, or by never flipping started to true. After that, readiness reports local state only. A database blip must not empty the fleet (see the observability topic). What it cannot catch Logic bugs that pass startup and return 500s under real traffic. That is a canary's job: a small share of traffic and an error-rate check before going wider.

The load balancer keeps idle keep-alive connections to the backend and reuses them. If the Go server's IdleTimeout is shorter than the balancer's, the server closes a connection just as the balancer sends a request on it; the request hits a closed socket and the balancer answers 502. Why the logs are empty: the request never reached a handler: the server had closed the connection before reading it. Only the balancer's logs show it (on AWS ALB, a 502 with no target status code). The rule: the side that reuses a connection must give up on it first, so the backend's idle timeout must be longer than the proxy's. AWS ALB defaults to 60 s, so IdleTimeout of 75–120 s; Google Cloud's load balancer keeps backend connections about 600 s, so its backends need more than that. The default trap: with IdleTimeout at zero, Go uses ReadTimeout, so a 5 s ReadTimeout silently closes keep-alive connections after 5 s idle. The same rule outbound: IdleConnTimeout must be shorter than the upstream's idle timeout. http.DefaultTransport sets 90 s, but a hand-built &http.Transport{} has none (0 means never), so reused connections fail with EOF or connection reset, and Go does not retry a non-idempotent request. During deploys: Shutdown closes idle connections too, so the same 502s cluster around rollouts. Deregistering the target at the balancer (deregistration delay, a preStop sleep) before SIGTERM handles that part.

Memory: limit equal to the request, with GOMEMLIMIT at about 90% of it. CPU: many teams drop the limit to avoid CFS throttling, but then Go 1.25 sees no cgroup limit and sets GOMAXPROCS to every core on the node, so set it from the CPU request yourself. Memory Memory is not compressible: going over the limit means SIGKILL, and a request below the limit lets the node overcommit and evict or OOM-kill you under pressure. limits.memory equal to requests.memory avoids both. The runtime does not read the memory limit. Set GOMEMLIMIT (900MiB for 1Gi) so the GC works harder before the kernel kills; the headroom covers memory outside Go's accounting, such as cgo. CPU With a CPU limit, Go 1.25+ (the go line in go.mod must be 1.25 or newer) sets GOMAXPROCS to the limit (rounded up, at least 2) and follows changes to it; before 1.25 that took automaxprocs. The price is throttling whenever a GC cycle or a burst spends the period's quota early. Without a limit, the pod can burst onto idle cores, but GOMAXPROCS equals the node's core count: 64 Ps for a pod that asked for 2, and more GC worker threads fighting for its share when the node is busy. Set GOMAXPROCS from requests.cpu through the downward API; the value is rounded up to a whole number. An explicit value also turns off Go 1.25's periodic updates, which is fine because requests do not change at runtime. Sizing Load-test one pod to find where p99 bends, set the CPU request a little above steady-state use, and let the HPA scale on CPU utilisation, which is measured against requests, not limits.

Stop taking new work, finish the work already taken, and close what that work depends on last: readiness to 503, wait for endpoints to update, drain the HTTP server and the consumer in parallel, then workers, then producers, then the DB pool, and telemetry at the very end. Why this order Each component closes after everything that uses it. Handlers use workers, producers and the DB, so closing the pool first turns a clean drain into a burst of sql: database is closed. The HTTP server and the consumer are independent entry points, so they drain in parallel within one budget. Telemetry goes last, so the drain itself is logged, traced and counted. The consumer Cancel fetching, not handling: one context for the poll loop that SIGTERM cancels, another for processing a message that only the shutdown deadline cancels. Otherwise a message dies halfway through its transaction. Commit offsets for what finished, then close the consumer so it leaves the group at once and its partitions move in seconds, not after the session timeout. What was fetched but not finished is redelivered to another member, which is why handlers must be idempotent. The budget One deadline for the whole drain, below terminationGracePeriodSeconds minus any preStop sleep. Whatever is still running at the deadline is cut, and its messages are delivered again: at-least-once by design.