Irbisa · Cheatsheet

Go in Production

Multi-stage Docker builds, config and secrets, graceful shutdown, health and readiness, GOMAXPROCS in containers, migrations

Middle Developer · 21 questions

  1. 01

    What is a container, how does it differ from a virtual machine, and what do namespaces and cgroups give it?

    A container is an ordinary Linux process that the kernel isolates: namespaces give it its own view of the system, and cgroups limit how much CPU and memory it may use.

  2. 02

    What do a Pod, a Deployment, a Service and a ConfigMap each do, and how does a Go service get from an image to receiving traffic in Kubernetes?

    A Pod runs one or more containers together; a Deployment keeps N Pods of one version running and rolls them to the next; a Service gives those Pods one stable name and address and sends traffic only to the Ready ones; a ConfigMap holds configuration that becomes env vars or files.

    apiVersion: apps/v1kind: Deploymentmetadata:  name: ordersspec:  replicas: 3 # keep three Pods of this version running…
  3. 03

    How should a Go service get its configuration, what does the 12-factor app say about it, and what happens when a required value is missing?

    Read it from the environment once at startup into one typed Config struct, with defaults for safe values and validation for the rest.

    type Config struct {	Addr            string	DatabaseURL     string // a secret: never log it	ShutdownTimeout time.Duration}…
  4. 04

    What is the difference between SIGTERM, SIGINT and SIGKILL, and which of them can a Go program catch?

    SIGTERM is the polite "please stop" that docker stop and Kubernetes send, SIGINT is Ctrl+C, and both can be caught with signal.NotifyContext.

  5. 05

    Why should a Go container run as a non-root user, and how do you set that up on scratch or distroless?

    Without user namespaces, UID 0 in a container is UID 0 on the host, held back only by dropped capabilities and seccomp, so a compromised root process is one kernel or runtime bug from the node.

    # Dockerfile: USER 65532:65532 (numeric, so runAsNonRoot can verify it)spec:  securityContext:    runAsNonRoot: true    runAsUser: 65532    runAsGroup: 65532…
  6. 06

    The Alpine-based container starts via entrypoint.sh, which reads a secret and runs /app/server. Pods now take the full 30 s to stop and never log a shutdown. Why?

    The script is PID 1 and the Go binary is its child, so the kubelet's SIGTERM goes to sh, which neither forwards it nor, as PID 1 without a handler, even dies of it.

    #!/bin/sh# entrypoint.shset -euDB_PASSWORD="$(cat /run/secrets/db-password)"export DB_PASSWORD…
  7. 07

    Why must secrets never be baked into a container image, and how should a Go service get them instead, at build time and at runtime?

    Every layer of an image is readable by anyone who can pull it, and deleting a file or overriding a variable in a later layer does not remove it from the earlier one.

    # syntax=docker/dockerfile:1FROM golang:1.25 AS buildWORKDIR /srcENV GOPRIVATE=github.com/acme/*COPY go.mod go.sum ./# wrong: ARG GITHUB_TOKEN, then RUN git config ... $GITHUB_TOKEN…
  8. 08

    What does -ldflags="-s -w" strip from a Go binary, and what still works afterwards: panic traces, pprof, Delve?

    -s drops the ELF symbol table and -w the DWARF debug info, which typically cuts the binary by about 30%.

  9. 09

    Find the bug in this graceful shutdown: on SIGTERM, requests that are in flight still get cut off.

    func run(h http.Handler, db *sql.DB) error {	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)	defer stop()	srv := &http.Server{Addr: ":8080", Handler: h, ReadHeaderTimeout: 5 * time.Second}	go func() {		<-ctx.Done()		sctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)		defer cancel()		srv.Shutdown(sctx)	}() 	if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {		return err	}	return db.Close()}

    ListenAndServe returns ErrServerClosed the moment Shutdown closes the listener, not when the drain ends.

  10. 10

    Find the bug: pods are killed with exit code 137 in the middle of a rolling update, and the Shutdown drain never finishes.

    spec:  terminationGracePeriodSeconds: 30  containers:    - name: api      image: registry.example.com/api:1.5.0      env:        - name: SHUTDOWN_TIMEOUT # the context deadline passed to srv.Shutdown          value: "25s"      lifecycle:        preStop:          sleep:            seconds: 30 # wait until the load balancer stops sending traffic

    The pod has 30 s in total, but the preStop sleep alone takes 30 s and Shutdown is allowed another 25 s.

  11. 11

    How do you expose pprof on a production service safely, and what can an attacker do with an open /debug/pprof/?

    Serve it on a separate admin listener that no Service, Ingress or load balancer exposes; loopback plus kubectl port-forward is the simplest.

    import "net/http/pprof" // its init also registers on http.DefaultServeMux func adminServer() *http.Server {	mux := http.NewServeMux()	mux.HandleFunc("/debug/pprof/", pprof.Index)	mux.HandleFunc("/debug/pprof/profile", pprof.Profile)…
  12. 12

    A pod restarts with OOMKilled and exit code 137, and the Go logs show nothing. What happened, and how is that different from a Go out of memory crash?

    The container's memory reached its cgroup limit and the kernel's OOM killer sent SIGKILL: no panic, no deferred calls, no last log line.

  13. 13

    How does the Horizontal Pod Autoscaler decide how many pods to run, and why is CPU a poor scaling signal for an I/O-bound Go API or a Kafka consumer?

    Every 15 s the HPA computes ceil(replicas × current / target) per metric and scales to the largest result, within minReplicas and maxReplicas.

    apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata:  name: apispec:  scaleTargetRef:…
  14. 14

    What should the CI pipeline of a Go service run, in what order, and what keeps it fast (module and build caches, test sharding)?

    Cheap checks first, then tests, then the artifact: go mod tidy -diff, go vet and the linter in seconds; then go test -race, integration tests and govulncheck in parallel; then one image per commit, tagged with its SHA.

    on:  pull_request:  push:    branches: [main]jobs:  checks: # seconds: fail before anything slow starts…
  15. 15

    What is the difference between rolling, blue-green and canary deployments, and what must a Go service support for any of them to have zero downtime?

    Rolling replaces pods a few at a time, blue-green brings up a full second copy and switches all traffic at once, and canary sends a small share of traffic to the new version and widens it while metrics stay healthy.

  16. 16

    Should database migrations run when the service starts or as a separate step of the deploy, and what goes wrong with each?

    In production, as a separate step before the rollout (a CI job or a Kubernetes Job from the same image), so one process applies them once, with its own timeout and DDL rights.

  17. 17

    Ten replicas each run the same hourly time.Ticker job, so the report goes out ten times. How do you run periodic work exactly once?

    Take the schedule out of the replicas, or make them agree on who runs: a Kubernetes CronJob starts one pod per run, or every replica tries to claim the run in the database and only one succeeds.

    // Every replica ticks once a minute and calls hourly. The first one to claim// the hour runs the job; the other nine see the claim and skip.func hourly(ctx context.Context, db *sql.DB, job string, run func(context.Context, time.Time) error) error {	period := time.Now().UTC().Truncate(time.Hour) // the run's identity: "the 13:00 report"	res, err := db.ExecContext(ctx, `		INSERT INTO job_runs (job, period) VALUES ($1, $2)…
  18. 18

    How does readiness gate a rolling update, and how do you make a broken release stop after its first pods instead of replacing the whole fleet?

    A rolling update removes more old pods only after new ones are Ready and stay Ready for minReadySeconds, so readiness is the rollout's brake.

    apiVersion: apps/v1kind: Deploymentmetadata:  name: apispec:  replicas: 10…
  19. 19

    Behind a load balancer, a Go service causes rare 502s at any load, and none of them appear in its own logs. How can IdleTimeout cause this, and what is the fix?

    The load balancer keeps idle keep-alive connections to the backend and reuses them.

  20. 20

    How do you set CPU and memory requests and limits for a Go service in Kubernetes, and what happens to GOMAXPROCS and GOMEMLIMIT if you drop the CPU limit?

    Memory: limit equal to the request, with GOMEMLIMIT at about 90% of it.

    containers:  - name: api    image: registry.example.com/api:1.5.0    resources:      requests:        cpu: "2"…
  21. 21

    A service runs an HTTP API, a Kafka consumer, background workers and a DB pool. On SIGTERM, in what order do you shut them down, and why?

    Stop taking new work, finish the work already taken, and close what that work depends on last: readiness to 503, wait for endpoints to update, drain the HTTP server and the consumer in parallel, then workers, then producers, then the DB pool, and telemetry at the very end.

    // Shutdown(ctx) stops taking new work and waits for work already taken.type drainer interface{ Shutdown(ctx context.Context) error } func drain(srv *http.Server, consumer, workers, producer, telemetry drainer, db *sql.DB) error {	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)	defer cancel()…