Docker & Containers

A container is an ordinary Linux process that the kernel isolates: namespaces give it its own view of the system, and cgroups limit how much CPU and memory it may use. A virtual machine boots a whole guest kernel on virtual hardware; all containers on a host share the host's kernel. Namespaces decide what it sees: its own root filesystem unpacked from the image (mnt), its own process tree in which it is PID 1 (pid), its own interfaces, IP and ports (net), hostname (uts) and user ids (user). cgroups decide how much it gets: cpu.max (the quota that Kubernetes limits.cpu sets and Go 1.25 reads for GOMAXPROCS), memory.max (past it the OOM killer, exit code 137), and limits on processes and I/O. Image vs container: the image is read-only filesystem layers; the container is a process on top of them with a thin writable layer that dies with it. The trade-off: a container starts in milliseconds and costs megabytes; a VM boots in seconds and reserves gigabytes. The price is the shared kernel: one kernel bug reaches everything on the host, so untrusted tenants get VMs, microVMs (Firecracker) or gVisor. On macOS, Docker Desktop runs a small Linux VM, because Linux containers need a Linux kernel.

An image is a read-only stack of filesystem layers plus a config (command, environment, user); a container is a running instance of an image with a thin writable layer on top; a registry stores images by name and tag. A layer holds the file changes of one build step. Layers are content-addressed: each one is identified by its sha256, so ten images built FROM the same base store it once, and docker pull and docker push move only the layers the other side lacks. A later layer cannot shrink an earlier one: RUN rm only hides a file, and its bytes stay in the layer that added it. That is why the Go toolchain lives in a build stage and never reaches the final image. Containers are disposable: the writable layer survives docker stop and docker start but is deleted by docker rm, so data that must outlive the container goes into a volume. Any number of containers can run from one image at once. Names: ghcr.io/acme/api:1.4.0 is registry, repository and tag. Without a registry host Docker assumes Docker Hub: golang:1.25 means docker.io/library/golang:1.25.

RUN executes at build time and saves a layer; ENTRYPOINT (the program) and CMD (its default arguments, replaced by whatever follows the image name in docker run) act when the container starts. Use the exec form ["/api"]: the binary becomes PID 1, with no /bin/sh -c in front. Overrides: docker run api -addr=:9090 runs /api -addr=:9090; --entrypoint swaps the program and drops CMD as well. Without an ENTRYPOINT, CMD is the whole command, so docker run api sh replaces all of it. Exec form vs shell form: ENTRYPOINT ["/api"] starts the binary directly as PID 1. ENTRYPOINT /api -addr=:8080 runs it under /bin/sh -c: distroless has no shell, so the container fails with exec: "/bin/sh": stat /bin/sh: no such file or directory, and elsewhere the shell can stay PID 1 and swallow SIGTERM. Shell form also ignores CMD and docker run arguments. The JSON must be valid: single quotes or a # comment at the end of the same line silently turn the exec form into the shell form. docker build --check reports it as JSONArgsRecommended. **No $VAR in the exec form**: there is no shell to expand it. A Go service reads its environment with os.Getenv anyway.

EXPOSE only documents the port the app listens on; it opens nothing. -p 8080:8080 publishes it: Docker forwards port 8080 of the host to port 8080 of the container. That traffic arrives on the container's network interface, not on its loopback, so the server has to listen on :8080. **host:container**: -p 9000:8080 serves on host port 9000 while the app still listens on 8080. -P publishes every EXPOSEd port on a random host port, which is about all EXPOSE changes. The loopback trap: http.ListenAndServe("127.0.0.1:8080", mux) (or localhost:8080, popular on macOS because it avoids the firewall prompt) accepts only connections from inside the container. Through the published port they are reset or refused, and the server logs nothing. Publishing opens every interface: -p 5432:5432 binds 0.0.0.0, so a dev Postgres is reachable from the whole network, and on Linux Docker's iptables rules bypass ufw. -p 127.0.0.1:5432:5432 keeps it local. **Container to container needs no -p**: on a user-defined network (Compose creates one per project) containers reach each other by name on the container port, postgres:5432. The default bridge network has no name resolution.

Both keep data outside the container's writable layer, so it survives docker rm. A named volume is storage Docker manages, the place for Postgres data; a bind mount maps a host directory into the container, for code, config and fixtures. down -v deletes the named volumes. Named volume: pgdata:/var/lib/postgresql/data. It lives in Docker's own storage (on a Mac, inside the Docker Desktop VM), survives docker compose down and image upgrades, and a new empty volume is first filled with whatever the image has at that path. Bind mount: ./migrations:/migrations:ro. The container sees the host's files and their owners as they are: on Linux a non-root container user cannot write to a directory owned by another UID, and on macOS file sharing is slower than a volume. **down vs down -v**: docker compose down removes containers and networks and keeps volumes. -v also removes the named volumes declared in the file and the anonymous ones, so the next up starts with an empty database. Host directories of bind mounts are never touched. The path depends on the major version: up to postgres:17 mount /var/lib/postgresql/data; from 18 the image keeps data in a versioned subdirectory (/var/lib/postgresql/18/docker) and expects the volume at /var/lib/postgresql. A volume still mounted at the old path makes the container exit at startup with an error about the new layout.

Use a multi-stage build: compile a static binary with CGO_ENABLED=0 in a golang image, then copy only that binary into distroless/static or scratch. On scratch what breaks is usually not the binary but the files it reads at runtime: CA certificates and timezone data. Copy go.mod and go.sum and run go mod download before copying the source, so the dependency layer stays cached; a --mount=type=cache for the build cache speeds up rebuilds further. CGO_ENABLED=0 gives a statically linked binary. With cgo on (the default for a native Linux build) the binary needs libc and the dynamic loader, and on scratch fails with exec /app: no such file or directory. On scratch the first HTTPS call fails with x509: certificate signed by unknown authority (no /etc/ssl/certs) and time.LoadLocation with unknown time zone (no /usr/share/zoneinfo). Copy the CA bundle from the build stage, and add import _ "time/tzdata" (about 450 KB) to embed the zone data. gcr.io/distroless/static-debian12:nonroot already has certificates, tzdata and a non-root user, which makes it the usual default. Run as non-root either way. -trimpath -ldflags="-s -w" drops local paths and debug symbols for a smaller binary; panic stack traces still work.

COPY . . comes before go mod download, so a change to any file invalidates that layer and every layer after it, the download included. Copy only go.mod and go.sum first, download, then copy the source; cache mounts keep the compiled packages between builds. How the cache decides: an instruction is reused only if its parent layer was reused and its inputs are unchanged: the command text for RUN, a checksum of the copied files for COPY. The first miss rebuilds everything below it. **Why go.mod and go.sum first**: they change only when dependencies do, so the go mod download layer stays cached across ordinary commits, and a source change misses only from the second COPY on. What a layer cannot keep: go build in a fresh layer starts with an empty build cache and compiles every dependency again. --mount=type=cache,target=/root/.cache/go-build keeps GOCACHE between builds on the same builder, so only changed packages compile; a cache mount on /go/pkg/mod saves the downloads when go.sum does change. Cache mounts stay on the builder: they are part of neither the image nor the exported layer cache, so a fresh CI runner starts with them empty (see the CI pipeline question). The context matters too: without a .dockerignore, a change in .git, a log file or a local binary also breaks COPY . ..

The build context is the directory passed to docker build (the final .), and the only host files COPY can read. Without a .dockerignore, COPY . . copies all of it, .git and .env files included, so secrets reach image layers and unrelated changes break the cache. Secrets: a .env, a *.pem or a kubeconfig copied by COPY . . sits in a layer of the build stage; in a single-stage Dockerfile it ships with the image and stays in its history even after RUN rm. Cache misses: COPY . . checksums everything it copies, so a git fetch, a new log file or a rebuilt local binary invalidates the source layer although no Go file changed. Speed: BuildKit transfers only what the build reads and only what changed since the last build, but COPY . . reads everything that is not ignored, so a stray node_modules or a 2 GB fixture slows every build. Allowlist, not denylist: start with * and re-include !go.mod, !go.sum, !cmd/, !internal/, so a new file stays out until someone decides otherwise. The Go side effect: with .git excluded, go build cannot stamp vcs.revision into the binary; pass the commit as a build argument into -ldflags "-X main.commit=$COMMIT". **COPY vs ADD**: COPY copies files from the context; ADD can also download a URL, clone a Git repository and unpack a local tar archive. Use COPY unless you want exactly that, so a .tar.gz is never unpacked by surprise.

A Go binary is machine code for one CPU architecture, and on an M-series Mac docker build produces a linux/arm64 image by default, which an amd64 kernel cannot run. Build both with docker buildx build --platform linux/amd64,linux/arm64, and let Go cross-compile instead of emulating. One tag, several images: a multi-platform image is an index (manifest list) with one manifest per platform under one tag, and each node pulls the one for its architecture. docker buildx imagetools inspect lists them. Emulation: with an unchanged Dockerfile, BuildKit runs every RUN of the foreign platform under QEMU, which Docker Desktop ships and a Linux builder has to install. It works for any language, but go build under emulation is several times slower. Cross-compilation, the Go way: FROM --platform=$BUILDPLATFORM runs the build stage natively, and GOOS=$TARGETOS GOARCH=$TARGETARCH makes Go emit the target's binary; with CGO_ENABLED=0 no extra toolchain is needed. Declare ARG TARGETOS TARGETARCH after go mod download, so that layer is shared by every platform. cgo breaks the shortcut: a C dependency needs a C cross-compiler for each target (set CC, or use the tonistiigi/xx helpers), or a fall back to emulation or native arm64 runners. One platform only: docker build --platform linux/amd64 is the quick fix when every server is amd64. How it shows up: the container exits at once, and a pod goes into CrashLoopBackOff with exec format error as its only log line.

A tag is a movable name in the registry, and latest is just the default tag, not "the newest": what it points to can change between two pulls. A digest (@sha256:...) is the hash of the image manifest itself, so it always means exactly the same bytes. **What goes wrong with latest**: nobody can say which build is running, two nodes that pulled at different times run different code, and "roll back to the previous latest" has no target. In Kubernetes: an image tagged :latest, or not tagged at all, defaults to imagePullPolicy: Always, so every pod start depends on the registry. Pushing a new latest does not change the Deployment, so no rollout starts, and pods pick up the new image one by one as they happen to restart. Immutable tags for releases: tag each build with the commit SHA or a version (api:9f2c1e4, api:1.4.0), switch on tag immutability in the registry (ECR, Harbor and Artifact Registry have it), and promote that same image from staging to production. Digests for base images: FROM golang:1.25 silently moves with every Go patch release and Debian update. FROM golang:1.25.14@sha256:... pins the exact image: the tag stays for humans, the digest decides what is pulled. Renovate or Dependabot open a PR when a new digest appears, so pinning does not mean never patching. A multi-platform tag has an index digest: pin that one, not the digest of the amd64 manifest inside it, or arm64 nodes get an amd64 image.

The script is PID 1 and the Go binary is its child, so the kubelet's SIGTERM goes to sh, which neither forwards it nor, as PID 1 without a handler, even dies of it. Go never sees the signal, and after 30 s everything gets SIGKILL. PID 1 is special: the kernel delivers a signal to a namespace's init process only if it installed a handler for it. sh waiting on a child has none, so SIGTERM is dropped. Go always installs handlers, so as PID 1 it does receive it. The fix: end the script with exec /app/server "$@". exec replaces the shell with the binary in the same PID 1, and the signal lands in signal.NotifyContext. Shell form has the same trap: ENTRYPOINT /app/server --port=8080 runs under /bin/sh -c; some shells exec a lone command, some do not. Use the exec form ENTRYPOINT ["/app/server", "--port=8080"], which on distroless is the only one that works at all: there is no /bin/sh. Symptoms: every rollout waits the full grace period per pod, the exit code is 137, and in-flight requests are cut because the drain never ran. Child processes: if the service spawns subprocesses, add a tiny init (tini, or docker run --init) that forwards signals and reaps zombies.

Without user namespaces, UID 0 in a container is UID 0 on the host, held back only by dropped capabilities and seccomp, so a compromised root process is one kernel or runtime bug from the node. A Go binary needs no privileges: set a numeric USER in the image and enforce it with runAsNonRoot. **On scratch**: USER 65534:65534 (nobody). With no /etc/passwd, only a numeric UID works. On distroless: use the :nonroot tag (UID 65532), and write USER 65532:65532 rather than USER nonroot. With runAsNonRoot: true the kubelet refuses a named user (image has non-numeric user (nonroot), cannot verify user is non-root) unless the pod sets runAsUser. Ports: listen on 8080, not 80: ports below 1024 need CAP_NET_BIND_SERVICE on many runtimes, and the Service can still expose 80. The rest of the hardening: readOnlyRootFilesystem: true with an emptyDir for temp files, allowPrivilegeEscalation: false, and all capabilities dropped.

Every layer of an image is readable by anyone who can pull it, and deleting a file or overriding a variable in a later layer does not remove it from the earlier one. Build-time secrets go through BuildKit secret mounts; runtime secrets come from the platform as env vars or mounted files. How they leak: ENV DB_PASSWORD=... sits in the image config (docker inspect), an ARG used by a RUN shows up in docker history, a copied .env stays in its layer after rm. And the image is pushed to a registry, cached in CI and pulled onto every node. Build time: usually a token for private modules. RUN --mount=type=secret,id=netrc,target=/root/.netrc go mod download exposes it to that one command and writes it into no layer. Runtime: a Kubernetes Secret, ideally synced from Vault or a cloud secret manager, injected as an env var or a file. The same image then runs in every environment. Env or file: env is simplest, but child processes inherit it, /proc/<pid>/environ shows it, and it changes only on restart. A mounted file can have tighter permissions and is updated in place when the secret rotates (not with subPath). In Go: load once at startup, keep it out of logs and error messages, and never pass it as a command-line flag: ps and /debug/pprof/cmdline show flags.

One compose.yaml: the service built from its Dockerfile, Postgres and Redis from official images, a named volume for the data. A plain depends_on waits only for the Postgres container to start; condition: service_healthy with a pg_isready healthcheck waits until it accepts connections. Why "started" is not enough: Postgres needs a few seconds to initialise, longer on the first run, so a service that pings the database at startup fails and exits. During that first run the image's temporary server listens on the Unix socket only, so pg_isready without -h already reports "accepting connections" while TCP is closed: check with -h 127.0.0.1. Migrations as a one-shot service: a migrate service applies the schema and exits; condition: service_completed_successfully starts the API only after it succeeded. The Go service still retries: depends_on acts only during docker compose up. In production nothing orders startup and Postgres can restart under a running service, so connect with a timeout and retry with backoff instead of relying on the order. Addresses come from the environment: inside Compose the database is postgres:5432, for go run on the host it is localhost:5432, and the binary only reads DATABASE_URL. A fast edit loop: many teams run only the dependencies in Compose (docker compose up -d postgres redis) and the service with go run, or use develop.watch so that docker compose watch rebuilds on file changes. **HEALTHCHECK is a Docker feature**: Compose and docker ps use it, plain Docker does not restart an unhealthy container, and Kubernetes ignores it in favour of its own probes.

Inside a container localhost is the container itself, and nothing in the api container listens on 5432. Compose puts both services on one network where every service name resolves to its container, so the address is postgres:5432. **Why go run worked**: ports: "5432:5432" publishes Postgres on the host's localhost. That mapping exists for the host only, and containers never go through it. Use the container port: with ports: "15432:5432" the host connects to 15432, but containers still dial postgres:5432; postgres:15432 is refused. Traffic between containers needs no ports entry at all. Keep the address in config: DATABASE_URL from the environment lets the same binary use localhost under go run and postgres under Compose; a hard-coded host is this bug waiting to happen. A database on the host itself: from a container use host.docker.internal, which Docker Desktop provides; on Linux add extra_hosts: ["host.docker.internal:host-gateway"]. The healthcheck was fine: service_healthy passed because Postgres was ready; the API simply dialled the wrong host.

Ship as little as possible, know what is inside, keep it patched, prove where it came from, and limit what the running container may do. For Go that means a static binary on distroless, a scanner and an SBOM in CI, digest-pinned bases rebuilt often, signed images and a locked-down runtime. Ship less A static binary on distroless/static or scratch brings no shell, package manager or curl: fewer CVEs to triage and nothing for an attacker to use after a remote code execution. A full golang or Debian base adds hundreds of packages the service never calls. Know what is inside Trivy, Grype or Docker Scout match OS packages against CVE databases and, because a Go binary records its module list (the one go version -m prints, still there after -s -w), Go modules and the standard library too, even on scratch. They match versions, so they also flag code you never call; govulncheck -mode=binary checks which vulnerable functions are linked into the binary (scan a build without -s: with no symbol table it reports every vulnerability of every module inside). Fail CI on fixable HIGH and CRITICAL findings and triage the rest. An SBOM (--sbom=true on docker buildx build attaches an SPDX document) is the inventory you query when the next big CVE lands, instead of pulling and rescanning every image. Keep it patched A clean image today has CVEs next month. Pin bases by digest, let Renovate bump them, and rebuild on a schedule even without code changes: a standard library CVE is fixed only by rebuilding with a newer Go. Prove where it came from Provenance attestations (--provenance=mode=max, SLSA) record what the image was built from and how. cosign sign signs the digest, and an admission policy (Kyverno, Sigstore policy-controller) refuses unsigned images. Limit what it can do Besides a non-root user: a read-only root filesystem, every capability dropped, no-new-privileges, the default seccomp profile, never --privileged. Never mount /var/run/docker.sock into a container: whoever controls the Docker daemon is root on the host.

Bring the tools from outside instead of shipping them in the image: a debug container that joins the target's process and network namespaces. In Docker that is --pid and --network set to container:api, or docker debug; in Kubernetes, an ephemeral container from kubectl debug --target. Docker docker run --rm -it --pid=container:api --network=container:api nicolaka/netshoot gives a shell in which ps shows /api as PID 1, and ss, curl localhost:8080 and tcpdump see the service's own network. The target's filesystem is at /proc/1/root/. Reading it needs the same UID and GID as the process or --cap-add=SYS_PTRACE; otherwise it is Permission denied. With Docker Desktop, docker debug api does all of this in one command and also works on images and stopped containers. Kubernetes kubectl debug -it pod/api-7d9f --image=busybox:1.37 --target=api --profile=general adds an ephemeral container (stable since 1.25): --target shares the process namespace of the api container, and the general profile adds SYS_PTRACE. It needs its own RBAC permission (pods/ephemeralcontainers) and cannot be removed: it stays until the pod is deleted. A pod in CrashLoopBackOff has nothing running to join. Start with kubectl logs --previous and kubectl describe pod; kubectl debug --copy-to=api-debug starts a copy of the pod in which you can change the image or the command. Often no shell is needed pprof and /metrics on the admin port, reached through kubectl port-forward, answer most questions about a Go service. SIGQUIT (docker kill -s QUIT api) makes the Go runtime print every goroutine's stack to stderr, which shows where a hung service is stuck. The process then exits with code 2, so do it on an instance you are about to replace anyway.

They are layers of one stack: dockerd builds and manages, containerd pulls images and runs containers, runc creates the process with namespaces and cgroups. Images follow the OCI specs, so a Docker-built image runs unchanged on containerd or CRI-O: dropping dockershim did not touch images. The stack docker run goes to dockerd (API, networks, volumes, BuildKit builds), which hands the container to containerd (pulls and unpacks layers, supervises each container through a shim), which calls runc: it sets up namespaces and cgroups, starts the process and exits. OCI defines the image format (manifest, config, layers), the registry API and the runtime bundle that runc reads. Tools on both sides interoperate: docker build, Buildah and ko (a Go image without a Dockerfile) produce images; containerd, CRI-O and Podman run them. Kubernetes after 1.24 The kubelet talks to runtimes over CRI, a gRPC API. Docker Engine never implemented it, so the kubelet carried an adapter, dockershim; since 1.24 nodes run containerd or CRI-O directly (Mirantis' cri-dockerd exists for clusters that must keep Docker Engine). Unchanged: Dockerfiles, docker build in CI, registries, images. Broken: whatever relied on the node's Docker daemon: pods that mount /var/run/docker.sock to build images (use BuildKit or Buildah in the cluster), agents that read Docker's log files, and docker ps on a node (use crictl ps). Podman Daemonless: each container is the child of a small monitor (conmon) rather than of a root daemon, and it runs rootless by default through user namespaces. Its CLI is Docker-compatible, and it adds pods and podman kube play for Kubernetes YAML. Underneath it uses the same OCI runtimes, crun or runc. Docker has a rootless mode too; the real difference is the daemon: access to Docker's socket is root on the host. Swapping the runtime Through a RuntimeClass a pod can run on gVisor (runsc) or Kata Containers (a micro-VM per pod) instead of runc, when sharing the host kernel is too risky.