Irbisa · cheatsheetSeptember 19, 2026

Runtime, Scheduler & GC

G-M-P, preemption, escape analysis, GOGC, GOMEMLIMIT, pprof

Senior Developer16 itemscompressed for a skim

01
A gateway holds 200 000 idle WebSocket connections with one goroutine per connection and the process sits at 3 GB. A colleague says the goroutines are the problem. What does a parked goroutine actually cost?
Easy
A goroutine starts with a 2 KB stack and grows it by copying, so a parked one costs roughly that plus a small g struct — an OS thread reserves 1-8 MB of stack up front, which is the whole reason the goroutine-per-connection design works at all.
```
// What 100k parked goroutines actually cost - measure it, don't argue about it.
func main() {
	var before, after runtime.MemStats
	runtime.GC()
	runtime.ReadMemStats(&before)
…
```
02
The pod is limited to 2 CPUs on a 64-core node, runtime.NumCPU() prints 64, and under load latency jumps in steps of about 100 ms. What is the runtime getting wrong, and what do you set?
Easy
GOMAXPROCS is the number of Ps — goroutines allowed to execute Go code simultaneously — and through Go 1.24 it defaults to the machine's logical CPU count rather than the cgroup quota, so a 2-CPU pod runs 64 Ps that the kernel throttles in 100 ms periods.
```
// One blank import does the same job: _ "go.uber.org/automaxprocs"
func init() {
	b, err := os.ReadFile("/sys/fs/cgroup/cpu.max")
	if err != nil {
		return // not in a cgroup v2 container: leave the default alone
	}
…
```
03
Someone set GOGC=400 on the API pods: GC CPU fell from 12% to 4% and the pods started getting OOM-killed at their 2 GiB limit. What does GOGC actually control?
Easy
GOGC is a ratio, not a size: the next cycle is targeted at live heap x (1 + GOGC/100), so GOGC=400 lets the heap reach five times the live set instead of the default two — and nothing in that formula knows what the container limit is.
```
func main() {
	// Equivalent to GOGC=100 GOMEMLIMIT=900MiB, but visible to code review.
	debug.SetGCPercent(100)
	debug.SetMemoryLimit(900 << 20) // ~90% of a 1 GiB container limit

	var live [][]byte
…
```
04
At 40k rps the CPU profile is one third runtime.mallocgc, the GC runs twenty times a second, and yet go tool pprof /debug/pprof/heap shows a flat 30 MB. Which number were you reading?
Easy
The heap profile opens on inuse_space — what is live at the moment of the sample — while the churn you are chasing is in alloc_space/alloc_objects, the bytes and objects allocated cumulatively since the process started.
```
// Profiles belong on an internal listener, never on the mux that serves users.
func debugServer(addr string) {
	mux := http.NewServeMux()
	mux.HandleFunc("/debug/pprof/", pprof.Index) // heap, goroutine, block, mutex
	mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
	go func() { _ = http.ListenAndServe(addr, mux) }()
…
```
05
Fanning 5 000 jobs out with go work(j) from a single goroutine leaves one core pinned while the others pick work up only after a visible delay. Follow one job from the go statement to a core: where can it sit?
Medium
A new goroutine goes into the local run queue of the P that created it — into that P's one-slot runnext first — and other Ps can only get it by work stealing, so a burst created on one P starts out effectively serial.
```
// ❌ 5000 goroutines created on one P: they queue behind their own producer
for _, j := range jobs {
	go work(j)
}

// ✅ one worker per P, created once, fed by a channel
…
```
06
Three variants of one service park 20 000 goroutines each: on a channel, in conn.Read, and in os.File.Read on a slow disk. The OS thread counts are 9, 9 and 2 000. Explain all three.
Medium
Only the third blocks a thread: a channel park hands the P back and parks the G, a network read parks the G on the netpoller (epoll/kqueue/IOCP), and a blocking syscall keeps its M, so sysmon has to take the P away and start another thread.
```
// Regular-file reads and cgo calls each hold an OS thread for their duration:
// bound them, or one burst leaves you with thousands of parked threads.
type diskPool struct{ sem chan struct{} }

func newDiskPool(n int) *diskPool { return &diskPool{sem: make(chan struct{}, n)} }
…
```
07
A colleague remembers a for loop with no function calls in it freezing an entire service, and says an upgrade fixed it. What changed, and where is a goroutine still not preemptible?
Medium
Until Go 1.14 preemption happened only at function-call safepoints — the stack-growth check in the prologue — so a loop that called nothing never yielded and every stop-the-world waited for it forever; Go 1.14 added asynchronous preemption, where sysmon signals the thread with SIGURG and the handler parks the goroutine.
```
// Go 1.14+ preempts this loop with a signal. Run the same binary with
// GODEBUG=asyncpreemptoff=1 and it never prints: runtime.GC() cannot
// stop the world, because the loop has no call and therefore no safepoint.
func main() {
	runtime.GOMAXPROCS(1)
…
```
08
go build -gcflags='-m=2' prints &Point{...} escapes to heap inside the constructor and &Point{...} does not escape at a call site three lines below. Which one is true, and what actually forces an escape?
Medium
Both are: the compiler analyses each function on its own, so a pointer returned from NewPoint has to be assumed to outlive the frame — but once NewPoint is inlined into its caller, the literal is re-analysed in the caller's frame and stays on the stack.
```
type Point struct{ X, Y int }

// Inlinable: "can inline NewPoint with cost 7".
func NewPoint(x, y int) *Point { return &Point{X: x, Y: y} }

//go:noinline
…
```
09
Your JSON-to-protobuf gateway spends 35% of its CPU in runtime.mallocgc and gcBgMarkWorker, and the code looks unremarkable. Which allocation patterns do you go looking for, and how do you prove each fix?
Medium
GC cost follows the allocation rate, not the heap size, so the work is removing allocations: unsized append, conversions between []byte and string, per-request buffers that should be pooled, and values boxed into interfaces on hot paths.
```
var bufs = sync.Pool{New: func() any { return new(bytes.Buffer) }}

// ❌ a Sprintf and a whole new string per row; the previous one is garbage
func renderSlow(rows []Row) string {
	out := ""
	for _, r := range rows {
…
```

The leak is fixed and the heap profile is flat at 300 MB, but the container's RSS sits at 1.4 GB and a `runtime.GC()` from the debug endpoint does not move it. Where is the memory?

Medium

RSS is not the heap. It also covers goroutine stacks, span and heap metadata, the allocator's free pages that the background scavenger returns only gradually, and anything cgo or your own mmap did — freeing an object never returns a page by itself.

// metrics.Read takes a lock. runtime.ReadMemStats stops the world for the same numbers.
var samples = []metrics.Sample{
	{Name: "/memory/classes/total:bytes"},         // tracks RSS for a pure-Go process
	{Name: "/memory/classes/heap/objects:bytes"},  // live objects plus floating garbage
	{Name: "/memory/classes/heap/free:bytes"},     // free pages the scavenger still holds
	{Name: "/memory/classes/heap/released:bytes"}, // returned to the OS, still mapped
…

11
p99 is 800 ms and the CPU profile is boring: nothing in it sums to anywhere near that latency. Which profiles answer "what was the request waiting for", and what has to be switched on first?
Medium
A CPU profile only samples goroutines that are running, so waiting is invisible to it; the ones that see waiting are the block and mutex profiles — both off by default — plus go tool trace for the per-goroutine timeline.
```
func main() {
	// Both are off by default: a CPU profile cannot see a goroutine that is waiting.
	runtime.SetBlockProfileRate(1_000_000) // one sample per millisecond blocked
	runtime.SetMutexProfileFraction(5)     // one contention event in five
	serve()
}
…
```
12
An old box, no profiler, no tooling, and a service that goes unresponsive for half a second every few minutes. You get one environment variable and a restart. What do you set, and what do you read?
Medium
GODEBUG=gctrace=1,schedtrace=1000 — one line per GC cycle and one line per second of scheduler state, both on stderr, which is enough to separate "the collector stopped me" from "nothing was scheduled".
```
// GODEBUG=gctrace=1,schedtrace=1000 ./service 2>gc.log
// The same two questions in-process, for when you cannot restart with an env var.
func gcHealth() string {
	var s debug.GCStats
	s.PauseQuantiles = make([]time.Duration, 5) // min, p25, p50, p75, max
	debug.ReadGCStats(&s)
…
```
13
An architect rejects Go for a 40 GB in-memory cache because "mark-and-sweep has to walk 40 GB with the world stopped". What is actually stopped, and what does grow with the heap?
Hard
Nothing walks the heap with the world stopped: Go runs a concurrent tri-colour mark-and-sweep collector that stops the world twice per cycle for bounded work — sweep termination and mark termination, typically well under a millisecond — and stays correct while your code runs by using a write barrier.
```
// ❌ ten million pointers the marker walks on every cycle
type slowCache struct{ m map[string]*Entry }

// ✅ no pointers inside: spans without pointers are never scanned
type cache struct {
	index map[uint64]extent // hashed key -> offset; no pointer in key or value
…
```
14
A load test climbs to 12k rps, then collapses to 4k with p99 going from 40 ms to 2 s. gctrace shows cycles running back to back and a CPU triple like 0.5+900/400/0+0.8 ms. What is throttling the handlers?
Hard
GC assist: a goroutine that allocates while a mark phase is running must first do a proportional share of the marking itself in runtime.gcAssistAlloc — that first number in the triple is assist CPU, and it dwarfing the dedicated workers means your handlers have been conscripted into the collector.
```
var bufs = sync.Pool{New: func() any { b := make([]byte, 32<<10); return &b }}

// ❌ one allocation the size of the request, made during the mark phase
func slow(w http.ResponseWriter, r *http.Request) {
	buf := make([]byte, r.ContentLength) // assist debt is linear in these bytes
	_, _ = io.ReadFull(r.Body, buf)
…
```
15
Once a day, during traffic spikes, a 2 GiB pod with a ~700 MB live heap is OOM-killed, and an old runbook suggests a 1 GiB ballast. What do you configure instead, and what exactly does that setting count?
Hard
Set GOMEMLIMIT (Go 1.19) to the container limit minus your non-Go memory — about 1.7 GiB in a 2 GiB pod — and leave GOGC=100: it is a soft limit the pacer targets across all runtime-managed memory, and it is what made the ballast trick obsolete.
```
// Read the cgroup limit, keep headroom for everything the limit does not count.
func setMemoryLimit(fraction float64) {
	b, err := os.ReadFile("/sys/fs/cgroup/memory.max") // cgroup v2; "max" if unlimited
	if err != nil {
		return
	}
…
```
16
Your cgo integration stores uintptr(unsafe.Pointer(&buf[0])) in a C struct and reads it back a second later. Tests pass; production corrupts memory. What happened in between?
Hard
The stack moved.
```
//go:noinline
func grow(n int) int {
	var pad [512]byte
	if n == 0 {
		return int(pad[0])
	}
…
```

Runtime, Scheduler & GC

A gateway holds 200 000 idle WebSocket connections with one goroutine per connection and the process sits at 3 GB. A colleague says the goroutines are the problem. What does a parked goroutine actually cost?

The pod is limited to 2 CPUs on a 64-core node, `runtime.NumCPU()` prints 64, and under load latency jumps in steps of about 100 ms. What is the runtime getting wrong, and what do you set?

Someone set `GOGC=400` on the API pods: GC CPU fell from 12% to 4% and the pods started getting OOM-killed at their 2 GiB limit. What does `GOGC` actually control?

At 40k rps the CPU profile is one third `runtime.mallocgc`, the GC runs twenty times a second, and yet `go tool pprof /debug/pprof/heap` shows a flat 30 MB. Which number were you reading?

Fanning 5 000 jobs out with `go work(j)` from a single goroutine leaves one core pinned while the others pick work up only after a visible delay. Follow one job from the `go` statement to a core: where can it sit?

Three variants of one service park 20 000 goroutines each: on a channel, in `conn.Read`, and in `os.File.Read` on a slow disk. The OS thread counts are 9, 9 and 2 000. Explain all three.

A colleague remembers a `for` loop with no function calls in it freezing an entire service, and says an upgrade fixed it. What changed, and where is a goroutine still not preemptible?

`go build -gcflags='-m=2'` prints `&Point{...} escapes to heap` inside the constructor and `&Point{...} does not escape` at a call site three lines below. Which one is true, and what actually forces an escape?

Your JSON-to-protobuf gateway spends 35% of its CPU in `runtime.mallocgc` and `gcBgMarkWorker`, and the code looks unremarkable. Which allocation patterns do you go looking for, and how do you prove each fix?

The leak is fixed and the heap profile is flat at 300 MB, but the container's RSS sits at 1.4 GB and a `runtime.GC()` from the debug endpoint does not move it. Where is the memory?

p99 is 800 ms and the CPU profile is boring: nothing in it sums to anywhere near that latency. Which profiles answer "what was the request waiting for", and what has to be switched on first?

An old box, no profiler, no tooling, and a service that goes unresponsive for half a second every few minutes. You get one environment variable and a restart. What do you set, and what do you read?

An architect rejects Go for a 40 GB in-memory cache because "mark-and-sweep has to walk 40 GB with the world stopped". What is actually stopped, and what does grow with the heap?

A load test climbs to 12k rps, then collapses to 4k with p99 going from 40 ms to 2 s. `gctrace` shows cycles running back to back and a CPU triple like `0.5+900/400/0+0.8 ms`. What is throttling the handlers?

Once a day, during traffic spikes, a 2 GiB pod with a ~700 MB live heap is OOM-killed, and an old runbook suggests a 1 GiB ballast. What do you configure instead, and what exactly does that setting count?

Your cgo integration stores `uintptr(unsafe.Pointer(&buf[0]))` in a C struct and reads it back a second later. Tests pass; production corrupts memory. What happened in between?

A gateway holds 200 000 idle WebSocket connections with one goroutine per connection and the process sits at 3 GB. A colleague says the goroutines are the problem. What does a parked goroutine actually cost?

The pod is limited to 2 CPUs on a 64-core node, runtime.NumCPU() prints 64, and under load latency jumps in steps of about 100 ms. What is the runtime getting wrong, and what do you set?

Someone set GOGC=400 on the API pods: GC CPU fell from 12% to 4% and the pods started getting OOM-killed at their 2 GiB limit. What does GOGC actually control?

At 40k rps the CPU profile is one third runtime.mallocgc, the GC runs twenty times a second, and yet go tool pprof /debug/pprof/heap shows a flat 30 MB. Which number were you reading?

Fanning 5 000 jobs out with go work(j) from a single goroutine leaves one core pinned while the others pick work up only after a visible delay. Follow one job from the go statement to a core: where can it sit?

Three variants of one service park 20 000 goroutines each: on a channel, in conn.Read, and in os.File.Read on a slow disk. The OS thread counts are 9, 9 and 2 000. Explain all three.

A colleague remembers a for loop with no function calls in it freezing an entire service, and says an upgrade fixed it. What changed, and where is a goroutine still not preemptible?

go build -gcflags='-m=2' prints &Point{...} escapes to heap inside the constructor and &Point{...} does not escape at a call site three lines below. Which one is true, and what actually forces an escape?

Your JSON-to-protobuf gateway spends 35% of its CPU in runtime.mallocgc and gcBgMarkWorker, and the code looks unremarkable. Which allocation patterns do you go looking for, and how do you prove each fix?

The leak is fixed and the heap profile is flat at 300 MB, but the container's RSS sits at 1.4 GB and a runtime.GC() from the debug endpoint does not move it. Where is the memory?

p99 is 800 ms and the CPU profile is boring: nothing in it sums to anywhere near that latency. Which profiles answer "what was the request waiting for", and what has to be switched on first?

An old box, no profiler, no tooling, and a service that goes unresponsive for half a second every few minutes. You get one environment variable and a restart. What do you set, and what do you read?

An architect rejects Go for a 40 GB in-memory cache because "mark-and-sweep has to walk 40 GB with the world stopped". What is actually stopped, and what does grow with the heap?

A load test climbs to 12k rps, then collapses to 4k with p99 going from 40 ms to 2 s. gctrace shows cycles running back to back and a CPU triple like 0.5+900/400/0+0.8 ms. What is throttling the handlers?

Once a day, during traffic spikes, a 2 GiB pod with a ~700 MB live heap is OOM-killed, and an old runbook suggests a 1 GiB ballast. What do you configure instead, and what exactly does that setting count?

Your cgo integration stores uintptr(unsafe.Pointer(&buf[0])) in a C struct and reads it back a second later. Tests pass; production corrupts memory. What happened in between?

The pod is limited to 2 CPUs on a 64-core node, `runtime.NumCPU()` prints 64, and under load latency jumps in steps of about 100 ms. What is the runtime getting wrong, and what do you set?

Someone set `GOGC=400` on the API pods: GC CPU fell from 12% to 4% and the pods started getting OOM-killed at their 2 GiB limit. What does `GOGC` actually control?

At 40k rps the CPU profile is one third `runtime.mallocgc`, the GC runs twenty times a second, and yet `go tool pprof /debug/pprof/heap` shows a flat 30 MB. Which number were you reading?

Fanning 5 000 jobs out with `go work(j)` from a single goroutine leaves one core pinned while the others pick work up only after a visible delay. Follow one job from the `go` statement to a core: where can it sit?

Three variants of one service park 20 000 goroutines each: on a channel, in `conn.Read`, and in `os.File.Read` on a slow disk. The OS thread counts are 9, 9 and 2 000. Explain all three.

A colleague remembers a `for` loop with no function calls in it freezing an entire service, and says an upgrade fixed it. What changed, and where is a goroutine still not preemptible?

`go build -gcflags='-m=2'` prints `&Point{...} escapes to heap` inside the constructor and `&Point{...} does not escape` at a call site three lines below. Which one is true, and what actually forces an escape?

Your JSON-to-protobuf gateway spends 35% of its CPU in `runtime.mallocgc` and `gcBgMarkWorker`, and the code looks unremarkable. Which allocation patterns do you go looking for, and how do you prove each fix?

The leak is fixed and the heap profile is flat at 300 MB, but the container's RSS sits at 1.4 GB and a `runtime.GC()` from the debug endpoint does not move it. Where is the memory?

A load test climbs to 12k rps, then collapses to 4k with p99 going from 40 ms to 2 s. `gctrace` shows cycles running back to back and a CPU triple like `0.5+900/400/0+0.8 ms`. What is throttling the handlers?

Your cgo integration stores `uintptr(unsafe.Pointer(&buf[0]))` in a C struct and reads it back a second later. Tests pass; production corrupts memory. What happened in between?