Runtime, Scheduler & GC

The garbage collector frees memory: Go has no free and no manual deallocation (the builtin delete only removes a map key), and the compiler decides whether a value lives on the stack or the heap. A Go program still leaks whenever it keeps something reachable that it no longer needs. new(T), &T{}, make and append allocate; whatever nothing points to any more is reclaimed by a later GC cycle. runtime.GC() runs a full collection and blocks the caller until it finishes. It belongs in tests and benchmarks; in production the runtime decides when to collect, and a forced cycle only burns CPU. debug.FreeOSMemory() collects and returns free pages to the OS, which is useful once after a known spike, not on a timer. Typical leaks: goroutines blocked forever on a channel, a map used as a cache with no eviction, a small subslice keeping a huge backing array alive, a global registry nobody cleans up.

The runtime runs many goroutines (G) on a few OS threads (M). A thread may run Go code only while it holds a P, one of GOMAXPROCS scheduling slots, each with its own run queue, so P caps parallelism and keeps scheduling mostly lock-free. The pieces G: a goroutine: its stack, saved registers and state (running, runnable, waiting). M: an OS thread. There can be far more Ms than Ps: threads blocked in system calls or cgo still exist but hold no P. P: a logical processor: a local run queue of up to 256 goroutines, a runnext slot for the goroutine just made ready, and a per-P allocation cache (mcache). Finding the next goroutine An M with a P takes runnext, then its local queue. Every 61st time it checks the global queue first, so nothing there starves. With an empty local queue it tries the global queue, then the network poller, then work stealing: it takes half the run queue of a random other P. A full local queue moves half of its goroutines to the global queue. Why P exists Before Go 1.1 there was one global run queue behind one lock, and every thread fought over it. With queues and caches inside P, most scheduling touches only local data, and a thread that blocks in a system call hands its P (queue and cache included) to another thread, so the work keeps running. runtime.Gosched() puts the current goroutine on the global queue and lets another run; real code almost never needs it.

It has been preemptive since Go 1.14. Before that a goroutine could be stopped only at a function call, so this loop kept the only P forever and main never woke up; now the runtime interrupts the loop with a signal and main prints done. Cooperative, before 1.14: the check lived in the function prologue, where Go tests whether the stack must grow. The runtime asked a goroutine to yield by tripping that check, so a loop with no calls never saw the request. Such a loop could also block a stop-the-world GC phase forever. Asynchronous preemption, 1.14+: sysmon notices a goroutine running longer than 10 ms and sends its thread SIGURG; the signal handler saves the state and enters the scheduler. Still not preemptible: a goroutine inside a cgo call or a blocking system call (its P is handed to another thread instead), and short runtime-internal sections. GODEBUG=asyncpreemptoff=1 switches the signal off, which is how you reproduce the old hang. Preemption keeps the runtime fair, not your latency low: a CPU-bound goroutine still uses its whole 10 ms slice before the goroutines queued behind it on that P get a turn.

sysmon is a runtime thread that loops without a P and watches the scheduler from outside: it takes Ps away from threads stuck in system calls, preempts goroutines that ran too long, and keeps the network poller and the GC moving. Retakes Ps: a thread blocked in a system call or cgo for more than one sysmon tick (20 µs) loses its P when other goroutines are waiting; if nothing is queued, after 10 ms. The other goroutines keep running. Preempts: a goroutine on the CPU for more than 10 ms is asked to yield, and since Go 1.14 is interrupted with a signal. Polls the network: if no thread has checked epoll for 10 ms, it does so and puts the goroutines whose sockets are ready back on a run queue. Forces a GC: if no collection has run for two minutes, it starts one. It sleeps 20 µs between checks while there is work, backing off to 10 ms when the process is idle. It has no P on purpose: it must work exactly when every P is busy or stuck, and since it never runs user Go code it needs neither a run queue nor an allocation cache.

It prints runtime: goroutine stack exceeds 1000000000-byte limit and fatal error: stack overflow, then exits with status 2. The stack kept doubling up to 512 MiB; the next doubling, 1 GiB, would pass the 1 GB limit, and a fatal error is not a panic, so deferred functions never run. Growth: every function prologue checks the space left. When it runs out, the runtime allocates a stack twice as large, copies the frames and rewrites every pointer into the old one. That is why a goroutine stack can grow far beyond a thread's 8 MB. Shrinking: during GC, a goroutine using less than a quarter of its stack gets one half the size, so a single deep burst does not hold the memory forever. The limit: 1 000 000 000 bytes on 64-bit systems (250 000 000 on 32-bit), adjustable with debug.SetMaxStack. Stacks double, so the largest one a goroutine reaches is 512 MiB; copying it needs the old and the new stack at once. Past the limit the runtime cannot continue safely and aborts the whole process. recover stops panics only. Stack overflow, concurrent map writes and running out of memory are fatal errors that nothing intercepts. The fix is a base case, or an explicit stack (a slice) for input whose depth you do not control.

GOMAXPROCS is the number of Ps: how many threads may run Go code at the same time. Up to Go 1.24 it defaulted to the machine's CPU count and ignored the container's CPU limit; since Go 1.25 the default respects the cgroup limit on Linux. The old failure: a pod limited to 2 CPUs on a 64-core node ran 64 Ps. Together they used up the CFS quota early in each 100 ms period, then the kernel throttled the process until the next one: p99 spikes while average CPU looked fine. The old fix: a blank import of go.uber.org/automaxprocs, which reads the cgroup quota at startup and calls runtime.GOMAXPROCS. Go 1.25+: the default is the lower of the CPU count and the cgroup CPU limit, and the runtime updates it when the limit changes. This needs go 1.25 or later in go.mod and no explicit GOMAXPROCS setting. CPU requests are ignored; only limits count. The GC uses a quarter of GOMAXPROCS for background marking, so a wrong value distorts GC behaviour too. It is not a thread limit and not a cap on concurrent work: goroutines blocked in system calls run on extra threads, and bounding concurrency takes a semaphore or a worker pool.

Import net/http/pprof on a private port, take go tool pprof -http=:8081 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30' under real load, and read top and the flame graph by cumulative time. Enabling it: the blank import registers /debug/pprof/* on http.DefaultServeMux; serve it on 127.0.0.1 or an internal port, never on the public listener. Flat vs cum: flat is time spent in the function itself, cum includes everything it calls. Start from the largest cum in your own code, then look for the flat leaf below it; in the flame graph a wide box is a hot path. Cost: the profiler samples the stack about 100 times a second, so the overhead is a few percent of CPU, small enough to leave the endpoint on and take a profile only when needed. What it misses: a CPU profile sees only on-CPU time. Goroutines waiting on locks, channels or I/O do not show up; for those use the block, mutex and trace tools. Before and after: -base before.pb.gz after.pb.gz shows only the difference, which is how you confirm a fix helped. Other profiles use the same endpoint: heap, goroutine, allocs, block, mutex.

Escape analysis is the compiler deciding, for each value, whether it can outlive the function that creates it: if not, it stays on the stack and costs nothing to free; if it can, it goes to the heap. Here only local stays on the stack. local: &Point{} never leaves the function, so -m reports &Point{...} does not escape. Taking an address does not by itself mean the heap. returned: the pointer outlives the frame (moved to heap: p). stored: a package-level variable can be read at any later time (&Point{...} escapes to heap). printed: fmt.Println takes ...any, and the compiler cannot prove the interface value stays local (n escapes to heap). This is why logging on a hot path allocates. Other common causes: a variable captured by a closure that a goroutine runs, a pointer sent on a channel, a make whose size is not a constant (Go 1.25+ keeps small ones on the stack), a value too large for a stack frame. Check with go build -gcflags=-m; -m=2 explains each decision. That output is the compiler's claim; allocs/op from go test -bench . -benchmem is the proof.

Go uses a concurrent, non-moving, tri-colour mark-and-sweep collector: it marks reachable objects while your goroutines keep running, stops the world only twice per cycle, usually for well under a millisecond, and sweeps lazily as memory is reused. The cycle Sweep termination (stop-the-world): finish the previous sweep, turn the write barrier on. Mark (concurrent): start from the roots (globals and goroutine stacks) and follow pointers. Background workers use a quarter of GOMAXPROCS; goroutines that allocate meanwhile may be made to help (mark assist). Mark termination (stop-the-world): finish marking, turn the barrier off. Sweep (concurrent): unmarked objects become free slots, reclaimed before their span is reused. Three colours and the barrier White is not reached yet, grey is reached but its pointers are not scanned, black is scanned. Marking drains the grey set; whatever is still white at the end is garbage. Your code moves pointers while marking runs. It could hide the only pointer to a white object inside an already-black one, and a live object would be freed. The write barrier reports pointer writes during marking so that cannot happen; since Go 1.8 its hybrid form also spares a rescan of stacks at the end. Trigger and design A cycle starts when the heap reaches the goal set by GOGC and GOMEMLIMIT, after two minutes without one, or on runtime.GC(). It is neither generational nor compacting: escape analysis keeps many short-lived values off the heap, the size-class allocator limits fragmentation without moving objects, and a generational design would need the write barrier on all the time.

GOGC sets how far the heap may grow past the live data before the next cycle: the goal is roughly live heap + (live heap + stacks + globals) × GOGC/100. With the default 100, 100 MB of live data means the next cycle starts near 200 MB. Raising it (GOGC=200, 400) gives fewer cycles and less GC CPU, paid for in memory: the heap peaks at about three or five times the live set. Lowering it: less memory, more frequent cycles, more CPU. **GOGC=off**: no collection triggered by the ratio at all; sensible only together with GOMEMLIMIT. The risk: the ratio knows nothing about your container. A spike that doubles the live heap doubles the goal too, and the process can be OOM-killed before the next cycle. GOMEMLIMIT is the ceiling that reacts to that. In code it is debug.SetGCPercent, which returns the previous value. When GC CPU is high, look at the allocation rate first (-alloc_space in pprof): tuning GOGC trades memory for CPU, while fewer allocations save both.

gctrace=1 prints one line per GC cycle to stderr (pause and mark times, heap sizes and GC's share of CPU); schedtrace=1000 prints the scheduler's state every second. Together they tell a GC problem from a scheduling problem with no profiler at all. Reading the sample lines: gc 42 @12.513s 4%: the 42nd cycle, 12.5 s after start; GC has used 4% of CPU since startup. As a rule of thumb, a share that stays above 10% points at allocation volume. 0.031+3.1+0.052 ms clock: stop-the-world sweep termination, concurrent marking, stop-the-world mark termination. Only the first and the last freeze your goroutines. 0.25+0.40/6.1/9.0+0.42 ms cpu: the middle triple is mark assist, background and idle marking. A large assist number means request handlers are paying for the collector. 118->121->62 MB, 124 MB goal: heap when the cycle started, when it ended, and what was live after marking; then this cycle's goal. SCHED lines show gomaxprocs, idleprocs, threads and the global and per-P run queues. Queues that never drain mean goroutines are ready but no CPU is free; a threads count that keeps climbing means blocking system calls or cgo. Add scheddetail=1 only briefly: it prints every G, M and P on each tick.

data[:256] is only a new header over the multi-megabyte array io.ReadAll returned, so each 256-byte preview keeps the whole upload reachable: the map grows by the size of every file, not by 256 bytes. Why: re-slicing copies the slice header (pointer, length, capacity), never the bytes. The GC frees an allocation as a whole, so one live pointer into it keeps all of it, including the spare capacity ReadAll grew. The fix: bytes.Clone copies the 256 bytes into a small new array; data becomes garbage once saveUpload returns. Run with 100 uploads of 2 MiB, the live heap is roughly 200 MiB before the fix and under 1 MiB after. Same trap with strings: s[:8] of a large string shares its bytes too; strings.Clone makes the copy. How to find it: an inuse_space heap profile blames io.ReadAll, the allocation site, not the map that holds the memory. When a site that should be short-lived dominates the live heap, look for whoever keeps a subslice of it. How to test it: HeapAlloc after runtime.GC() stays flat over many uploads.

Only the last two hold a thread. A goroutine waiting on a channel or a socket is parked and its thread runs other goroutines; a blocking system call or cgo call keeps its thread for the whole wait, and the runtime hands that thread's P to another thread. **Channel, mutex, time.Sleep: pure runtime events: the goroutine is marked as waiting and taken off the thread, with no system call at all. Socket: network descriptors are non-blocking and registered with the netpoller** (epoll on Linux, kqueue on macOS, IOCP on Windows). A read that would block parks the goroutine; when data arrives the poller makes it runnable again. 20 000 idle connections are 20 000 parked goroutines and no extra threads. **Regular file, cgo**: disk files cannot be polled on Linux, so the thread sits inside read(2). If the call has not returned within a few tens of microseconds and other goroutines are waiting, sysmon gives its P to another thread so the rest of the program keeps running. The cost: each simultaneous blocking call is one more thread. Idle threads are kept for reuse rather than destroyed, and past 10 000 threads the process crashes (debug.SetMaxThreads changes the limit). So 20 000 goroutines reading files from a slow disk can mean 20 000 threads. Bound blocking system calls and cgo calls with a semaphore sized to what the disk or the library can really do in parallel.

The compiler inlines a function whose body costs at most 80 nodes of its syntax tree. The gain is not only a saved call: the inlined body is analysed again inside the caller, so a pointer that escapes from a small constructor can stay on the caller's stack. What blocks it: a body over budget, defer, a go statement, recover, or //go:noinline; -gcflags=-m=2 prints the reason, for example function too complex: cost 106 exceeds budget 80. Mid-stack inlining: since Go 1.12 functions that call other functions can be inlined too, so small wrappers and constructors disappear. Escape interaction: NewPoint returning &Point{} reports escapes to heap inside itself and does not escape at a call site where it was inlined. That is why tiny constructors cost nothing, and why //go:noinline in a benchmark can add an allocation. PGO: with a CPU profile saved as default.pgo in the main package (picked up automatically since Go 1.21), hot call sites get a much larger budget and hot interface calls are devirtualized; typical gains are a few percent of CPU. Other passes: constant folding, dead-code elimination, bounds-check elimination (-gcflags=-d=ssa/check_bce lists the checks that remain) and the SSA backend's register allocation. Inlining makes the binary larger; the budget is the compiler's guess at that trade-off. -gcflags=-m shows can inline NewPoint and inlining call to NewPoint.

Go's allocator, modelled on TCMalloc, rounds each small object up to one of about 68 size classes and serves it from a per-P cache without a lock. Only when that cache runs dry does it touch shared structures, and objects over 32 KB come straight from the page heap. The layers **mcache**: one per P, holding a span for each span class. Most allocations just take the next free slot in that span: no lock, no atomic. **mcentral**: one per span class (size class, with or without pointers), shared: hands a span with free slots to an mcache that ran out. **mheap: the page heap (8 KB pages): carves new spans out of memory from the OS and serves large objects directly. Tiny allocator**: pointer-free objects under 16 bytes are packed together into one 16-byte block. Fragmentation Each span holds objects of a single size, so a freed slot is exactly the right size for the next allocation of that class. External fragmentation stays low without compaction; the price is rounding waste inside each object, kept to about 12.5% by the choice of classes. Spans are also split by whether the type contains pointers, so the GC skips pointer-free memory entirely. Empty pages go back to the OS through the background scavenger. In practice: an allocation is cheap but not free (tens of nanoseconds, plus GC work later), a struct one byte over a size class wastes the gap, and pointer-free data is cheaper for the collector.

GC cost follows the allocation rate, not the heap size, so the fix is fewer allocations: find the top sites with -alloc_space, change them, and prove each change with allocs/op before and after. Preallocate what you can count: make([]T, 0, n) and make(map[K]V, n). A slice appended up to 10 000 elements reallocates and copies itself about twenty times on the way. Avoid hidden copies and boxing: string(b) and []byte(s) copy; fmt.Sprintf turns every argument into an interface. Use strconv.AppendInt, strings.Builder or a bytes.Buffer. Watch what escapes: interfaces and closures on the hot path often push values to the heap; -gcflags=-m shows which. Reuse scratch buffers with sync.Pool: get, Reset, put. Do not put back a buffer one huge request grew, or the pool keeps that memory alive. Prefer pointer-free layouts: []Point instead of []*Point is one allocation instead of n, and nothing for the GC to scan. Prove it twice: go test -bench . -benchmem over several runs compared with benchstat, then -alloc_space and GC CPU in production. If allocations cannot go lower, a higher GOGC buys CPU with memory.

buf.Bytes() returns a view of the pooled buffer's own memory, and the deferred Put gives that buffer to the next caller while this one still holds the view. Another request's Reset and Encode overwrite the bytes, so a response carries someone else's JSON. Why: Bytes() does not copy; the slice is valid only until the buffer is next modified. defer runs as encode returns, so the buffer is already back in the pool, and the next Get on the same P hands it out at once. Why it hides: with one caller nobody calls Get in between and the output is right, so unit tests pass. It shows up only under concurrency, mostly as mixed or truncated bodies and only rarely as a crash. The fix: copy out before the buffer goes back: bytes.Clone(buf.Bytes()) (or []byte(buf.String())), a single copy. That is one allocation of the exact size; the pool still saves the buffer's repeated growth. Better still, encode straight into the io.Writer and return nothing. How to catch it: go test -race with several goroutines that call encode and keep the results reports a data race between Reset or Write and the caller's read. Review rule: nothing taken from a pooled object (Bytes(), a subslice, a pointer to a field) may outlive the Put.

RSS is not the heap. It also counts goroutine stacks, runtime metadata and free heap pages the allocator kept instead of returning to the OS, while the heap profile shows only sampled live objects as of the last GC. **runtime/metrics splits it**: /memory/classes/heap/objects:bytes is live objects plus garbage not yet swept, /memory/classes/heap/free:bytes is free pages still mapped, /memory/classes/heap/stacks:bytes is goroutine stacks. Free pages: after a spike the heap shrinks but the pages stay with the process. The background scavenger returns them gradually; debug.FreeOSMemory() does it at once, which is fine after a known one-off spike and wasteful on a timer. Stacks: tens of thousands of goroutines, or a few that once recursed deeply, hold stack memory that never appears in a heap profile. Not Go memory at all: cgo allocations and memory-mapped files count in RSS and in none of the runtime's numbers. If /memory/classes/total:bytes is far below RSS, look there. **GOMEMLIMIT** makes the scavenger return memory more eagerly as the process nears the limit. The usual story is a peak, not a leak: one oversized request grew the heap and the process kept the pages. Cap request bodies and stream large payloads instead of reading them whole.

Confirm the live heap keeps growing across GC cycles, then diff two **inuse_space heap profiles taken minutes apart: what grew is the leak. alloc_space counts everything allocated since start; it explains GC CPU, not leaks. Confirm it**: the live heap after GC (gctrace's third heap number, or the /gc/heap/live:bytes metric) rises under steady load. A high but flat RSS is a different problem. **inuse_space**: bytes still live as of the last GC, by allocation site; the default view. Add ?gc=1 to the URL to collect first. **alloc_space: total bytes allocated since start. A service can churn gigabytes a second with a flat 30 MB live heap; read this view when GC burns CPU. Check goroutines too**: a goroutine count that grows with traffic is the most common leak, since every stuck goroutine holds its stack and everything it references. Since Go 1.27 there is a goroutineleak profile (/debug/pprof/goroutineleak; experimental behind GOEXPERIMENT=goroutineleakprofile in 1.26) that lists goroutines blocked on something nothing else can reach; it can miss leaks through globals or runnable goroutines. Usual causes: a map used as a cache with no eviction (maps never shrink), a subslice keeping a large backing array alive, goroutines blocked forever. A heap profile shows where memory was allocated, not who still holds it; from that site, follow the references to the owner.

Set **GOMEMLIMIT** a little below the container limit (about 1.8 GiB for a 2 GiB pod) and keep GOGC=100. It is a soft ceiling: as Go's total memory approaches it, the runtime collects more often instead of letting the heap grow into the OOM killer. What it counts: everything the Go runtime maps: heap, goroutine stacks and runtime metadata (/memory/classes/total:bytes minus released memory). Not cgo allocations, not the binary, not the page cache; hence the headroom. Soft: if the live heap truly exceeds the limit, nothing is killed by Go: the runtime collects almost continuously, the GC is capped near 50% of CPU, and the service crawls. Alert on GC CPU; do not treat the limit as capacity. **GOGC=off plus a limit: the heap grows freely up to the limit and is collected only there. Good for a batch job's throughput; for a service it means always running at the edge. Not automatic**: unlike GOMAXPROCS since Go 1.25, the runtime does not read the cgroup memory limit itself. Set the variable in the deployment or call debug.SetMemoryLimit at startup. The ballast was a pre-1.19 trick: a large, never-touched allocation that inflated the live heap so GOGC computed a bigger goal. It raised the floor and was never a ceiling, so a spike still went straight to OOM.

Mark assist makes a goroutine that allocates while marking is in progress do a share of the marking before its allocation returns. When the allocation rate outruns the background workers, request handlers spend their time marking instead of serving: throughput falls and latency explodes. The debt: the runtime spreads the remaining mark work over the bytes the program may still allocate before the heap goal. Allocate n bytes and you owe a proportional amount of scanning, paid on the spot. Where you see it: in the CPU triple of gctrace, 0.5+900/400/0+0.8 ms cpu, 900 ms is assist against 400 ms of background marking. The metric is /cpu/classes/gc/mark/assist:cpu-seconds; go tool trace shows assist slices inside your handlers. Why it spirals: assists slow handlers, requests queue, more requests in flight keep more memory live, and the next cycle starts sooner. The GC CPU limiter caps the collector near 50% of CPU, which saves the process but not the latency. The way out: cut the allocation rate on the hot path, since the debt is linear in the bytes allocated during a cycle. Then give the heap headroom with a higher GOGC or a GOMEMLIMIT that is not already binding, and check nothing forces cycles: a runtime.GC() in a handler, or a GOMEMLIMIT below the live heap. Limiting concurrency at the entrance also breaks the spiral: fewer requests in flight means a smaller live heap.

Nothing walks the heap with the world stopped: marking is concurrent, and the two pauses per cycle do a small fixed amount of work, usually well under a millisecond. What grows with the heap is the marking work, which is GC CPU, and the headroom GOGC asks for. Marking cost scales with the pointers to follow, not with bytes: 40 GB of map[string]*Entry is hundreds of millions of objects to visit every cycle, 40 GB of []byte is a handful. Headroom: at GOGC=100 the goal is about twice the live heap, 80 GB. For a big cache set GOMEMLIMIT near the machine's memory and raise GOGC, so cycles are rare. Pointer-free memory is never scanned: spans whose type has no pointers are skipped, and so are maps whose keys and values contain no pointers. That is the trick behind bigcache and freecache: one large byte arena plus a map of hashed keys to offsets. Writes during marking cost assists, so the write path should allocate little. So Go is a reasonable choice for a large cache if the data is stored without pointers; millions of small, pointer-rich objects are where it hurts.

Go 1.26 made Green Tea the default marking algorithm: instead of chasing small objects one by one across the heap, it queues whole spans (8 KB pages) and scans the marked objects on each page together. The knobs are unchanged; GC CPU typically drops 10–40% on allocation-heavy programs. Why it is faster: the classic marker jumps between objects in pointer order, so most of its time goes to cache misses. Working page by page gives memory locality, and on newer x86 CPUs it also scans with vector instructions. What it covers: small objects, which make up most heaps; large objects are marked the old way. What stays the same: concurrent tri-colour marking, the write barrier, GOGC, GOMEMLIMIT, the two short pauses. No code change is needed. Upgrading: compare GC CPU (/cpu/classes/gc/total:cpu-seconds), heap size and latency before and after. If something regresses, build with GOEXPERIMENT=nogreenteagc and report it: the Go 1.26 notes expect the opt-out to be removed in 1.27, so check whether your toolchain still accepts it. It was available as an experiment in Go 1.25 (GOEXPERIMENT=greenteagc). It does not change the advice: fewer allocations and fewer pointers still beat any collector improvement.

A CPU profile samples only goroutines that are running, so waiting never shows up in it. Waiting is in the block and mutex profiles, both off by default, and in the goroutine profile; time spent ready but not scheduled appears only in an execution trace. Block profile: runtime.SetBlockProfileRate(rate): where goroutines waited on channels, select, sync.Cond and locks. A rate of 1_000_000 ns samples about one event per millisecond spent blocked. Mutex profile: runtime.SetMutexProfileFraction(5): lock contention, attributed to the code that held the lock, which is the code you have to change. Goroutine profile: /debug/pprof/goroutine?debug=2 needs no setup: every stack with its state and how long it has waited. Four thousand goroutines in the same pool acquire is a queue, not a CPU problem. Execution trace: /debug/pprof/trace?seconds=5, opened with go tool trace: each goroutine's timeline, scheduler latency, system calls, GC phases and assists. It alone shows a goroutine ready to run with no P free, or a request stalled behind GC. Flight recorder (Go 1.25): trace.FlightRecorder keeps the last seconds of trace in memory, so you write it out when a slow request is detected instead of trying to catch one live. Both profiling rates cost something on a hot path: sample, do not set them to 1 everywhere.

A cgo call holds its OS thread for its whole duration and costs tens of nanoseconds, against about one for a Go call. The GC cannot see what C keeps, so Go memory C holds on to can be freed or reused, and a value on a goroutine stack moves when the stack grows. Cost and scheduling Each call switches to the system stack and marks the P as in a system call; a slow call loses its P to another thread via sysmon, so 1 000 concurrent slow C calls are 1 000 threads. Calls from C back into Go are far more expensive. The build pays too: cross-compiling needs a C toolchain, the binary is dynamically linked unless CGO_ENABLED=0, and C memory bugs are invisible to the race detector and the GC. Pointer rules Go memory passed to C is valid only for the duration of the call; C may keep it afterwards only if it is pinned with runtime.Pinner (Go 1.21+). The default cgocheck catches some violations at runtime. A uintptr is just an integer, not a reference: the GC may free the object and reuse its memory, and a value on a goroutine stack moves when the stack grows and is copied. Tests pass because nothing happens between storing the address and reading it. In production a GC cycle or a stack growth lands in between, and C writes into memory that now belongs to something else. If C must own a buffer for a long time, allocate it with C.malloc and free it explicitly.