sync, Atomics & the Memory Model

sync has locks (Mutex, RWMutex), coordination tools (WaitGroup, Once, Cond) and two special containers (Map, Pool); sync/atomic adds lock-free operations on single values. Channels are the other tool, for handing data from one goroutine to another. **Mutex: one goroutine at a time inside a critical section; the default way to protect shared state. RWMutex: many readers or one writer; pays off only when reads are long and writes are rare. WaitGroup**: wait until a set of goroutines has finished; wg.Go(f) since Go 1.25. **Once, OnceValue: run initialisation exactly once, however many goroutines call it. Cond: sleep until a condition on locked state changes; rarely used, a channel usually does it better. Map**: a concurrent map for write-once or disjoint-key workloads, not a general replacement for map. **Pool: reuse short-lived objects such as buffers to cut allocations; the GC may empty it at any time. sync/atomic**: atomic.Int64, atomic.Bool, atomic.Pointer[T] for counters, flags and swapping one pointer. The zero value of each is ready to use, and none may be copied after first use (a copy is a separate, unrelated lock; go vet reports it as copylocks).

**wg.Add(1) runs inside the goroutine, so it can run after wg.Wait() has already seen a zero counter.** fetchAll then returns rows while the goroutines are still writing it: missing results, and a data race that -race reports. go func() only schedules the goroutine. The loop can start all of them and reach Wait before a single one has called Add. The rule: call Add on the goroutine that starts the work, before the go statement, and make defer wg.Done() the first line so an early return still decrements. Go 1.25 added wg.Go(f): it does the Add(1), the go and the deferred Done in one call, so there is nothing left to put in the wrong place. f must not panic. Since Go 1.25 go vet also flags this exact bug: WaitGroup.Add called from inside new goroutine. Capturing i and id in the closure is safe since Go 1.22: every loop iteration has its own variables.

**defer mu.Unlock() covers every exit**: the early return, the one someone adds next month, a panic in the middle. An explicit Unlock covers only the paths that existed when you wrote it. A return that jumps over the Unlock leaves the mutex locked forever, and every later Lock hangs. The price of defer is scope: it holds the lock until the function returns, which is wrong when the function goes on to make a network call or other slow I/O. Then split by hand: lock, copy what you need, unlock; do the slow work; lock again to store the result.

In a deadlock goroutines wait for each other and nothing moves; in a livelock they keep running but keep undoing each other's progress; in starvation the system moves, but one goroutine rarely or never gets the resource. Deadlock: two goroutines each hold a lock the other needs, or a goroutine waits on a channel nobody will ever send to. The CPU is idle. Go prints fatal error: all goroutines are asleep - deadlock! only when every goroutine is blocked; in a server with other live goroutines it just hangs silently. Livelock: two workers both detect a conflict, back off and retry in lockstep forever, for example with TryLock and the same fixed retry delay. The CPU is busy and nothing finishes. Random jitter in the back-off breaks the symmetry. Starvation: a greedy goroutine keeps winning the lock while another waits. sync.Mutex protects against it: once a waiter has waited over 1 ms, the mutex switches to starvation mode and hands the lock over in FIFO order. The fixes differ accordingly: one fixed lock order for deadlock, jitter for livelock, fair hand-off or shorter critical sections for starvation.

**The goroutine deadlocks on itself: sync.Mutex is not reentrant, so the Lock inside Get waits for an Unlock that can only run after it.** If nothing else runs, the runtime stops with fatal error: all goroutines are asleep - deadlock!; in a server every caller just hangs. A Go mutex has no owner. It does not record which goroutine locked it (another goroutine may even call Unlock), so it cannot tell a recursive call from a competing one. This is deliberate. While you hold the lock, the invariant it protects may be half-updated; a reentrant call would run on that broken state and hide the bug. The idiom: exported methods lock, an unexported helper does the work and documents that the caller already holds c.mu (often named getLocked). Locked methods never call other locked methods. TryLock (Go 1.18) returns false instead of blocking. It does not solve reentrancy, and its own documentation warns that needing it is often a sign of a deeper design problem. Unlocking an unlocked mutex is also fatal, not a recoverable panic: fatal error: sync: unlock of unlocked mutex.

**Publish holds mu while it does a blocking send on an unbuffered channel, so one subscriber that never reads keeps the mutex forever.** Publish sits in ch <- msg for the stuck client with mu locked; every Subscribe and every other Publish waits for that mutex, and the hub is dead. Nothing is wrong with the lock itself, and -race and go vet stay silent: the program is correct, it just waits for something that never happens. A critical section must be short and must not wait on anything you do not control: a channel send or receive, network or disk I/O, another lock, a callback into user code. The time spent under the lock is the time every other caller is blocked. The fix: copy what you need under the lock (slices.Collect(maps.Values(h.subs))), unlock, then send. Now mu guards only the map. The send itself must not block either: give each subscriber a buffered channel and use select with default (drop or disconnect a slow client), or use a per-subscriber goroutine with a context. Symptom to recognise: a goroutine dump (kill -QUIT, or /debug/pprof/goroutine?debug=2) shows many goroutines in sync.Mutex.Lock and one in chan send with the same lock held further up its stack. The copy can race with Unsubscribe: a message may arrive on a channel just after it was removed. Never close a subscriber channel from Publish; let its owner do it.

**Use sync.Once, or since Go 1.21 sync.OnceValue: the init function runs exactly once for all callers, and every caller returns only after it has finished.** A plain nil check gives neither guarantee. With if client == nil { client = newClient() } two goroutines can both see nil and both build a client. It is also a data race, so a reader can see the pointer before the fields behind it. Once also publishes the result: everything f wrote is visible to every goroutine whose Do has returned. Do counts the call as done even if f panics, so a failed init never retries. OnceValue and OnceValues re-panic with the same value on every call. OnceValues caches the error forever. For init that should retry after a failure, use a mutex and a nil check instead of Once. When the init is cheap and cannot fail, a package-level var x = newX() is simpler still: package initialisation runs once, before main.

A data race is two goroutines touching the same memory at the same time, at least one writing, with no synchronisation; a race condition is a logic bug where the result depends on timing. -race finds only the first, so code where every access is locked can still be wrong. if a.Balance() >= n { a.add(-n) } takes the lock twice. Two goroutines both pass the check on a balance of 100 and both withdraw 80: every access was locked, the pair was not. Look for check-then-act split across two lock acquisitions: if !exists { create }, or a read through a getter followed by a write through a setter. Fix it by deciding and mutating under one hold of the lock: a Withdraw method on the type, not two locked accessors that a caller combines. A lock protects an invariant, not a field.

A data race does not just hand you a stale value: the compiler and the CPU may produce results that no sequence of stale reads can explain. Nothing forces a reload. for !ready {} may load ready into a register once and spin forever. Nothing orders payload = build() against ready = true, so a reader that sees ready == true can still see payload == nil and crash. On arm64 the CPU reorders the two stores too, which is why such bugs show up only on the production fleet. The no-tearing argument holds only for word-sized values. An interface, a slice header or a string can be read half from one write and half from another, and a torn interface can crash the program. The fix: stop with ctx.Done() or close(stop); for a plain flag use atomic.Bool (stop.Store(true) / stop.Load()); to hand over data, publish it through one atomic.Pointer[T]. atomic.Bool costs one instruction on the read path. There is no performance argument for the racy version.

An atomic makes one read, write or read-modify-write of a single value indivisible and visible to other goroutines, without a lock. Use it for a counter, a flag or swapping one pointer; switch to a Mutex as soon as two values, or a check and an update, must change together. API: since Go 1.19 the typed atomic.Int32, atomic.Int64, atomic.Uint64, atomic.Bool and atomic.Pointer[T], with Load, Store, Add, Swap and CompareAndSwap. Prefer them to atomic.AddInt64(&x, 1): no accidental plain reads, and 64-bit alignment is handled for you. Cost: an uncontended atomic is a single instruction, cheaper than a Lock/Unlock pair, and it never parks the goroutine. Under heavy contention it still bounces one cache line between cores. Limits: each operation is atomic, a sequence is not: Load then Store loses updates. Compound logic needs a CompareAndSwap retry loop, and anything spanning several fields needs a lock. Ordering: Go atomics are sequentially consistent: a Store happens before the Load that observes it, so an atomic can publish other data. There are no relaxed or acquire/release variants to tune. Lock-free structures: built from such CAS loops; rarely worth writing yourself. Default to a mutex; reach for atomics when the state really is one word, or when a profile shows the lock.

Each operation is atomic; the pair is not. A read-modify-write has to be one step, which means CompareAndSwap in a retry loop. Two goroutines with 90 and 100 both load the old max of 50 and both pass the compare. The 100 lands, then the 90 overwrites it. No memory access raced, so -race has nothing to report. CompareAndSwap(old, new) writes only if the value still equals old. If someone moved it, you Load again and decide again; the loop is the algorithm, and the same loop is the building block of every lock-free structure. Add and Swap are already single steps, so a plain counter needs no loop. Prefer the Go 1.19 types such as atomic.Int64 over atomic.AddInt64(&x, n) on a plain field: they stay 64-bit aligned on 32-bit platforms, where a misaligned int64 makes the old functions panic.

Mutex lets one goroutine in at a time; RWMutex lets many readers in together and gives writers exclusive access. But every RLock and RUnlock is an atomic update of one shared reader counter, so for a short read the readers still fight over one cache line and nothing is gained. RWMutex pays off when readers hold the lock long enough to overlap (walking a large structure, building a response) and writes are rare. For a single map lookup it is no faster than Mutex, and it can be slower. Writers pay more: Lock takes an internal mutex and then waits for every active reader to leave. A waiting writer blocks new readers, so writers cannot starve. The consequence: recursive read locking can deadlock. Goroutine A holds RLock, a writer queues, A calls RLock again and waits behind the writer forever. Measure both before choosing, across core counts: go test -bench . -cpu=1,4,8,16. When one lock is the bottleneck either way, stop sharing it: shard the map, or publish an immutable snapshot through atomic.Pointer[T].

A Mutex is two fields: a state word (locked, woken and starving bits plus a waiter count) and a semaphore. Lock first tries one compare-and-swap from 0 to locked; only if that fails does it spin briefly and then park the goroutine on the semaphore queue. Fast and slow path Fast path: an uncontended Lock is a single CAS, inlined into the caller; Unlock is a single atomic add. Spinning: under contention it spins up to 4 short rounds, and only on a multicore machine with other Ps running and an empty local run queue, betting that the holder is about to release. Parking: after that the goroutine sleeps on the semaphore. The scheduler runs other goroutines; no OS thread is blocked. Normal mode Waiters queue in FIFO order, but a woken waiter has to compete with newly arriving goroutines, which are already on a CPU and usually win. That gives high throughput (a running goroutine can take the lock several times in a row) at the price of unfair tail latency. Starvation mode If a waiter fails to get the lock for more than 1 ms, the mutex switches mode: Unlock hands ownership directly to the first waiter, and newcomers neither spin nor grab the lock but queue at the tail. It switches back to normal when a waiter gets the lock and is either the last one in the queue or waited less than 1 ms.

sync.Map is a concurrent map tuned for the two cases its documentation names: keys written once and read many times, like a cache that only grows, and goroutines working on disjoint sets of keys. For everything else the docs recommend a plain map with your own lock. What you give up: keys and values are any, so every read needs a type assertion and values get boxed. There is no Len: counting means Range, which walks everything and is not a consistent snapshot. Invariants: a lock lets you keep the map consistent with something else: a size counter, a second index, check-then-insert across several keys. sync.Map offers only per-key atomic operations such as LoadOrStore, LoadAndDelete, Swap and CompareAndSwap. Where it wins: read-mostly access from many cores: loading an existing key takes no lock, so it scales where the shared reader counter of one RWMutex does not. Go 1.24: the implementation was rewritten as a concurrent hash-trie, so heavy insert and delete churn no longer pays the old penalty. Performance is less of an argument against it now; type safety and the missing Len still are. When one lock is too hot and you still want types and a count, use a sharded map: N plain maps, each with its own RWMutex, chosen by key hash.

-race instruments every memory access at compile time and, at run time, tracks which accesses are ordered by locks, channels and atomics. Two accesses to the same address, at least one a write, with no ordering between them are reported as WARNING: DATA RACE with both stacks. Dynamic only: it sees races only between accesses that actually happened in that run. A test with one goroutine, or a path behind a flag that is off in tests, finds nothing, so a green run is not a proof. Limited history: it remembers only the last few accesses to each memory word, so two accesses far apart in time can slip through. Not its job: deadlocks, goroutine leaks and logic races such as check-then-act across two locks are invisible to it. Cost: the Go docs put it at 5–10x memory and 2–20x CPU, and it builds a different binary. It belongs in CI and perhaps one staging canary, never on the production fleet. Make the tests produce the concurrency: many goroutines on the same key, go test -race -count=10 on the suspect package, and GORACE=halt_on_error=1 to stop at the first report.

A Cond lets goroutines sleep until locked state changes: Wait atomically unlocks L and parks, Signal wakes one waiter, Broadcast wakes all, and Wait re-locks before it returns. A channel does the same job and also works inside select, which a Cond cannot. Always re-check the predicate in a for loop, never an if: by the time Wait has re-taken the lock, another goroutine may have consumed what you were woken for. A Cond cannot appear in a select, so a Wait cannot have a timeout, a ctx.Done() or a shutdown case. Teams then add a goroutine that calls Broadcast on a ticker: a busy poll dressed up as synchronisation. The translation is direct: Signal is a send on a channel, Broadcast is closing one, and a bounded queue is a buffered channel. Cond still fits when many goroutines wait on a complex predicate over shared state and get woken again and again.

**Snapshot returns the registry's own map, not a copy, so the read lock protects only the return statement.** The caller ranges over the map with no lock held while Add writes to it, and the runtime aborts with concurrent map iteration and map write. A map, like a slice or a pointer, refers to shared data: returning it hands the caller exactly what the mutex guards. RUnlock runs on return, and every read after that is unsynchronised. The fix copies while the lock is still held: maps.Clone(r.peers) (Go 1.21+). The caller owns its copy and can range, sort or modify it freely. A mutex guards accesses, not values: any getter on a locked type that returns a map, a slice or a pointer into guarded state leaks it out of the critical section. Review such getters for exactly this. Cloning costs O(n) per call. For a large map read far more often than written, keep an immutable map in an atomic.Pointer and have Add (still under the mutex) publish a fresh copy, so Snapshot is a single atomic load. -race catches it when a test ranges over Snapshot() while another goroutine calls Add. A leaked slice fails the same way but silently, with no runtime crash.

Pool keeps per-processor free lists of objects you return with Put, so the next Get reuses one instead of allocating; that cuts GC pressure from short-lived objects such as encode buffers. It is not a cache: the GC drops its contents within two cycles, and it can keep oversized objects alive. How it works Each P has its own local list, so Get and Put on the current P take no lock; an empty P steals from the others before calling New. Each GC moves the pool's contents into a victim cache and the next GC drops them, so an object survives at most two cycles. When it hurts Low traffic: two GCs run between requests and the pool is emptied, the hit rate drops to near zero and New becomes the normal path. No harm, but no gain either. Unbounded size: a returned buffer keeps the capacity it grew to. One huge request parks a huge buffer per P until two GCs pass; drop anything over a size limit instead of calling Put. Stale data: Get may return a used object, so reset it right after Get, not only before Put. Wrong objects: connections or anything that needs Close or a hard limit: items vanish without notice. Use a real pool, such as the one inside database/sql. Values instead of pointers: Put of a non-pointer boxes it into an interface and allocates, defeating the point. Add a pool only after -benchmem or an allocation profile shows the allocations matter.

Happens-before is the ordering the memory model guarantees: if write w happens before read r, r sees w or a later write. Inside one goroutine program order provides it; between goroutines only channels, locks, Once, WaitGroup and atomics do. Channels: a send happens before the matching receive completes; a close before a receive that returns because the channel is closed; on an unbuffered channel the receive happens before the send completes. The k-th receive on a channel of capacity C happens before the (k+C)-th send completes, which is why a buffered channel works as a semaphore. sync: Unlock happens before the next Lock returns; f in once.Do(f) before any Do returns; Done before the Wait it unblocks; an atomic Store before a Load that observes it. Goroutines: the go statement happens before the goroutine starts, but a goroutine's exit is ordered before nothing. "It must be done by now", time.Sleep and runtime.Gosched are not edges. Why it bites: without an edge the compiler may reorder or cache two plain writes, and arm64 CPUs commit stores out of order. That is how code correct on a laptop panics once a week on an arm64 fleet. The contract: a program with no data races behaves as if sequentially consistent. A racy program loses that guarantee; Go limits the damage, but you cannot reason about it. go test -race checks exactly this relation: a report means two accesses with no edge between them.

**The outer cfg == nil check is an unsynchronised read of a pointer another goroutine writes, which is a data race that can expose a half-built struct. Publish the config through atomic.Pointer[Config]: build a fresh value, Store it, readers Load.** The mutex orders only the inner check and the write; a reader that never takes the lock is outside that ordering entirely. Publication is the dangerous half. load() allocates a Config, fills the fields, then stores the pointer; with no ordering point the pointer can become visible before the fields, and a request reads a zero Rate. Load() on the request path is one instruction and no lock, and Store(c) publishes every field reachable through the pointer. The contract that comes with it: build a fresh *Config, publish it, never mutate it after the Store. A reload that fails keeps serving the old value. RWMutex around the pointer also works, but every request then pays for the shared reader counter. Built once and never reloaded? That is sync.OnceValue or sync.OnceValues.

**A lock-order inversion: transfer(x, y) locks x then y while transfer(y, x) locks y then x, so each waits for the lock the other holds.** The fix is one fixed order that every acquisition path obeys. Go's deadlock detector fires only when every goroutine is blocked, so a server with hundreds of healthy goroutines never reports it. It looks like a slow leak of stuck requests; send SIGQUIT or read /debug/pprof/goroutine?debug=2 and look for goroutines parked in sync.Mutex.Lock. Impose the order with a key the objects already have, such as the account id, and always lock in ascending order. The two goroutines then queue behind the same first lock instead of forming a cycle. Handle a == b separately: with one account the code would lock the same non-reentrant mutex twice. The structural rule that prevents most of these: never call out of your package while holding a lock, because that is how an unknown second lock gets taken inside your critical section.

False sharing: neighbouring counters sit inside one cache line, and CPU caches stay coherent per line, not per variable. A write to one element invalidates the whole line in every other core's cache, its neighbours included. A core cannot own half a line. The atomic Add takes the whole line exclusively and knocks out the copies other cores held for their own counters. The elements are logically separate but physically adjacent, so the more cores write, the harder the line bounces. Per-operation cost that rises with -cpu=1,2,4,8,16 is the signature, and -race stays silent because nothing actually races. Pad each element to a full cache line, and pad the element type, not the slice, so make([]padded, n) lays them out one per line. Padded, the same benchmark stays flat as cores are added. Often the better fix is to stop sharing: count in a local variable and add to the shared counter once at the end.