Standard Library in Practice

encoding/json only sees exported fields: the ones whose name starts with a capital letter. A struct whose fields are all lowercase marshals to {} with a nil error. A tag on an unexported field does nothing; go vet reports struct field total has json tag but is not exported. To include the field you have to export it. json:"user_id" renames the key, json:"-" drops the field, omitempty skips empty values, and ,string carries a number inside a JSON string. On decode, a key is matched against the tag or the field name, falling back to a case-insensitive match. The trap is that nothing complains: {} is valid JSON, so a test that only checks err passes forever.

os.ReadFile loads a whole file into memory in one call and suits small files such as configs. For anything large or unbounded, os.Open the file and stream it as an io.Reader: line by line with bufio.Scanner, or in chunks with io.Copy. **os.ReadFile / os.WriteFile: open, read or write, and close in one call; memory use equals the file size. os.Open + bufio.Scanner**: one line at a time in constant memory. Check s.Err() after the loop: a read error also just ends it. **io.Copy(dst, src): moves bytes from any reader to any writer through a 32 KB buffer, and between files and sockets can hand the copy to the kernel. defer f.Close()** goes right after the error check. For a file you write, check the error from Close as well: a failed write can surface only there. os.Create truncates an existing file; to append, use os.OpenFile(name, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644).

bufio.Writer keeps your output in an in-memory buffer (4 KB by default) and writes to the underlying file only when the buffer fills or when you call **Flush**. Write copies into the buffer and returns n, nil, so whatever is still in the buffer when the function returns is lost. f.Close() closes the file, not the writer wrapped around it: Close knows nothing about the buffer. Call Flush after the loop and check its error: a failed write to the file is remembered and returned there. A bare defer w.Flush() throws that error away.

It prints "A" "65" 0 strconv.Atoi: parsing "65 ": invalid syntax. string(n) treats an integer as a Unicode code point, not as a number to format; strconv.Itoa and strconv.Atoi are the conversions you want, and Atoi rejects the trailing space. string(int) still compiles for historical reasons, but go vet (which go test also runs) reports conversion from int to string yields a string of one rune. Write string(rune(n)) when you really mean the character. strconv.Itoa / Atoi cover int; ParseInt(s, 10, 64), ParseFloat, ParseBool and FormatInt(n, 16) cover other sizes and bases. Parse errors are *strconv.NumError wrapping strconv.ErrSyntax or strconv.ErrRange, so errors.Is works on them. Nothing is trimmed for you: run strings.TrimSpace on input from users or files first.

A Go layout is an example of one reference time, Mon Jan 2 15:04:05 MST 2006, written the way you want the output to look. Anything that is not part of that timestamp is copied through literally, so YYYY-MM-DD formats as itself. The numbers are a mnemonic: 1 month, 2 day, 3 hour (15 for the 24-hour clock), 4 minute, 5 second, 6 year, -0700 the zone. Prefer the constants: time.RFC3339 for APIs, time.DateOnly and time.DateTime (Go 1.20) for plain dates. time.Parse with no zone in the layout returns UTC, not local time. time.ParseInLocation reads the same text as wall time in a zone you name.

//go:embed copies files from the package directory into the binary at build time: into a string, a []byte or an embed.FS. Code then reads them from memory, so the working directory of the process stops mattering. One file goes into a string or []byte; many files go into an embed.FS, which implements fs.FS, so template.ParseFS and http.FileServerFS (Go 1.22) take it directly. The directive sits right above a package-level variable, and the file imports embed (a blank _ import is enough for string and []byte). Patterns are relative to the package directory and cannot use ... A directory pattern skips files whose names start with . or _; prefix it with all: to include them. The bug it fixes: os.ReadFile("templates/x.html") resolves against the process's working directory. It works when you run from the directory that holds the files (a typical go run .) and fails under systemd or a container WORKDIR, where the working directory is somewhere else. Embedded files are read-only and fixed at build time: changing one means a rebuild.

crypto/rand is for anything an attacker must not guess: tokens, session ids, keys, reset links. math/rand/v2 is a fast, non-cryptographic generator for jitter, sampling, shuffling and simulations. math/rand output can be predicted by anyone who sees enough of it or knows the seed, so it never produces secrets. Seeding: since Go 1.20 the global math/rand functions are seeded randomly at start-up and rand.Seed is deprecated; since Go 1.24 calling it does nothing unless GODEBUG=randseednop=0. math/rand/v2 (Go 1.22) has no global Seed at all, renames Intn to IntN, adds the generic rand.N (works for time.Duration), and offers PCG and ChaCha8 sources when you need a reproducible stream. crypto/rand.Read(b) fills a byte slice from the OS; since Go 1.24 it never returns an error, and rand.Text() returns a ready 26-character token with 128 bits of randomness.

Every streaming type in the standard library speaks the same two one-method interfaces, **io.Reader and io.Writer**, so compression, hashing and counting are wrappers you stack on each other, and io.Copy drives the whole chain. Read(p []byte) (int, error) names no file or socket, so *os.File, net.Conn and a request body are interchangeable, and a test passes strings.NewReader("...") instead of a mock. io.TeeReader(r, w) copies everything read from r into w on the way past; io.MultiWriter(a, b) duplicates every write. io.Copy(dst, src) runs the loop with a 32 KB buffer, so memory stays flat whatever the file size. io.Pipe() connects a producer that writes to an API that wants an io.Reader, such as an HTTP request body, with nothing buffered in between. io.LimitReader(r, n) caps a stream but reports the cut as a plain io.EOF; read n+1 bytes when you need to detect it. Closing the wrappers is not optional: gzip.Writer.Close writes the trailer, and without it the archive is unreadable.

bufio.Scanner is a convenient tokenizer (a line or a word per Scan()), but it refuses any token over 64 KB (bufio.MaxScanTokenSize). bufio.Reader is a plain buffered reader with no such cap: you call ReadString('\n') or ReadBytes yourself. On a too-long token Scan() just returns false, and the reason is only in s.Err(). A loop without that check stops halfway and reports success. The cap is per token, not per file: gigabytes of short lines are fine, one 70 KB line is fatal. Raise it with s.Buffer(make([]byte, 0, 64*1024), 4*1024*1024) before the first Scan: the start size, then the maximum it may grow to. Reader.ReadString grows the line as needed and returns the last line together with io.EOF when the file has no trailing newline, so handle the data before the error. s.Bytes() points into the scanner's buffer and is overwritten by the next Scan; copy it if you keep it.

omitempty skips only false, 0, "", nil pointers and interfaces, and zero-length slices, maps and arrays; a struct is never empty to it, so a zero time.Time goes out as "0001-01-01T00:00:00Z". omitzero (Go 1.24) skips a field holding its type's zero value. omitzero also calls the type's IsZero() bool method when it has one, which is why it works for time.Time. omitempty on a number means 0 can never be sent: the client cannot tell a score of 0 from no score. When 0 is meaningful, use *int; nil is omitted, a pointer to 0 is sent as 0. With both options, json:",omitempty,omitzero", the field is dropped if either rule says so. Without omitempty, a nil slice encodes as null and an allocated empty one as []; a client that expects an array needs the second. Before Go 1.24 the workaround was a *time.Time field or a custom MarshalJSON.

It prints float64, then {"order_id":12345678901234568} (one off). Decoding into any turns every JSON number into a float64, which holds integers exactly only up to 2^53, so the id was rounded on the way in and the rounded value was written back out. Into an any the decoder produces only bool, float64, string, []any, map[string]any and nil. There is no integer case and no error. dec.UseNumber() decodes numbers as json.Number (a string holding the original text, with Int64() and Float64() methods), so large ids survive. A relay that only forwards should not decode the number at all: keep the payload as json.RawMessage and pass the original bytes on. Where you care about the value, decode into a struct with int64 fields; unknown keys are skipped for free. JavaScript clients have the same 2^53 limit, which is why APIs often send 64-bit ids as strings with json:",string".

Unknown keys are silently discarded, so the renamed key vanished and MaxConns kept its zero value; known keys are also matched case-insensitively, which hides other typos. A key that matches no field is parsed and thrown away, and the field keeps whatever the struct already held: for a fresh struct, zero. Matching tries the tag, then the field name, then a case-insensitive comparison, so {"MAX_CONNS":7} really does fill a field tagged json:"max_conns". dec.DisallowUnknownFields() turns the first case into an error; use it for config and internal APIs. Decode reads one value, so also check that the next Decode returns io.EOF. Fill the struct with your defaults before decoding, then validate: a config with MaxConns <= 0 should refuse to boot.

o is declared once, outside the loop, and json.Unmarshal sets only the fields present in the input. A line without coupon keeps the previous order's Coupon, so that order is billed with someone else's discount. Why: Unmarshal into an existing value does not zero it first: keys missing from the JSON leave their fields as they were. That is documented and handy for decoding over defaults, but here the "defaults" are the previous line. The fix: declare var o Order inside the loop, so every line decodes into a zero value; o = Order{} at the top of the loop does the same. A json.Decoder loop that calls Decode(&o) over NDJSON needs the same fix. Slices make it worse: for a []string field, Unmarshal resets the length to zero and reuses the backing array, so the next line overwrites the tags of an order already appended to out. A map field gets the new keys merged into the old ones. Catch it: no linter flags this. A table test where a line with every field is followed by a line without the optional ones does.

json.Marshal checks whether a type implements json.Marshaler before it looks at fields. type payload = Event is an alias (the same type with the same methods), and embedding it promotes MarshalJSON onto the anonymous struct, so the inner json.Marshal calls this method again, forever. Each call adds stack frames until Go's 1 GB stack limit, and the runtime stops with fatal error: stack overflow. It is not a panic: recover cannot catch it, and the whole process dies. The plain version, return json.Marshal(e) inside Event.MarshalJSON, loops the same way. The fix is one character: type payload Event without = is a defined type with Event's fields and tags but an empty method set, so marshalling it uses the default encoder. The outer At string field shadows the promoted At, so only that one field changes format. UnmarshalJSON goes on the pointer receiver and uses the same trick with *payload.

encoding/json/v2 arrived in Go 1.25 as an experiment behind GOEXPERIMENT=jsonv2: a rewrite with strict defaults and fast decoding. easyjson and sonic are chosen when a profile shows JSON eating CPU on a hot path. Stricter by default: duplicate keys are an error (v1 keeps the last), invalid UTF-8 is an error (v1 substitutes U+FFFD), and names match case-sensitively (v1 falls back to case-insensitive). Saner output: a nil slice encodes as [] and a nil map as {}, not null; omitempty means "omit if the value would encode as empty JSON". Streaming and options: MarshalWrite and UnmarshalRead work on an io.Writer / io.Reader, a type can implement MarshalJSONTo(*jsontext.Encoder) error to stream its output, and behaviour is tuned with options passed per call, not global settings. With the experiment on, v1 encoding/json runs on the new engine but keeps v1 semantics. easyjson generates code with no reflection, at the cost of regenerating on every struct change; sonic uses JIT and SIMD on amd64/arm64, at the cost of unsafe and platform limits. Until it leaves GOEXPERIMENT, keep encoding/json as the default in production code.

Package reflect lets code inspect and change values whose type is known only at run time: their kind, fields, struct tags and methods. encoding/json, fmt, text/template and database/sql scanning are built on it, which is how they handle your types without generated code. reflect.TypeOf(v) gives the static description (fields, tags via Field(i).Tag.Get("json")); reflect.ValueOf(v) gives the data, and Kind() says whether it is a struct, pointer, slice and so on. Speed: every value passes through an any, often a heap allocation, and fields are found at run time, so the compiler cannot inline or optimise. Expect it to be an order of magnitude slower than direct code; encoding/json caches per-type field lists to soften this. Safety: type errors move from compile time to panics: Elem on a non-pointer, SetInt on a string field, or setting an unexported or non-addressable value. To modify a struct you pass a pointer and call Elem(); CanSet() tells you first. Alternatives: generics for code that is the same across types, interfaces for behaviour, code generation (stringer, easyjson, sqlc) for hot paths. Keep reflection for genuinely open-ended types: serializers, validators, config loaders.

Go's regexp implements RE2: it matches with automata instead of backtracking, so time is linear in the input and no pattern or string can trigger catastrophic backtracking (ReDoS). The price is no backreferences and no lookahead or lookbehind. (\w+) \1 and foo(?=bar) do not compile: regexp.Compile returns invalid escape sequence for the first and invalid or unsupported Perl syntax for the second. Linear is not the same as fast: on simple patterns a backtracking engine can win, and the match semantics are leftmost-first like Perl (regexp.CompilePOSIX gives leftmost-longest). Compiling parses the pattern and builds a program, which costs far more than one match. A regexp.MustCompile inside a function pays that on every call; a package-level var re = regexp.MustCompile(...) pays it once, and *Regexp is safe for concurrent use. MustCompile panics on a bad pattern: right for a constant checked at start-up, wrong for a user-supplied one, where you call Compile and return the error. Many "regexes" in a codebase are really strings.HasPrefix, strings.Contains or strings.Cut, with nothing to compile at all.

Start from a struct of defaults, let environment variables override them and command-line flags override both, then validate the finished struct once at start-up and refuse to boot if anything is wrong. os.LookupEnv and flag cover this without a framework. os.Getenv returns "" both for an unset and an empty variable; os.LookupEnv tells the two apart. flag parses typed values (flag.Duration, flag.Int) and prints usage for free. Parse once in main, or use your own flag.FlagSet as below, never in init or inside a library. In containers, environment variables are the usual channel; a JSON file suits large structured config, decoded with DisallowUnknownFields. Collect every validation error and return them together with errors.Join, so one deploy shows all the mistakes. Pass the Config value down explicitly instead of reading a package-level global everywhere: tests then build their own.

net.Listen("tcp", addr) returns a listener; a loop calls Accept and hands each net.Conn to its own goroutine. For production you add deadlines so a silent client cannot hold a goroutine forever, and a shutdown that closes the listener, wakes idle handlers and waits for them. A net.Conn is an io.Reader and io.Writer, so bufio, io.Copy and io.ReadFull work on it unchanged. TCP is a byte stream with no message boundaries: one Write can arrive as several Reads. Frame messages with a delimiter or a length prefix. Deadlines are absolute times, not timeouts: call SetReadDeadline(time.Now().Add(d)) before each read, and set a write deadline too, or a client that stops reading holds the handler in Write. A timed-out read fails with an error for which errors.Is(err, os.ErrDeadlineExceeded) is true. Closing the listener makes Accept return net.ErrClosed, which is the signal to stop; then wait for the handlers with a sync.WaitGroup. Closing the listener does not stop running handlers: give them a quit signal or a read deadline set to the past, and bound the wait with a timeout. An untrusted client can send a never-ending line: cap the line length (Scanner.Buffer or a LimitReader per message) and limit concurrent connections if needed.

A time.Time from time.Now() carries two clocks: a wall-clock reading and a monotonic one. Sub, Since and Until use the monotonic reading whenever both values have it, and that clock only moves forward. The wall clock is what NTP, a DST change or a manual date moves. The monotonic one comes from the OS (CLOCK_MONOTONIC on Linux) and cannot be stepped, so time.Since(start) stays right across the jump. Anything that drops the monotonic reading leaves only wall-clock arithmetic: Round(0), UTC(), In, formatting and parsing back, MarshalJSON, a time read from a database, or converting with Unix()/UnixNano(). Subtracting such values is plain wall-clock arithmetic, which is why it went negative. So measure elapsed time with time.Since, and store instants as wall clock: RFC 3339 or Unix seconds. Never subtract two stored timestamps and call the result a latency. Compare two time.Time values with t.Equal(u): == also compares the monotonic reading and the location.

One json.Decoder over the reader decodes one value at a time in constant memory, and a json.RawMessage field parks the data bytes undecoded until type says which struct they belong in, so each line is decoded into Go values once. dec := json.NewDecoder(r), then for { err := dec.Decode(&ev); if err == io.EOF { break } ... }, not for dec.More(), because More() returns false on a read error and hides it. The decoder holds only its buffer and the current value, and skips the newlines between values itself. Unlike bufio.Scanner it has no token limit: a 10 MB line is fine, the decoder grows its buffer. Switch on ev.Type and unmarshal ev.Data into the right struct. A line you reject is written back out without ever being decoded into Go types. enc.Encode(v) writes one value followed by a newline; that is NDJSON already. Put a bufio.Writer under it and check the error from Flush. After a syntax error the decoder cannot resynchronise, so stop and report dec.InputOffset(); a type error in one record leaves the stream intact. io.ReadAll plus json.Unmarshal needs the whole 40 GB in memory, and decoding through map[string]any first builds Go values for every line twice and turns numbers into float64.

unsafe.Pointer converts to and from any pointer type and uintptr, letting code reinterpret memory and do pointer arithmetic outside the type system. unsafe.String makes a string over a slice's bytes without a copy, so when those bytes change, a supposedly immutable string changes with them. What it allows: *T to unsafe.Pointer to *U reads the same bytes as another type (math.Float64bits does exactly that), unsafe.Add does pointer arithmetic, Sizeof and Offsetof expose layout, and unsafe.Slice, SliceData, String and StringData (Go 1.17 and 1.20) convert between slices, strings and pointers. **The uintptr rule**: a uintptr is a plain number. The GC does not keep its target alive and stacks move when they grow, so convert it back to a pointer in the same expression; go vet reports possible misuse of unsafe.Pointer otherwise. Why the zero-copy string breaks: string(b) copies, and that copy is what makes strings immutable. A string from unsafe.String shares the buffer, so the next Read into it, or a sync.Pool buffer reused by another request, rewrites it silently: a map key changes after insertion (seen["alice"] is false in the example), and a cache shows another request's data. The other direction is worse: writing through unsafe.Slice(unsafe.StringData(s), len(s)) into a string literal hits read-only memory, and the process dies with fatal error: fault, which recover cannot catch. Usually none of it is needed: the compiler already avoids the copy in m[string(b)], string(b) == "x", switch string(b) and for range string(b). Reach for unsafe only when a profile shows the copy and nothing ever writes those bytes again.

The loop checks err before it uses n. gzip.Reader returns the last chunk of data together with io.EOF, so the loop breaks and drops those bytes, hashing a truncated stream without reporting an error. **The io.Reader contract**: Read may return n > 0 and a non-nil error from the same call, and the caller must process the n bytes before looking at err. gzip.Reader does exactly that at the end of the stream; os.File and bytes.Reader return 0, io.EOF on a separate call, which is why tests on plain files pass. The fix: write buf[:n] first, then check err. Simpler still: io.Copy(h, r) follows the contract for you, and so does io.ReadAll when the data fits in memory. Not only gzip: any reader may return data with an error, including third-party and wrapping readers, so a loop that returns on err != nil before using n is wrong in general, not just for this one source; bufio.Reader.ReadString has the same data-plus-EOF shape. Catch it: in tests, wrap the source in iotest.DataErrReader, which returns the final data together with io.EOF; check readers you write yourself with iotest.TestReader.