Logs, metrics and traces: what question does each one answer that the other two cannot?

**Metrics** tell you *that* something is wrong and by how much, **traces** tell you *where* in a request the time or the error went, and **logs** tell you *exactly what happened* in one event.

What is structured logging, how does log/slog do it, and which log level do you use for what?

Structured logging writes each event as a **fixed message plus typed key/value fields**, so the log store can filter and count by field instead of grepping text; `log/slog` (Go 1.21+) does this in the standard library.

How does Prometheus get metrics out of a Go service, and which runtime metrics does client_golang expose without any code from you?

Prometheus **pulls**: every scrape interval (typically 15–30 s) it sends `GET /metrics` to each target, and `promhttp` answers with the current value of every metric as text.

What is the difference between a counter, a gauge, a histogram and a summary, and which one do you use for request latency?

A **counter** only goes up and is read through `rate()`; a **gauge** goes up and down and is read as is; a **histogram** counts observations into buckets so percentiles can be computed across pods; a **summary** computes percentiles inside one process.

What are a trace, a span and a trace id, and how does the trace id travel from one service to the next?

A **trace** is the whole journey of one request: a tree of **spans**, each one a timed operation (a handler, a query, an outgoing call) with attributes and a status.

What are SLI, SLO and SLA, and what is an error budget?

An **SLI** is a measured ratio of good events (say, requests answered without a 5xx in under 300 ms).

What do RED and USE stand for, and which of the two do you instrument first on an HTTP service compared with a queue worker?

On an HTTP service start with **RED** (Rate, Errors, Duration) per route; on a queue worker start with **USE**-style saturation (queue depth and age of the oldest message) and add RED for the jobs it processes.

Handlers repeat "request_id", rid in every slog call, and lines from deeper packages lack it. How do you fix that, and what is the trap with a logger in Context?

Bind the request's attributes once with **`logger.With(...)`** and pass that child logger down.

slog.Info("auth ok", "user", u) put a live bearer token into the log store, and that struct is logged from a dozen places. Where do you fix it once?

Give the type a **`LogValue() slog.Value`** method: `slog.LogValuer` is resolved by the handler at output time, so every call site that logs that value emits the redacted form.

One failing DB call produces five ERROR lines and five error-counter increments, one per layer. What is the rule, and where does the single log line belong?

**Handle an error once: either log it or return it, never both.** Each layer adds context by wrapping with `fmt.Errorf("load order %s: %w", id, err)`, and the outermost boundary writes the single record.

A request returned 500, yet its span in the trace is green. What did the code skip, and how is span.RecordError different from span.SetStatus?

**`span.RecordError(err)` only adds an exception event; the span's status stays `Unset` until you call `span.SetStatus(codes.Error, err.Error())`**, and the status is the field a backend colours, filters and alerts on.

In the trace, the repository's query span shows up as a sibling of GetOrder instead of its child, and failed lookups never appear at all. Why?

Two bugs: the `ctx` returned by `tracer.Start` is **discarded**, so the repository's span hangs off `GetOrder`'s parent; and `span.End()` runs only on success, so every failed lookup leaves a span that is **never ended and never exported**.

Both services run the OpenTelemetry SDK, yet the downstream one starts a brand-new trace for every request the gateway sends it. What is missing?

**The Go SDK's default propagator is a no-op**: nothing writes or reads the W3C `traceparent` header until you call `otel.SetTextMapPropagator(...)`, in both processes.

What do liveness, readiness and startup probes each decide, and what should each one check in a Go service?

**Liveness** decides whether the container is restarted, **readiness** whether the pod receives traffic, and a **startup** probe holds both off until slow initialisation is done.

Every rolling deploy drops a few requests although the service calls srv.Shutdown on SIGTERM. What is Kubernetes doing at that moment, and how do you wait it out?

**`SIGTERM` and the pod's removal from the Service endpoints happen at the same time, not in sequence**, and the removal takes seconds to reach kube-proxy and the load balancers, which keep sending traffic to a server `Shutdown` has already closed.

A user_id label on http_requests_total left Prometheus with four million series and timing-out queries. Why does one label cost so much, and how do you bound it?

**Every distinct combination of label values is a separate time series**, with its own index entries and its own chunks on disk, so a `user_id` label multiplies the whole metric by the number of users.

A 20-second database blip became a four-minute outage of a service whose readiness probe pings that database. Where did the other three and a half minutes come from?

**Because every replica failed the probe at once, the Service had no endpoints**; it came back only after the probes passed again and the pods had reconnected their pools and warmed up, and the reconnect stampede slowed the database that had just recovered.

You sample 1% of traces head-based, and a customer's 9-second checkout is missing from the backend. What can 1% answer, and what would tail sampling change?

**A head sampler decides at the root span, before anything interesting has happened**, so a rare slow request survives only by luck: 1% answers questions about the shape of traffic, never "this one request".

p99 tripled at 14:05 with flat CPU and no deploy. How do you get from the dashboard to the slow code path in production, and what must already be in place for pprof?

**A tripled p99 at flat CPU means waiting, not computing**: narrow with the labels you already have, open one trace from the slow bucket, then profile block, mutex or goroutine, not CPU.

You get paged at 3am for CPU > 80% and p99 > 500ms alerts that clear before anyone looks, yet a real two-hour degradation paged nobody. How do you redesign alerting?

**Page on a symptom the user feels, measured as a burn rate against an SLO, and never on a cause.** An alert must answer whether the error budget is being spent faster than you can afford: a rate over a window, not a value at an instant.

Irbisa · Cheatsheet

Observability & Production Ops

Q: Find the bug: failed charges show up as ERROR lines, but a log search on the err field never finds them.

`ErrorContext` gets `err` without a key, so `slog` writes it under the placeholder key `!BADKEY`: the line is there, but it has no `err` field to search, filter or alert on.

slog, Prometheus, OpenTelemetry, probes, SLOs, debugging a p99 spike

Senior Developer · 24 questions

01
Logs, metrics and traces: what question does each one answer that the other two cannot?
Metrics tell you that something is wrong and by how much, traces tell you where in a request the time or the error went, and logs tell you exactly what happened in one event.
02
What is structured logging, how does log/slog do it, and which log level do you use for what?
Structured logging writes each event as a fixed message plus typed key/value fields, so the log store can filter and count by field instead of grepping text; log/slog (Go 1.21+) does this in the standard library.
```
// wrong: the fields are baked into the sentence, so every question is a grepfunc placeOld(id, tenant string, err error) {	log.Printf("order %s failed for tenant %s: %v", id, tenant, err)} var level = new(slog.LevelVar) // Leveler: flip at runtime, no redeploy…
```

Find the bug: failed charges show up as `ERROR` lines, but a log search on the `err` field never finds them.

func (s *Server) handleCharge(w http.ResponseWriter, r *http.Request) {	orderID := r.PathValue("id")	log := s.log.With("request_id", requestID(r)) 	if err := s.payments.Charge(r.Context(), orderID); err != nil {		log.ErrorContext(r.Context(), "charge failed", "order_id", orderID, err)		http.Error(w, "payment failed", http.StatusBadGateway)		return	}	log.InfoContext(r.Context(), "charge ok", "order_id", orderID)	w.WriteHeader(http.StatusNoContent)}

ErrorContext gets err without a key, so slog writes it under the placeholder key !BADKEY: the line is there, but it has no err field to search, filter or alert on.

04
How does Prometheus get metrics out of a Go service, and which runtime metrics does client_golang expose without any code from you?
Prometheus pulls: every scrape interval (typically 15–30 s) it sends GET /metrics to each target, and promhttp answers with the current value of every metric as text.
```
func newMetrics() (*prometheus.Registry, *prometheus.CounterVec) {	reg := prometheus.NewRegistry() // empty: nothing is implicit here 	reg.MustRegister(		collectors.NewGoCollector(			collectors.WithGoCollectorRuntimeMetrics(collectors.MetricsScheduler),…
```
05
What is the difference between a counter, a gauge, a histogram and a summary, and which one do you use for request latency?
A counter only goes up and is read through rate(); a gauge goes up and down and is read as is; a histogram counts observations into buckets so percentiles can be computed across pods; a summary computes percentiles inside one process.
```
var (	requests = promauto.NewCounterVec(prometheus.CounterOpts{		Name: "http_requests_total",		Help: "Requests by route and status class.",	}, []string{"route", "status"})…
```
06
What are a trace, a span and a trace id, and how does the trace id travel from one service to the next?
A trace is the whole journey of one request: a tree of spans, each one a timed operation (a handler, a query, an outgoing call) with attributes and a status.
```
func initTracing(ctx context.Context) (func(context.Context) error, error) {	exp, err := otlptracegrpc.New(ctx) // endpoint from OTEL_EXPORTER_OTLP_ENDPOINT	if err != nil {		return nil, err	}	tp := sdktrace.NewTracerProvider(…
```
07
What are SLI, SLO and SLA, and what is an error budget?
An SLI is a measured ratio of good events (say, requests answered without a 5xx in under 300 ms).
08
What do RED and USE stand for, and which of the two do you instrument first on an HTTP service compared with a queue worker?
On an HTTP service start with RED (Rate, Errors, Duration) per route; on a queue worker start with USE-style saturation (queue depth and age of the oldest message) and add RED for the jobs it processes.
09
Handlers repeat "request_id", rid in every slog call, and lines from deeper packages lack it. How do you fix that, and what is the trap with a logger in Context?
Bind the request's attributes once with logger.With(...) and pass that child logger down.
```
type ctxKey struct{} func withLogger(ctx context.Context, l *slog.Logger) context.Context {	return context.WithValue(ctx, ctxKey{}, l)}…
```
10
slog.Info("auth ok", "user", u) put a live bearer token into the log store, and that struct is logged from a dozen places. Where do you fix it once?
Give the type a LogValue() slog.Value method: slog.LogValuer is resolved by the handler at output time, so every call site that logs that value emits the redacted form.
```
type Token string // Resolved by the handler at output time, everywhere this value is logged.func (Token) LogValue() slog.Value { return slog.StringValue("REDACTED") } type User struct {…
```
11
One failing DB call produces five ERROR lines and five error-counter increments, one per layer. What is the rule, and where does the single log line belong?
Handle an error once: either log it or return it, never both. Each layer adds context by wrapping with fmt.Errorf("load order %s: %w", id, err), and the outermost boundary writes the single record.
```
// wrong: logged here, and again by every caller abovefunc (r *Repo) load(ctx context.Context, id string) (Order, error) {	var o Order	if err := r.db.QueryRowContext(ctx, q, id).Scan(&o.ID); err != nil {		slog.ErrorContext(ctx, "query failed", "err", err)		return Order{}, err…
```

This handler serves the first request and fails every one after it. What happens, and where should the counter be created?

func (s *Server) handleOrder(w http.ResponseWriter, r *http.Request) {	orders := prometheus.NewCounter(prometheus.CounterOpts{		Name: "orders_created_total",		Help: "Orders created.",	})	prometheus.MustRegister(orders) 	if err := s.create(r.Context(), r); err != nil {		http.Error(w, "create failed", http.StatusInternalServerError)		return	}	orders.Inc()	w.WriteHeader(http.StatusCreated)}

The second call to prometheus.MustRegister panics with duplicate metrics collector registration attempted, because a counter with that name is already in the default registry.

How do you choose histogram buckets for request latency, and why does the default top bucket distort p99?

Put bucket boundaries around the SLO and past the timeout: the quantile is interpolated inside a bucket, so with the default top bound of 10 s a bad p99 reads as about ten seconds.

var latency = prometheus.NewHistogramVec(prometheus.HistogramOpts{	Name: "http_request_duration_seconds",	// Boundaries around the SLO (300ms) and out past the timeout (10s),	// so the p99 does not saturate at the last bucket.	Buckets: []float64{.005, .01, .025, .05, .1, .2, .3, .5, 1, 2.5, 5, 10, 30},…

Find the bug: the checkout latency histogram puts every request in the lowest bucket, including the ones clients time out on.

var checkoutDuration = promauto.NewHistogram(prometheus.HistogramOpts{	Name:    "checkout_duration_seconds",	Help:    "Checkout handler latency.",	Buckets: []float64{.05, .1, .3, .5, 1, 2.5, 5, 10},}) func (s *Server) handleCheckout(w http.ResponseWriter, r *http.Request) {	start := time.Now()	defer checkoutDuration.Observe(time.Since(start).Seconds()) 	if err := s.checkout(r.Context(), r); err != nil {		http.Error(w, "checkout failed", http.StatusInternalServerError)		return	}	w.WriteHeader(http.StatusCreated)}

The arguments of a deferred call are evaluated at the defer statement, so time.Since(start) is taken right after start is set: every request records about zero seconds, the p99 looks perfect and a latency alert can never fire.

15
A request returned 500, yet its span in the trace is green. What did the code skip, and how is span.RecordError different from span.SetStatus?
span.RecordError(err) only adds an exception event; the span's status stays Unset until you call span.SetStatus(codes.Error, err.Error()), and the status is the field a backend colours, filters and alerts on.
```
var tracer = otel.Tracer("github.com/acme/orders") func (s *Service) Load(ctx context.Context, id string) (Order, error) {	// Attributes at Start are visible to the sampler; later ones are not.	ctx, span := tracer.Start(ctx, "OrderService.Load",		trace.WithAttributes(attribute.String("order.id", id)))…
```
16
In the trace, the repository's query span shows up as a sibling of GetOrder instead of its child, and failed lookups never appear at all. Why?
```
func (s *Service) GetOrder(ctx context.Context, id string) (Order, error) {	_, span := tracer.Start(ctx, "GetOrder") 	o, err := s.repo.Find(ctx, id) // repo starts its own "db.query" span from ctx	if err != nil {		return Order{}, fmt.Errorf("get order %s: %w", id, err)	} 	span.End()	return o, nil}
```
Two bugs: the ctx returned by tracer.Start is discarded, so the repository's span hangs off GetOrder's parent; and span.End() runs only on success, so every failed lookup leaves a span that is never ended and never exported.
17
Both services run the OpenTelemetry SDK, yet the downstream one starts a brand-new trace for every request the gateway sends it. What is missing?
The Go SDK's default propagator is a no-op: nothing writes or reads the W3C traceparent header until you call otel.SetTextMapPropagator(...), in both processes.
```
func initPropagation() {	// Without this the SDK propagates nothing at all.	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(		propagation.TraceContext{}, // W3C traceparent / tracestate		propagation.Baggage{},	))…
```
18
What do liveness, readiness and startup probes each decide, and what should each one check in a Go service?
Liveness decides whether the container is restarted, readiness whether the pod receives traffic, and a startup probe holds both off until slow initialisation is done.
```
type Probes struct {	started  atomic.Bool  // set once config, caches and pools are ready	draining atomic.Bool  // set on SIGTERM	lastTick atomic.Int64 // unix nanos, set at start and bumped by the worker loop}…
```
19
Every rolling deploy drops a few requests although the service calls srv.Shutdown on SIGTERM. What is Kubernetes doing at that moment, and how do you wait it out?
SIGTERM and the pod's removal from the Service endpoints happen at the same time, not in sequence, and the removal takes seconds to reach kube-proxy and the load balancers, which keep sending traffic to a server Shutdown has already closed.
20
A user_id label on http_requests_total left Prometheus with four million series and timing-out queries. Why does one label cost so much, and how do you bound it?
Every distinct combination of label values is a separate time series, with its own index entries and its own chunks on disk, so a user_id label multiplies the whole metric by the number of users.
```
func metrics(next http.Handler) http.Handler {	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {		rec := &statusRecorder{ResponseWriter: w, status: http.StatusOK}		next.ServeHTTP(rec, r) 		route := r.Pattern // set by the mux while routing: read it after next…
```
21
A 20-second database blip became a four-minute outage of a service whose readiness probe pings that database. Where did the other three and a half minutes come from?
Because every replica failed the probe at once, the Service had no endpoints; it came back only after the probes passed again and the pods had reconnected their pools and warmed up, and the reconnect stampede slowed the database that had just recovered.
22
You sample 1% of traces head-based, and a customer's 9-second checkout is missing from the backend. What can 1% answer, and what would tail sampling change?
A head sampler decides at the root span, before anything interesting has happened, so a rare slow request survives only by luck: 1% answers questions about the shape of traffic, never "this one request".
```
func provider(exp sdktrace.SpanExporter) *sdktrace.TracerProvider {	return sdktrace.NewTracerProvider(		sdktrace.WithBatcher(exp),		// ParentBased: only a root flips the coin; downstream follows traceparent.		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.01))),	)…
```
23
p99 tripled at 14:05 with flat CPU and no deploy. How do you get from the dashboard to the slow code path in production, and what must already be in place for pprof?
A tripled p99 at flat CPU means waiting, not computing: narrow with the labels you already have, open one trace from the slow bucket, then profile block, mutex or goroutine, not CPU.
```
// Importing this package registers its handlers on http.DefaultServeMux from// its init(), which is exactly why the public server must never use that mux.import "net/http/pprof" func startDebug() *http.Server {	mux := http.NewServeMux() // our own mux, registered explicitly…
```
24
You get paged at 3am for CPU > 80% and p99 > 500ms alerts that clear before anyone looks, yet a real two-hour degradation paged nobody. How do you redesign alerting?
Page on a symptom the user feels, measured as a burn rate against an SLO, and never on a cause. An alert must answer whether the error budget is being spent faster than you can afford: a rate over a window, not a value at an instant.
```
groups:  - name: orders-slo    rules:      - record: orders:error_ratio:rate5m        expr: |          sum(rate(http_requests_total{job="orders",status="5xx"}[5m]))…
```

Observability & Production Ops

Logs, metrics and traces: what question does each one answer that the other two cannot?

What is structured logging, how does `log/slog` do it, and which log level do you use for what?

Find the bug: failed charges show up as `ERROR` lines, but a log search on the `err` field never finds them.

How does Prometheus get metrics out of a Go service, and which runtime metrics does `client_golang` expose without any code from you?

What is the difference between a counter, a gauge, a histogram and a summary, and which one do you use for request latency?

What are a trace, a span and a trace id, and how does the trace id travel from one service to the next?

What are SLI, SLO and SLA, and what is an error budget?

What do RED and USE stand for, and which of the two do you instrument first on an HTTP service compared with a queue worker?

Handlers repeat `"request_id", rid` in every `slog` call, and lines from deeper packages lack it. How do you fix that, and what is the trap with a logger in `Context`?

`slog.Info("auth ok", "user", u)` put a live bearer token into the log store, and that struct is logged from a dozen places. Where do you fix it once?

One failing DB call produces five `ERROR` lines and five error-counter increments, one per layer. What is the rule, and where does the single log line belong?

This handler serves the first request and fails every one after it. What happens, and where should the counter be created?

How do you choose histogram buckets for request latency, and why does the default top bucket distort p99?

Find the bug: the checkout latency histogram puts every request in the lowest bucket, including the ones clients time out on.

A request returned 500, yet its span in the trace is green. What did the code skip, and how is `span.RecordError` different from `span.SetStatus`?

In the trace, the repository's query span shows up as a sibling of `GetOrder` instead of its child, and failed lookups never appear at all. Why?

Both services run the OpenTelemetry SDK, yet the downstream one starts a brand-new trace for every request the gateway sends it. What is missing?

What do liveness, readiness and startup probes each decide, and what should each one check in a Go service?

Every rolling deploy drops a few requests although the service calls `srv.Shutdown` on `SIGTERM`. What is Kubernetes doing at that moment, and how do you wait it out?

A `user_id` label on `http_requests_total` left Prometheus with four million series and timing-out queries. Why does one label cost so much, and how do you bound it?

A 20-second database blip became a four-minute outage of a service whose readiness probe pings that database. Where did the other three and a half minutes come from?

You sample 1% of traces head-based, and a customer's 9-second checkout is missing from the backend. What can 1% answer, and what would tail sampling change?

p99 tripled at 14:05 with flat CPU and no deploy. How do you get from the dashboard to the slow code path in production, and what must already be in place for `pprof`?

You get paged at 3am for `CPU > 80%` and `p99 > 500ms` alerts that clear before anyone looks, yet a real two-hour degradation paged nobody. How do you redesign alerting?