When an order-search endpoint feels slow or a worker's memory keeps climbing, guessing at the cause wastes hours and usually picks the wrong target. Go ships a profiler called pprof in the standard library, and it answers the question "where is the time and memory actually going?" with sampled stack traces you can read like a flame graph. This lesson covers the profile types pprof collects, the two ways to enable it (HTTP for long-running services, direct calls for one-shot tools and benchmarks), and how to read the output without getting lost.

The instinct when something is slow is to read the code, find a loop that looks expensive, and rewrite it. That's almost always a waste. Most performance problems hide somewhere boring: an unexpected JSON marshal in a hot path, a regex compile inside a loop, a sync.Mutex held longer than it should be, a slice that grows from 16 to 8 million entries while everyone watches the wrong function.

Profiling cuts through the guesswork. The Go runtime samples your program's stack traces at a fixed rate while it runs, and pprof aggregates those samples into a report. If a function appears in 30% of the samples, it's responsible for roughly 30% of CPU time. There's no instrumentation overhead worth caring about (the default CPU sample rate is 100 Hz), and no source changes for the most common cases.

The same machinery covers six different profile types, each answering a different question.

The CPU and heap profiles handle 80% of real investigations. The rest are specialists, useful when you already suspect concurrency trouble.

Profiling sampling overhead is roughly 1% for the default CPU profiler. The block and mutex profilers do more work and need explicit opt-in via runtime.SetBlockProfileRate and runtime.SetMutexProfileFraction, because their cost would otherwise be visible in production.

Go gives you two collection paths, and they exist because long-running services and one-shot tools have different needs.

The HTTP path is for services. You import net/http/pprof for its side effects, the package registers handlers on http.DefaultServeMux under /debug/pprof/, and you pull profiles from a live process over the network. The direct path is for everything else: a CLI tool that runs to completion, a benchmark that needs to measure a specific function, a worker that should write a profile to disk on SIGUSR1. You call pprof.StartCPUProfile or pprof.WriteHeapProfile yourself.

Wiring the HTTP endpoint

A bare import is all it takes. The blank identifier means "import for side effects only", because you don't actually call anything in the package; you just want its init() to register routes.

With the server running, every profile is one HTTP request away.

The profile endpoint is special: it runs a CPU sampler for a configurable duration and returns the result. Everything else returns a snapshot of the current state.

Importing net/http/pprof registers routes on http.DefaultServeMux. If your service uses a different mux (chi, gorilla, gin), those routes won't show up automatically and you need to mount them by hand. Some teams prefer that, because it forces an explicit decision about which mux exposes profiling.

Never expose pprof publicly

The pprof endpoints leak detailed information about your binary, your data, and your concurrency state. A goroutine profile shows function arguments. A heap profile reveals object sizes that can identify customer data shape. The cmdline endpoint shows the exact flags the process was started with, including any tokens passed via the CLI.

Bind pprof to localhost or to a private admin port, and reach it through SSH tunnels, a VPN, or a sidecar with auth. The example above uses localhost:6060 for that reason. If you must expose it on a routable interface, wrap it in middleware that requires authentication. Treat pprof traffic the same way you'd treat database admin traffic, not the same way you'd treat your public API.

This pattern keeps the convenience of net/http/pprof without the exposure risk. The blank import would have leaked pprof onto the default mux, which is why the named import gives you explicit control.

Collecting from a one-shot tool

The other path uses runtime/pprof directly. You decide when to start sampling and when to write the result. This is what benchmarks and command-line tools do.

After the program exits, the file cpu.out holds the CPU profile, ready for go tool pprof. For a heap snapshot at the end of a run, swap pprof.StartCPUProfile/StopCPUProfile for pprof.WriteHeapProfile(f) after the work finishes.

Once you have a profile, the workflow is the same regardless of how you collected it. The go tool pprof command opens an interactive prompt or, with the -http flag, launches a local web UI with flame graphs.

The seconds=30 query parameter tells the endpoint how long to sample. Thirty seconds is a good default for a service under steady load. For a flakier signal (low traffic, sporadic spike), bump it to 60 or 120.

Inside the prompt, three commands cover most uses: top, list, and web.

Two columns matter most: flat is the time spent inside the function itself, and cum is the cumulative time including its callees. regexp.(*Regexp).doMatch is the single biggest direct consumer. matchQuery has a smaller flat but a larger cum, which says it spends most of its time calling other functions (like doMatch).

Once you've picked a suspect, list shows the function's source with per-line sample counts.

Line 17 is the bug. regexp.MustCompile runs inside the loop, compiling the same pattern for every order. The fix is to hoist it out of the loop and compile once, which usually drops the CPU profile for this function by a factor of 100 or more on a realistic input.

For visual exploration, web opens a graphviz call graph in a browser, and go tool pprof -http=:8080 cpu.out launches a full web UI with flame graphs, source views, and graph filters. The flame graph view is the most useful default, because it shows the call stack hierarchically with width proportional to time.

go tool pprof -http=:8080 needs Graphviz installed (brew install graphviz on macOS, apt-get install graphviz on Debian). Without it, the graph view fails with an error, though the flame graph still works.

The heap profile actually contains four different views of the same data. You pick which one with the -sample_index flag or the sample_index setting inside pprof.

The default for go tool pprof is inuse_space, and that's almost always what you want for "we're running out of memory". For "GC is eating 30% of our CPU", switch to alloc_space or alloc_objects to find the allocation hot paths.

The top entry is suspicious. addLineItem allocates more objects than JSON decoding. Looking at the source explains why.

fmt.Sprintf allocates a new string on every call, and append may reallocate the backing slice. The fix depends on context, but for a hot cart-builder path, pre-sizing the slice and using strings.Builder or strconv.AppendFloat keeps allocations bounded.

The heap profiler samples one allocation per 512 KB by default. For workloads with many small allocations, that means small objects are under-sampled. Set runtime.MemProfileRate = 1 to capture every allocation, at the cost of significant overhead. For a 10x sample rate (1 in 50 KB), set it to 50 * 1024.

A goroutine leak is a goroutine that should have exited but didn't. The classic case is a worker waiting on a channel that nobody will ever send to, or an HTTP handler that started a background goroutine and never gave it a way to stop.

The signal is simple: the goroutine count climbs but never comes back down. The goroutine profile shows you exactly where each one is parked.

debug=2 gives full stack traces with goroutine state. The [chan receive, 14 minutes] annotation means the goroutine has been parked on a channel receive for 14 minutes. If two goroutines are stuck on the same line of code for the same length of time, they're almost certainly leaks.

The fix in this case is usually a context.Context with cancellation:

When the parent context cancels, every worker exits cleanly. Without that, a worker waiting on w.queue will park forever when the producer stops sending.

A leaked goroutine costs roughly 8 KB of stack memory plus whatever it's holding in scope. A thousand of them is 8 MB plus retained heap, which can keep large objects alive past the point you'd expect them to be collected. Goroutine leaks often appear as "memory leaks" first.

CPU and heap profiles cover most investigations, but they're blind to a specific class of problem: a goroutine that's runnable but waiting. If your handler spends 200ms blocked on a mutex it's not the CPU that's slow, it's the contention. The block and mutex profilers handle those cases.

These two profilers cost more than CPU and heap, so they're off by default. You opt in with two runtime calls.

The block profile captures every place a goroutine waited on a channel, a mutex, or a sync primitive. The mutex profile is narrower: it records contention specifically on mutexes, with information about who held the lock when others were waiting.

The 8.1s blocked on sync.(*Mutex).Lock means goroutines spent a cumulative 8.1 seconds waiting to acquire mutexes. That's not 8.1 seconds of any single request; it's the sum across all goroutines during the profile window. The mutex profile narrows it down to which mutex.

SetBlockProfileRate(1) records every blocking event with nanosecond resolution. That's fine in development but adds measurable overhead under heavy contention. A common production setting is SetBlockProfileRate(10000), which samples events of 10us or longer. For mutexes, SetMutexProfileFraction(100) (1 in 100) is a typical compromise.

The go test command knows about pprof. Two flags, -cpuprofile and -memprofile, write profiles for the benchmark run.

The benchmark output already hints at trouble: 524288 B/op is half a megabyte allocated per call, which is large for a search over 10,000 in-memory orders. Open the memory profile and the culprit is usually a slice that grows without pre-sizing or a []byte to string conversion in a hot loop.

The -benchmem flag (go test -bench=. -benchmem) adds the B/op and allocs/op columns without writing a profile. It's the fast way to see whether allocations changed between two versions of a function. Combined with benchstat (a separate tool), you can compare two profile runs statistically and know whether your optimization actually helped.

Time for an end-to-end walkthrough. A service exposes /orders/search, and the team reports p99 latency has climbed from 80ms to 600ms after a recent release. We have the binary, access to the production-like staging environment, and pprof wired up on localhost:6060.

Step one is a CPU profile under realistic load.

json.Marshal and reflect.Value.Interface together are over 55% of CPU. That's a strong signal: JSON encoding is the bottleneck. The list command on renderResult shows why.

The handler is calling json.Marshal once per result, in a loop. That's allocating a fresh byte slice for every order and going through reflect every time. The fix is one of:

After switching to json.NewEncoder(w).Encode(results), a second profile shows the JSON cost dropping to about 8% and total CPU time roughly halving. The new top of the profile is the actual database query, which is the floor we can't push past without changing the data layer.

The same walkthrough applies to heap problems. If memory keeps climbing, pull ?debug=2 from the goroutine endpoint, look for stuck stacks, then pull inuse_space from the heap endpoint. Most production issues fit one of those two patterns.

A profile collected during low traffic is mostly noise. Profile during real load (peak hours, load test, replay) so the samples reflect what users actually hit. A profile from a quiet midnight window will tell you about background tasks, not the slow endpoint.

Pulling profiles by hand is fine for investigation, but it misses things. A spike that lasts 10 seconds at 3am won't be there when you arrive in the morning. Continuous profilers sample profiles on a schedule (every 10 or 60 seconds), store them in a time-series backend, and let you query by time window the same way you query logs or metrics.

The two main open-source options are Parca and Pyroscope. Both speak pprof natively: they scrape the same /debug/pprof/* endpoints, store the profiles, and provide a UI that can diff two time ranges to show what changed. Cloud vendors offer the same thing as a managed service (Google Cloud Profiler, Datadog Continuous Profiler, Grafana Cloud Profiles).

The cost is low. A pprof endpoint returning a 30-second CPU profile every minute is roughly 0.5% CPU overhead. Storage cost depends on retention and sample size, but a few gigabytes per service per month is typical. The value shows up the first time you can answer "what was running during the spike at 3:14am" without rerunning the workload.

Continuous profiling pays off most for services that are hard to reproduce locally (high concurrency, real user traffic, complex state). For a CLI tool or a batch job, on-demand profiles are usually enough.