Concurrent code fails in ways sequential code never does. A goroutine that should have exited is still hanging around, two goroutines mutate the same counter and produce the wrong total, a service that handled traffic fine yesterday is suddenly leaking memory because its goroutine count keeps climbing. This lesson covers the tools Go ships for finding those bugs: counting goroutines at runtime, dumping stacks on demand, profiling with pprof, catching data races with -race, and getting scheduler visibility through GODEBUG.

The simplest piece of observability is runtime.NumGoroutine(), which returns how many goroutines are alive right now, including the one calling it. It's cheap, requires no imports beyond the runtime package, and is the first check when you want to confirm "did my background workers actually exit?"

Consider an order-processing service that spawns a goroutine per order to do some background bookkeeping. After processing 100 orders, the count should be back to baseline. If it isn't, you have a leak.

The count went from 1 (just main) to 101 (main plus 100 workers) and back to 1 once all workers finished. The brief sleep after wg.Wait() gives the runtime a moment to mark the exited goroutines as dead; in tight benchmark loops you may see the count drop on the next tick.

runtime.NumGoroutine() is most useful in three places. Inside tests, you can assert the count is unchanged before and after a function runs to catch leaks introduced by a code change. Inside metrics endpoints, you can expose the count alongside CPU and memory so an SRE can graph it. Inside ad-hoc debugging, you can print it at suspect points to narrow down where the count starts climbing.

Counting tells you something is wrong. Stack dumps tell you what each goroutine is doing right now, which is what you need to figure out why nothing is making progress. Go offers three ways to get stack traces, each suited for a different situation.

debug.PrintStack() for the Current Goroutine

runtime/debug.PrintStack() writes a stack trace for the calling goroutine to standard error. It's the equivalent of "where am I right now?" and is useful for tracing how a particular code path got reached.

This is one goroutine's stack only. The function names, file paths, and line numbers show the call chain that led to the PrintStack call. Notice that the program continued after the dump; PrintStack is non-fatal, it just prints and returns.

runtime.Stack() for Programmatic Access

When you want the trace as a string instead of printed to stderr, runtime.Stack(buf, all) writes it into a byte slice you control. The second argument decides whether you get just the current goroutine (false) or every goroutine in the process (true).

Each goroutine block starts with its ID and current state in brackets (running, sleep, chan receive, IO wait, select, and so on). The state is the single most useful piece of information for diagnosis. A goroutine sitting in chan receive is waiting for a channel send that may never come. A goroutine in select is waiting for any of several cases, none of which have fired yet. A goroutine in sleep is in time.Sleep. Counting goroutines stuck in the same state is how you spot leaks at a glance.

SIGQUIT: The Built-in Panic Button

If a Go program hangs, sending it SIGQUIT (signal 3) makes the runtime dump every goroutine's stack to stderr and then exit. On a Unix terminal, this is Ctrl+\. From another shell, it's kill -QUIT <pid>. This is the single most useful trick when production is stuck and you have a few seconds to grab evidence before restarting.

Run it and hit Ctrl+\ after a couple of seconds. The runtime prints something like this.

Read the states. Goroutine 18 holds the mutex and is in sleep (the long time.Sleep we wrote). Goroutines 19 and 20 are in semacquire, waiting to acquire the same mutex. The bug is immediately visible: one goroutine is sleeping while holding a lock the others need.

SIGQUIT is brutal because it kills the process after dumping. In production, you usually want a less destructive option, which is what pprof gives you.

You can also opt into a non-fatal version using SIGUSR1 handlers you wire up yourself, or just expose the pprof endpoint described next. But for "I have an SSH session and 30 seconds before this thing gets restarted", SIGQUIT is the fastest.

Stack dumps are great for a single point in time. For ongoing investigation, especially "the goroutine count keeps climbing, where is the leak?", the net/http/pprof package fits well. Importing it for its side effects registers HTTP handlers under /debug/pprof/ on the default http.ServeMux. You add a tiny HTTP server, and now you can pull profiles on demand.

The blank import _ "net/http/pprof" looks unusual but is the standard way to attach the pprof handlers without explicitly calling any function. With the program running, visit:

The ?debug=2 query parameter asks for human-readable text. The response is a stack trace of every goroutine, grouped by what they're doing. Each entry shows how long the goroutine has been in its current state. Long-lived sleepers and blocked receivers are usually the leak suspects.

For aggregated analysis instead of raw text, drop the query parameter and use the go tool pprof interactive shell:

Inside the shell:

The top command groups goroutines by where they're stuck. Here, all 50 are in main.processOrder. The list command shows source lines with goroutine counts annotated, pointing directly at the offending line. For a real leak hunt, you'd take two profiles a minute apart and look for which function's goroutine count is growing.

Standalone pprof Without an HTTP Server

If you can't expose an HTTP endpoint, perhaps it's a CLI tool or a sensitive environment, write the profile to a file directly with runtime/pprof:

The file contains the same stack-trace format as /debug/pprof/goroutine?debug=2. Open it with go tool pprof goroutines.prof or just read it with any text editor.

The pprof flow is the standard approach for debugging real Go services. Here's how the pieces fit together:

The diagram shows the three branches off a single pprof endpoint. For a quick look, hit the URL in a browser. For aggregating thousands of similar stacks, use go tool pprof. For after-the-fact analysis or scripting, dump the profile to disk. All three reach the same underlying data.

Stack dumps and pprof tell you where goroutines are. The race detector tells you when two of them are touching the same memory without coordination. This is one of the harder classes of concurrency bug because it often doesn't crash, it just produces wrong answers.

Consider an order counter shared across goroutines. The intent is to count how many orders got processed.

Run it normally and you might see 1000, or 847, or 993. The output varies because counter++ is three operations internally (load, add, store), and two goroutines can interleave their loads, both add 1, and both write back the same value, losing one increment.

Now run it with the race detector:

The report names both goroutines, the memory address they touched, the file and line where each touch happened, and the stack that created each goroutine. The exit status of 66 means "we found a race and considered the run a failure". That's exactly the signal you want from CI.

The fix is to coordinate access. A sync.Mutex works:

The mutex makes counter++ atomic from the perspective of other goroutines. Run this version with -race and the warning disappears, the answer is consistently 1000.

For something this simple, sync/atomic is more lightweight than a mutex:

Both fixes pass the race detector. Pick atomic for small primitives (counters, flags, sequence numbers) and a mutex when the critical section touches multiple fields or runs more than a trivial amount of code.

What the Race Detector Will and Won't Catch

The race detector instruments memory access. It only flags races that actually happen during the run. If a race condition exists in your code but every goroutine happened to execute in a serial order during this particular run, the detector won't see it.

This is why race-detector runs need workload diversity. Run your tests with -race and -count=10. Run them under load. Run them on different machines with different scheduler timing. A clean run isn't proof of correctness, it's just proof that this particular run was race-free.

The cost is real. Programs compiled with -race use roughly 5x to 10x more memory and run 2x to 20x slower depending on workload, with allocation-heavy code at the worse end. That makes -race perfect for CI and unsuitable for production binaries.

When goroutines aren't progressing the way you expect, the GMP scheduler itself may be worth a look. Setting the GODEBUG environment variable enables runtime tracing without recompiling.

The 1000 is milliseconds between trace lines. Every second, the runtime prints one line summarizing scheduler state to stderr.

A typical line looks like this:

The fields, briefly:

A healthy program under heavy load shows nonzero numbers in the per-P queues and few idle processors. A program that should be busy but isn't, lots of idle processors and tiny run queues, may be bottlenecked on I/O or on synchronization (a lock that everyone is waiting for).

Setting GODEBUG=schedtrace=1000,scheddetail=1 produces more verbose output, including every goroutine's state. This is overkill for routine debugging but useful when you suspect a scheduling quirk and want every detail.

GODEBUG doesn't require code changes; you set it on the command line or in the service's process environment. That makes it easy to enable in a single deployment for diagnosis without rebuilding the binary.

When pprof's aggregated view isn't enough and you need to see exactly what every goroutine was doing at every microsecond, the execution tracer is the deepest tool Go ships. You generate a trace file with the runtime/trace package, then open it in a browser-based timeline viewer.

Then:

That command starts a local web server and opens a browser tab with views like "Goroutine analysis", "Network blocking profile", "Synchronization blocking profile", and a per-P timeline showing exactly when each goroutine was scheduled and on which OS thread.

The tracer is heavyweight. Trace files grow fast (megabytes per second of program runtime) and the runtime overhead is noticeable. Use it when you've narrowed a problem to a specific workload and need to see microsecond-scale scheduling behavior. For most "where are my goroutines stuck?" questions, pprof is enough.

For step-through debugging of a specific goroutine's state at a breakpoint, delve is the Go community's debugger of choice. It's a separate tool outside the scope of this chapter; the standard tooling above will resolve most production debugging without it.

Symptoms in concurrent code map to specific tools. The chart below shows the path from "what's wrong?" to "what do I run?"

The diagram pairs each common symptom with the tool that resolves it fastest. Hangs want stack dumps. Climbing counts want pprof diffs. Flaky correctness wants the race detector. Scheduling oddities want schedtrace or the execution tracer. When in doubt, start with pprof goroutine?debug=2; it answers more questions than any other single tool.

Three specific scenarios recur enough to be worth spelling out as recipes.

"My Service Hangs Under Load"

Symptoms: requests stop being served, no error logs, CPU is low. The most likely cause is goroutines stuck waiting on each other (a deadlock) or waiting on something external that's not responding (an unbounded wait).

Run kill -QUIT <pid> on the hung process and read the dump. Look for:

If killing the process isn't an option, hit /debug/pprof/goroutine?debug=2 instead. Same content, no process death.

"Goroutine Count Keeps Climbing"

Symptoms: memory usage grows over hours or days, runtime.NumGoroutine() reports more goroutines every time you check, no obvious request-rate increase to explain it.

Take two pprof goroutine profiles a minute or more apart:

Compare the counts grouped by stack. The function names that appear far more often in t2.txt are leaking goroutines. This profile-driven diff is how you find the leak source in code you didn't write or can't easily reason about.

"Tests Pass But Production Behaves Weirdly"

Symptoms: a feature works correctly in unit tests, occasionally returns wrong values in production, exact reproduction is hard. The classic signature of a data race.

Run the relevant tests with -race and -count=100 to expose timing-sensitive races:

If the race appears, the detector's output names the file and line. Fix with a mutex or atomic, rerun, confirm the warning is gone. If the race doesn't appear under -race, write a test that exercises the production code path with concurrent calls, then run that test with -race. The detector can only catch what runs.

A short end-to-end example pulls the most common tools into one program. It exposes pprof, prints the goroutine count periodically, and uses a mutex to keep its order counter correct.

The first print shows 500 worker goroutines plus the main goroutine, the pprof server goroutine, and the watchdog. The count drops as workers finish. The final count is 4 (main, pprof server, the pprof server's HTTP accept loop, and one runtime housekeeping goroutine), not 1, because the pprof server keeps its goroutines alive. The processed count is exactly 500, which the mutex guarantees. While this program is running, you can hit http://localhost:6060/debug/pprof/goroutine?debug=2 in a browser to see live stacks for every goroutine, including the watchdog parked in time.Sleep and the pprof server's accept loop.

That single program demonstrates: counting goroutines, exposing pprof, mutex-protected shared state, and a watchdog that gives you signal during the run. Most production Go services have something close to this skeleton.