Imagine a backend service that normally responds in 80 milliseconds suddenly takes two seconds. The process is still running, no obvious error appears in its logs, and restarting it makes the problem disappear.

It is tempting to open top, notice one unusual number, and declare a cause. Perhaps CPU usage looks high, free memory looks low, or the load average is larger than the number of CPUs. Each observation may be relevant, but none is a diagnosis by itself.

Reliable debugging requires a disciplined way to move from an externally visible symptom to the mechanism that produced it:

The direction matters. Starting with a favorite tool or a familiar failure mode encourages confirmation bias. Starting with the symptom keeps the investigation tied to what users and the system are actually experiencing.

Systematic debugging is the process of reducing uncertainty with measurements until one causal explanation fits the evidence better than its alternatives.

This chapter develops a reusable method for investigating Linux systems. It applies whether the complaint is high latency, low throughput, errors, a stalled process, or an overloaded host.

A system is observable when its externally available information is sufficient to reason about what is happening inside it.

On Linux, useful evidence comes from several forms:

These forms answer different questions. A metric can show that p99 latency rose at 14:03, but it may not identify the affected requests. A log may explain one failure, but it cannot prove that the same event caused a host-wide slowdown. A current-state snapshot can show what exists now, but not necessarily what happened five minutes ago.

Good investigations correlate independent evidence. If service latency, a kernel queue, and one process's activity all change at the same time, the case is stronger than if only one graph looks unusual.

Observability is not the same as monitoring. Monitoring checks selected conditions and alerts when they cross defined boundaries. Observability provides the evidence needed to investigate conditions that may not have been predicted in advance. An alert begins an investigation; it does not complete one.

“The server is slow” is too vague to debug. Before examining the machine, turn the complaint into a statement that can be measured.

A useful symptom statement answers:

For example:

This statement is far more useful than “checkout is slow.” It defines the boundary of the problem and immediately suggests comparisons: affected versus unaffected zones, before versus after 14:03, successful versus timed-out requests, and p50 versus p99 latency.

The distinction between latency, throughput, and errors is especially important:

These can move independently. A service may preserve throughput while latency rises because more work waits in a queue. It may reject work quickly, producing low latency but a high error rate. It may also show acceptable average latency while a small but important group of requests becomes extremely slow.

Do not replace the reported symptom with a convenient proxy. High CPU usage is not the symptom if users reported high latency. CPU usage becomes evidence only after its relationship to latency has been established.

Many investigations become much easier once the scope is known.

Ask whether the issue affects one request, one process, one host, a group of hosts, or every instance of the service. Compare affected and unaffected parts of the system while the incident is active.

Suppose every process on one host is slow, while identical hosts are healthy. A host-level resource or configuration becomes plausible. If only one process on that host is affected, a system-wide shortage is less likely. If all instances became slow immediately after the same deployment, shared application behavior or configuration deserves attention.

Useful dimensions for comparison include:

Comparison is powerful because the healthy case acts as a control. Two otherwise similar hosts with different behavior often reveal more than an isolated snapshot of the unhealthy host.

Scope also prevents category errors in virtualized and containerized environments. A process can be limited by its container even when the host has idle CPU and free memory. Conversely, a healthy-looking container can suffer because the shared host or storage service is overloaded. Every number must be interpreted relative to the boundary at which it was measured.

Time is the primary key of an incident. Without aligned timestamps, related events cannot be correlated reliably.

Record the current time and time zone at the start of an investigation:

The first command anchors later observations to wall-clock time. uptime reports how long the machine has been running, the number of logged-in users, and recent load averages. It can also reveal that a supposedly long-running host rebooted recently.

Next, identify changes near the beginning of the symptom:

A recent change is a strong lead, not automatic proof. A deployment at 14:00 and an alert at 14:03 are correlated in time. The deployment becomes a likely cause only when a plausible mechanism and supporting measurements connect it to the symptom.

Preserve transient evidence before restarting or killing a process. A restart may be the correct short-term mitigation, but it destroys queue state, process state, open-resource information, and other clues. When impact permits, capture a small system snapshot first:

These commands provide only an initial survey. Some may be unavailable in a minimal image, and process or kernel details may require additional permission. Save command output with its timestamp and scope rather than copying isolated numbers into notes.

Be aware that the first line of vmstat usually contains averages since boot; the later lines describe the requested sampling intervals. Mixing those meanings can create a false timeline.

A measurement is meaningful only in context.

Four gigabytes of available memory may be normal on an 8 GiB host and alarming on a 1 TiB host. A load average of 20 may indicate heavy contention on a four-CPU machine but ordinary activity on a much larger one. A 10-millisecond storage latency may be excellent for one device and a serious regression for another workload.

A baseline describes the system's expected behavior under a comparable workload. Useful baselines include:

Compare rates with rates and distributions with distributions. A one-minute average should not be compared casually with a one-second peak. A daily request count says little about a ten-second burst. The baseline must use a time window and workload that make the comparison fair.

Do not assume that “historical” means “healthy.” A service can run in a degraded state for weeks. A good baseline combines history with an explicit understanding of acceptable behavior.

Linux exposes many correlated values. Debugging requires deciding where each value belongs in a causal chain.

Consider this sequence:

Only the first step is a change in the outside world. Everything after it is a consequence, which is why fixing the last step alone does not hold.

Request latency is the external symptom. A growing application queue is an intermediate effect. Slow storage service is a resource bottleneck. Larger requests are the workload change that triggered the bottleneck.

Calling the queue the “root cause” would be incomplete: the queue is where waiting became visible, not why the system lost capacity. Calling the large requests the only cause may also be incomplete if the system was expected to handle them but a storage regression reduced its capacity.

A useful root-cause statement includes the mechanism:

This statement is testable. It predicts that the affected requests perform more reads, storage waiting rises with latency, workers spend more time waiting, and reducing that work or adding sufficient capacity improves the symptom.

When the constrained resource is not yet known, use a broad and shallow survey. The goal is not to explain every counter. It is to rule resource classes in or out.

A practical framework is USE:

Apply these questions to every relevant resource: CPUs, memory capacity, storage devices, network interfaces, and constrained pools inside the application or container.

Utilization alone is not enough. A CPU can be fully utilized without harmful waiting if all required work still completes on time. A storage device can show moderate average utilization while short bursts produce large queues and tail latency. A thread pool can be saturated even though the host's CPUs are mostly idle.

Saturation often explains latency more directly. Once arrivals exceed a resource's ability to complete work, unfinished operations accumulate. New work waits behind old work, so response time can rise sharply even when throughput changes little.

Errors include more than application error messages. Retransmissions, allocation failures, throttling events, device retries, dropped packets, and exhausted limits may all be evidence. Some failures are handled internally and appear only in kernel counters or system logs.

A high-level survey can follow this path:

Do not follow every branch at full depth. Use the first pass to find the branch with evidence that changes alongside the symptom.

Load average is a common source of premature conclusions.

On Linux, load average reflects tasks that are runnable and tasks in uninterruptible sleep, not a percentage of CPU usage. Runnable tasks either execute on a CPU or wait to execute. Tasks in uninterruptible sleep commonly wait for a kernel operation such as I/O to complete.

This explains an apparently contradictory observation:

The machine may have many tasks waiting in uninterruptible sleep rather than competing for CPU time. It may also have experienced a burst that affects the load average after instantaneous CPU activity has fallen. Load average tells you that work is runnable or stuck in a counted wait state; it does not identify the responsible resource.

The three values printed by uptime are exponentially smoothed load averages representing roughly 1, 5, and 15 minutes. They are not exact averages over three independent fixed windows. Use them to recognize a trend, then inspect current task states and resource-specific evidence.

Similarly, CPU utilization near 100% does not prove that CPU capacity is causing the reported problem. Ask whether runnable work is waiting, whether the affected operation needs that CPU, and whether its latency changes with the CPU pressure.

After identifying a suspicious resource class, determine who is creating the demand or experiencing the wait.

The investigation normally moves through levels:

Each step narrows the search rather than guessing at the end. Starting from a suspected code path instead is what turns an investigation into a series of unrelated hunches.

Skipping levels leads to weak conclusions. A busy device does not prove that the target service is responsible for its traffic. A process with high resident memory does not prove that it caused memory pressure. A thread observed in a system call once does not prove that the call dominates its time.

At each level, connect the suspected owner to the evidence:

Linux process IDs provide a useful bridge between system-wide and process-specific views. Once a process is identified, record its command, parent, start time, execution state, and container or service identity. This prevents confusion after a restart reuses a PID or when several processes have similar names.

On a multithreaded server, the process may be too coarse a unit. One thread can consume a CPU, hold up a queue, or wait on an operation while the process-wide average hides it. Narrow to threads only after process-level evidence points there.

A hypothesis should predict evidence that has not merely been restated from the symptom.

Weak hypothesis:

Strong hypothesis:

The stronger hypothesis makes several predictions:

Try to disprove the hypothesis. Look for evidence that would be inconsistent with it: the image worker uses a different device, the API makes no storage requests during the affected operation, or latency remains high when storage waiting disappears.

Write down plausible alternatives before diving deeply:

Then run the cheapest, safest observation that distinguishes among them. If CPUs are mostly idle and request threads are sleeping on network operations, H1 loses support. If only the affected container reports throttling, H4 gains support.

This approach avoids collecting large amounts of interesting but irrelevant data.

Performance problems are changes over time. A single snapshot can miss a short burst or capture an unrepresentative instant.

Commands that accept an interval can reveal direction and correlation:

iostat and pidstat are commonly provided by the sysstat package and may not be installed by default.

Sample long enough to include the symptom, but use a resolution fine enough to expose meaningful bursts. If latency spikes for two seconds every minute, a five-minute average can hide the pattern completely.

Also preserve distributions when tail behavior matters. An average response time of 100 milliseconds can describe both of these populations:

The operational impact is very different. Compare percentiles, counts, and maxima alongside averages, and make sure each statistic covers the same interval.

Rates require two observations. A cumulative counter saying that a machine has performed one million context switches since boot is not evidence of a current problem. The change in that counter over a known interval provides the rate that can be compared with the symptom.

Start with tools that are broad, inexpensive, and unlikely to disturb the workload. Increase detail only when the current evidence justifies it.

A sensible progression is:

High-resolution tracing can produce large volumes of data and impose measurable overhead. Attaching a debugger can pause a process. Even repeatedly reading extensive diagnostic state consumes CPU and I/O. The act of observing a system can change its behavior; this is the observer effect.

Match the tool to the question. Do not collect every available event when a counter can answer the question safely. Conversely, do not keep staring at aggregate CPU usage when the remaining question is which code path consumes the CPU.

Production constraints matter. Prefer read-only inspection, bound the duration and output of expensive commands, avoid recording sensitive request data, and know whether a tool requires elevated privileges. If the risk of measurement is larger than the current impact, reproduce the problem in a controlled environment or use existing telemetry.

Consider an API running on an eight-CPU Linux host. Its request rate remains steady, but p99 latency rises from 180 milliseconds to 2 seconds. The alert begins at 14:03.

The investigation starts with the symptom rather than the host:

Deployment records show no code change, but a data-compaction job started on the affected host at 14:00. This is a candidate cause, not yet a conclusion.

A broad resource survey finds:

The evidence argues against CPU and memory as the immediate constraints. It supports the hypothesis that compaction competes with the API for storage service.

The process-level view shows API workers waiting during the same intervals in which storage operations slow. Temporarily rate-limiting the compaction job causes pending storage work and API p99 latency to fall together. Small requests improve less because they were never strongly affected.

A precise finding is now possible:

The immediate mitigation is to limit or pause compaction. A long-term fix might schedule it differently, control its I/O rate, isolate workloads, or reduce the API's storage demand. Selecting among those fixes requires workload and product context, but the diagnosis identifies the mechanism each fix must address.

Finally, verify the user-visible symptom. It is not enough that one storage graph improved. Request p99 must return to its normal range without unacceptable errors or lost throughput.

Incident response has two related goals: restore acceptable service and understand the failure well enough to prevent recurrence.

When impact is severe, mitigation may take priority. Restarting a process, removing traffic, disabling a feature, or limiting a batch job can be appropriate. Record what changed and when, because the response to that change is itself evidence.

Prefer one controlled change at a time when circumstances allow. If a team restarts the service, doubles its CPU limit, clears a cache, and shifts traffic simultaneously, recovery does not reveal which action mattered.

Treat mitigation as an experiment:

Improvement after an action supports a hypothesis but does not always prove it. A restart resets many kinds of state at once. It may clear a leaking resource, remove queued work, reconnect to a dependency, or simply move the workload to a healthy host. “A restart fixed it” describes a recovery action, not a root cause.

An investigation is complete only when the original symptom has been checked again.

After mitigation or repair:

Document the result as a causal narrative:

Include observations that ruled out attractive alternatives. Negative evidence saves future investigators from repeating dead ends.

A good incident record distinguishes facts, interpretations, and unknowns. “Storage latency reached 80 ms” is an observation. “Compaction caused the API slowdown” is an interpretation supported by several observations. “Why compaction ran outside its normal window” may remain an open question.

The complete method can be condensed into eight steps:

This runbook is intentionally independent of any one Linux command. Tools change, permissions differ, and production environments expose different telemetry. The reasoning remains the same.

Systematic debugging begins with a precise, user-visible symptom rather than a tool or an unusual metric. Establish its scope and timeline, preserve transient evidence, and compare the affected system with a meaningful baseline.

Survey each relevant resource for utilization, saturation, and errors. Then narrow from the system to the resource, process, thread, operation, and responsible code or dependency.

Treat hypotheses as predictions that evidence can disprove. Prefer interval measurements over isolated snapshots, increase diagnostic resolution gradually, and account for measurement overhead.

A mitigation is not automatically a diagnosis. Verify recovery at the original service boundary and document the mechanism connecting the trigger, constrained resource, waiting work, and observed impact.