A backend service normally answers requests in 40 ms. This morning, its p99 latency jumped to 900 ms. CPU usage is low, and a dashboard shows high “I/O wait,” so the storage device is blamed.

That conclusion might be correct, but the evidence is incomplete. The service could be waiting for synchronous file flushes, a network filesystem, memory reclaim, a degraded RAID array, or another process's write-back. It might not be waiting for storage at all.

“Slow I/O” is a symptom, not a diagnosis.

Effective diagnosis connects application symptoms to a specific workload, filesystem, block-device path, and time interval. It uses several measurements together rather than treating one percentage as proof.

An I/O request crosses several layers, and each layer exposes a different view:

No single tool observes the complete path.

Application timing includes everything below the application, plus locks, scheduling, runtime pauses, and application logic. Block-device statistics observe requests that actually reach a block device, but cached reads do not appear there. Per-process counters attribute some I/O to a task, but buffered write-back can happen later in a kernel worker.

A disciplined investigation moves through five questions:

The goal is not to collect every available metric. It is to form a hypothesis and gather the smallest set of evidence that can confirm or reject it.

“The database is slow” is too broad to investigate efficiently. A useful symptom statement includes:

For example:

This points toward a write-path or synchronization problem and preserves the time interval needed to correlate system metrics.

Measure rates over an interval

Many Linux counters are cumulative since boot. One reading does not show a rate:

Two readings with timestamps can:

Tools such as iostat, vmstat, and pidstat calculate these differences for convenient sampling.

Their first report can contain averages since system startup. For a current incident, use an interval and either ignore the first report or request that it be omitted. Also preserve the application and system timestamps so the samples can be aligned.

Compare with a healthy baseline

A latency value has little meaning without workload and context. An 8 ms read may be normal for a random HDD request and alarming for a lightly loaded local NVMe device.

Compare the incident with a known healthy period under a similar request rate. The difference often matters more than a generic threshold:

The workload rate stayed constant while latency and backlog rose. That is stronger evidence than any of the values alone.

Before reading device statistics, determine which device actually stores the affected data.

For a path such as /var/lib/orders/data.db:

An illustrative result is:

This identifies the mounted filesystem and its source. FSTYPE is important: a local block-backed filesystem, NFS mount, and memory-backed filesystem require different next steps.

Then inspect the block-device topology:

A path can be layered:

Statistics reported at one level do not translate directly to the level below. One filesystem write can appear as several drive operations by the time it reaches the bottom.

Statistics at each level answer a different question. High latency on a logical RAID device should be correlated with its members. One slow member can delay the logical request even when the other members look healthy.

Device names are also easy to confuse. /dev/nvme0n1p2 is a partition of /dev/nvme0n1; /dev/dm-0 may be the kernel name behind a friendlier /dev/mapper/... path. Record the mapping before interpreting a row from iostat.

If findmnt shows a network filesystem, quiet local disks do not disprove an I/O problem. The relevant wait can be in the client network path, remote server, or remote storage. The local block-device layer does not see the remote device's service time.

On Linux systems with the sysstat tools installed, a useful starting command is:

The options request extended device statistics, omit devices with no activity, omit the since-boot first report, and sample every second.

Field names vary slightly across sysstat versions, but these are central:

Read and write fields should be examined separately. A device can serve reads quickly while writes stall behind flushes or internal garbage collection.

Interpret workload before limits

Consider this shortened sample:

The workload is dominated by writes. Its average write request is roughly:

Write latency is much higher than read latency, and the average queue contains about 41 requests. This supports the hypothesis that the write path is backlogged.

The throughput of 35,200 KiB/s is not high by sequential NVMe standards, but this is not a large sequential workload. It is thousands of small operations, possibly including persistence constraints. Comparing only the byte rate with a manufacturer's maximum bandwidth would be misleading.

Relate latency, rate, and queue depth

For a stable system, Little's Law provides a useful consistency check:

Using the write side of the sample:

That is close to the reported total average queue depth of 41.2 after allowing for reads and measurement differences.

The relationship does not identify the root cause. It helps verify that the measurements describe the same workload and shows how modest latency becomes a large queue at high request rates.

%util is not a universal saturation percentage

For a device that services one request at a time, %util near 100% often means it had no idle time and may be saturated.

Modern SSDs, NVMe devices, and RAID arrays can process many requests concurrently. They can report activity during nearly the entire interval while still having capacity for more work. Conversely, a latency-sensitive application can be limited by one synchronous request at a time while %util remains low.

Treat %util as “the device had at least one request active,” not as “this device used exactly this percentage of its maximum performance.” Confirm saturation using latency, queue growth, throughput behavior, and the known workload.

Average latency can hide stalls

await is an average over the sample. Suppose 999 operations finish in 1 ms and one takes 1 second:

An average near 2 ms hides the one-second outlier that may determine the application's p99 or maximum latency.

When the symptom is tail latency, obtain a latency distribution from application instrumentation or a tracing tool that can build block-I/O histograms. Do not expect one-second averages to explain rare stalls.

vmstat provides a compact view of runnable tasks, blocked tasks, memory, swapping, block I/O, and CPU states:

Useful fields include:

A rising b count at the same time as high device latency supports the idea that tasks are blocked behind I/O. Nonzero si or so can show that memory pressure is adding swap traffic to the storage workload.

What %iowait does and does not mean

CPU I/O-wait time is frequently misread. It is broadly the time a CPU was idle while the system had outstanding I/O.

It is not:

A CPU can run another task while one task waits for I/O. On a multicore system, the waiting task is not inherently associated with one CPU, making per-CPU attribution imperfect. Low %iowait can also coexist with a serious storage problem if CPUs remain busy doing other work.

Use %iowait as a clue that must be correlated with device latency, blocked tasks, and application timing.

Once a device is known to be active, identify who is generating the requests.

pidstat can report per-task I/O activity:

Important fields include:

An illustrative result might reveal:

The backup process generates most of the bandwidth, while the service accumulates more synchronous I/O delay. This suggests interference: a bandwidth-heavy background workload is increasing latency for a smaller, synchronous workload.

Inspect one process directly

Linux exposes cumulative counters in /proc/<pid>/io:

The output separates syscall-level byte counts from storage-level byte counts:

rchar counts bytes returned by file-reading system calls, which can include data served from cache. read_bytes counts bytes actually fetched from the storage layer for block-backed filesystems.

A large rchar with a small read_bytes value indicates that many reads did not require physical storage I/O during the observed lifetime. Similarly, buffered writes complicate the timing relationship between wchar, write_bytes, and device activity.

Access to another process's counters can be restricted. The counters are also cumulative, so take timestamped differences when investigating a specific interval.

Blocked process state

The process state D means uninterruptible sleep. It often appears while a task waits in a kernel I/O path.

wchan shows the kernel wait location when available. Repeatedly seeing the affected process in state D can support a blocking-I/O hypothesis.

State D does not uniquely mean “physical disk.” Network filesystems, device faults, and other kernel waits can produce the same state. A single instantaneous observation can also catch a harmless short wait, so sample during the incident and correlate it with other evidence.

Block-device activity is not synchronized one-to-one with application system calls.

A cached read can complete without issuing a block request. A buffered write can return after dirtying memory, while the actual device write happens later. This creates two common diagnostic surprises.

The application is busy but the device is quiet

If a read-heavy service is fast and the device shows little activity, its working set may be served from the page cache. The device is not failing to report the reads; the reads never reached it.

If a slow service shows no corresponding device activity, investigate where the operation waits before assuming the storage hardware is responsible. It may be blocked on an application lock, a remote filesystem, or a dependency that is not a local block device.

The device is busy after the writer becomes quiet

A process can rapidly dirty cached file pages and then stop writing. Kernel write-back can continue sending those pages to storage afterward, possibly under worker-thread context.

Relevant memory counters are:

Dirty is memory waiting to be written. Writeback is memory currently being written.

Observe their direction over time:

These are hypotheses, not automatic conclusions. Correlate the counters with iostat, process rates, and the application's write timing.

Avoid clearing caches merely to “see what happens” on a production system. That changes the workload, can cause a large read storm, and destroys the evidence you are trying to observe.

A device can have spare aggregate throughput while an application is still storage-limited.

Consider a transaction loop:

Only one transaction is outstanding at a time. If each synchronization takes 8 ms, the loop can complete at most about:

The device may report low bandwidth and a shallow queue. The bottleneck is the application's serial durability dependency, not necessarily the device's maximum streaming bandwidth.

When attaching a tracer is safe and permitted, strace can show relevant calls and their elapsed time:

An abbreviated trace might show:

The writes are accepted quickly, while each fsync() waits around 20 ms. The trace localizes the application-visible delay to the durability boundary.

Tracing adds overhead and can expose sensitive arguments or data. Restrict it to the required calls, use a short capture interval, and follow production access controls.

System-call latency is still broader than physical-device service time. A call can wait on filesystem locks, write-back, or several lower-level operations. Correlate it with block-device metrics before naming the hardware as the cause.

Capacity pressure can produce failures, long allocation paths, or unexpected behavior that looks like a general I/O problem.

Check block space for the affected path:

Check inode availability:

A filesystem can have free bytes but no free inodes, preventing new files from being created.

If du reports less usage than df, a process may still hold a deleted file open. Its directory entry is gone, so pathname-based traversal does not count it, but its allocated blocks remain until the last descriptor closes.

On systems with lsof, unlinked open files can be listed with:

Do not truncate or close a process's file descriptors merely because a large deleted file appears. First identify the owning service and understand how it handles logs and descriptors.

Space checks are fast and useful, but a nearly full percentage alone does not prove the source of latency. Filesystem behavior depends on allocation patterns, reserved space, fragmentation, and implementation.

High latency can come from retries, timeouts, controller resets, media errors, or redundancy recovery. Average performance counters show the delay but may not explain it.

Inspect recent kernel messages:

Relevant messages can mention:

The exact wording depends on the device and driver. Correlate message timestamps with the application incident.

For software RAID, these read-only checks show array state and rebuild progress:

A rebuild or consistency check can consume substantial member bandwidth and raise application latency even though the logical array remains available.

Device-specific health tools can add evidence:

These commands may require elevated privileges and the relevant packages. Interpret error counters, temperature, spare capacity, and health fields using the device's documentation. A single “healthy” summary does not disprove intermittent timeouts or performance problems visible in kernel logs.

Never begin a destructive repair or remove a member based only on a performance symptom. Preserve evidence, verify the exact target device, and follow a recovery procedure appropriate to the storage system.

One-second averages can miss a 100 ms burst that severely affects request latency. When ordinary metrics identify the device but not the cause, block-I/O tracing can show individual operations or latency distributions.

Linux tracing tools built with eBPF can answer questions such as:

Tool collections often provide commands with names such as biolatency, biosnoop, or biotop, but availability and arguments differ by distribution. Verify the installed tool's help and kernel support before use.

Task context is not always the original application. Buffered write-back can be submitted by a kernel worker after the application dirtied the data, so correlate block traces with per-process counters and dirty-memory behavior.

Deeper tracing should follow, not replace, basic localization. Capturing every block event on a busy server can generate substantial output and overhead. Filter by device, operation, process, and time window whenever possible.

Most importantly, keep the measurement boundary clear:

Different boundaries can legitimately report different latency for the same logical request.

Diagnosis comes from combinations of signals.

High latency, growing queue, flat throughput

If request rate rises, throughput stops increasing, await rises, and aqu-sz grows, the device or lower storage path is likely at its effective capacity for that workload.

Confirm that the request size and read/write mix are stable. Then inspect whether all physical members share the load or one member is slower.

Low bandwidth, low queue, slow application

Low throughput does not mean storage is irrelevant. The application may issue one dependent request or fsync() at a time. Each operation can be slow while the device is idle between requests.

Trace the slow operation and compare its call latency with device latency. A network filesystem or application lock can produce the same high-level symptom with quiet local devices.

Device writes with no obvious writing process

If wkB/s is high while current process writers are quiet, check whether Dirty is falling. Kernel write-back may be draining data generated earlier.

Also check RAID recovery, filesystem maintenance, swapping, and virtual-machine or container hosts where attribution may occur at another layer.

High throughput with low latency

A device moving many bytes is not necessarily overloaded. Large sequential requests can produce high bandwidth, stable low latency, and a manageable queue.

Do not optimize a healthy busy device merely because %util is high.

Intermittent latency spikes with errors

If kernel messages show timeouts or resets at the same moments as latency spikes, suspect the device, connection, controller, or lower storage system rather than normal queueing alone.

Capture exact timestamps and device identifiers. Error recovery can stall many requests behind one problematic command.

Reads are slow only after memory pressure

If application read latency rises while major faults, swap activity, and device reads increase, the system may have lost useful cached data or be paging.

The storage device is doing more work, but the initiating condition is memory pressure. Treating this only as a disk-throughput problem misses the cause.

The overall process can be summarized as a decision flow:

Write down the evidence as a short causal statement:

This is testable. It is stronger than “I/O wait was high.”

After a configuration or workload change, repeat the same measurements under comparable load. A convincing result improves the application symptom and the system signal predicted by the hypothesis.

Diagnosing I/O problems begins with a precise application symptom and time window. Map the affected path to its filesystem, logical devices, and physical members before interpreting device statistics.

Use iostat to relate IOPS, request size, throughput, latency, queue depth, and activity. Add vmstat, per-process counters, page-cache state, filesystem capacity, kernel errors, and RAID status to explain the surrounding system. Neither %util nor %iowait proves storage saturation by itself.

The strongest diagnosis connects observations across layers into a testable cause: who generated the work, where it waited, and why that wait affected the application. Use targeted tracing only when ordinary interval metrics cannot resolve the remaining question.