Diagnosing I/O Problems

Suppose a backend service normally responds in 40 ms, but its p99 latency suddenly jumps to 900 ms. CPU usage is low, and the dashboard shows high I/O wait, so storage seems like the obvious culprit.

But that signal alone is not enough.

The service might be waiting on file flushes, a network filesystem, memory reclaim, a degraded storage device, or write-back triggered by another process. The bottleneck may not even be storage.

“Slow I/O” is a symptom, not a diagnosis.

Good I/O debugging connects the application slowdown to a specific workload, filesystem, device path, and time window. No single metric tells the whole story, so the goal is to combine several signals and follow the request through the I/O stack.

This chapter develops an application-to-device investigation workflow using storage-path mapping, device and process metrics, cache and write-back evidence, flush waits, capacity checks, error logs, and deeper tracing.