An API that normally responds in 80 milliseconds suddenly takes several seconds. CPU usage is low, memory looks comfortable, and the process has not crashed. The host's load average, however, is climbing.

It is tempting to conclude that “the disk is slow.” That diagnosis is too vague to act on. The affected path may live on a local SSD, a logical volume, a network filesystem, or an ephemeral container layer. The problem might be a full filesystem, exhausted inodes, a device queue, writeback congestion, an I/O limit, or a failing device.

A useful disk investigation turns that vague symptom into a specific statement:

Writes to the application's data filesystem are waiting behind a sustained queue on the underlying volume, and a backup process is generating most of the competing I/O.

This chapter develops a method for reaching that level of precision with standard Linux tools.

“Disk problem” usually refers to one of three different incident classes.

Capacity failures prevent new data from being stored. Applications may report No space left on device, Disk quota exceeded, or failures while creating files. The exhausted resource may be data blocks, inodes, or an administrative quota.

Performance failures allow I/O to complete, but too slowly. Reads, writes, database commits, file uploads, or log operations may stall. The important evidence is latency, throughput, request rate, and the amount of work waiting.

Integrity or availability failures involve I/O errors, timeouts, device resets, or a filesystem becoming read-only. These require checking kernel messages and the storage environment, not merely looking at utilization.

Begin with the application's exact symptom. An ENOSPC error points toward capacity. A slow fsync points toward write durability latency. An EIO error or a read-only filesystem points toward a lower-level failure. Do not start by running every storage command you know; first decide which question you are trying to answer.

It is possible for more than one class to appear together. For example, a nearly full filesystem may become slow as an application repeatedly rotates or deletes files. Even then, keeping the questions separate makes the investigation clearer.

Before reading device metrics, determine where the affected data actually lives. Start with the application's path rather than guessing a device name:

Example output might look like this:

TARGET is the mount point containing the path. SOURCE identifies the mounted source, and FSTYPE tells you whether this is a local filesystem, a network filesystem, an overlay filesystem, or something else. The mount options also reveal important states such as ro, which means read-only.

Next, inspect how the source relates to the system's block devices:

A path may pass through several layers:

This mapping matters because the same I/O can appear at more than one layer. A request counted on /dev/dm-0 may also be counted on the NVMe device beneath it. Adding both values would double-count the work.

It also prevents a common category error. If the path is on NFS, local block-device statistics do not describe the complete request path. If it is on tmpfs, the limiting resource is memory rather than a physical disk. If it is inside an overlay filesystem, the relevant data may be in the overlay's backing directory rather than on the device that first seems obvious.

Containers add another wrinkle: a process may have a different mount namespace from the shell used for investigation. When host and container views disagree, inspect the mapping from the same namespace as the process, or use the container runtime's diagnostic facilities.

For a capacity symptom, check both byte space and inode space on the affected filesystem:

df -hT reports allocated and available filesystem space. Supplying the application path avoids searching a long mount list and reduces the chance of examining the wrong filesystem.

df -i reports inode usage. Most traditional Linux filesystems need an inode for every file and directory. A workload that creates millions of tiny files can exhaust inodes while plenty of byte capacity remains. In that case, creating a new empty file can fail even though df -h looks healthy.

If neither blocks nor inodes are exhausted, continue checking the error rather than assuming it was misleading. Possibilities include:

Use the OPTIONS column from findmnt to confirm the current mount mode. Quota commands and configuration differ by filesystem, so the application's reported identity and the filesystem type determine what to inspect next.

When df and du Disagree

df and du answer different questions.

df asks the filesystem how many blocks are allocated. du walks directory entries and totals the space reachable through those names. Because one examines filesystem allocation and the other examines the visible directory tree, their totals do not have to match.

To find large visible directories within one filesystem, use a bounded traversal such as:

The -x option prevents the traversal from crossing into other mounted filesystems. Be careful on a large production tree: du must visit files and can itself create metadata I/O.

A particularly important discrepancy occurs when df reports a full filesystem but du cannot find the space. On Linux, deleting a filename does not immediately release the file's blocks if a process still has the file open. The directory entry disappears, so du cannot reach it, but the filesystem keeps the data until the last file descriptor is closed.

Find open files with no remaining directory links using:

To narrow the search to a particular filesystem:

This commonly happens with logs. A log file is deleted or rotated, but the service continues writing through its old file descriptor. The durable fix is normally to make the service reopen its logs or to restart it in a controlled manner. Killing a process or truncating an arbitrary descriptor during an incident can cause data loss and should not be an improvised first response.

Other sources of df/du differences include filesystem metadata, reserved blocks, snapshots, copy-on-write behavior, sparse files, compression, and files visible only from another mount namespace. Treat the disagreement as evidence about accounting boundaries, not as proof that either tool is wrong.

Storage performance cannot be represented by a single percentage. Four measurements describe most block-I/O workloads:

Request size connects IOPS and throughput:

A workload performing 100,000 requests per second at 4 KiB per request transfers about 390 MiB/s. Another workload can reach roughly the same throughput with 1,600 requests per second at 256 KiB each. The byte rates are similar, but the pressure placed on the device is very different.

The workload shape also matters. Small random reads, large sequential writes, and synchronous metadata updates exercise storage differently. Read and write performance may be asymmetric, especially on virtual storage or when a device is doing internal maintenance.

Queue length connects concurrency and latency. Under reasonably steady conditions, Little's Law gives a useful consistency check:

For example, 2,000 operations per second with an average latency of 20 milliseconds implies roughly 40 outstanding operations:

This is not a replacement for measurement. It is a way to check whether the observed request rate, latency, and queue depth tell a coherent story.

vmstat provides a quick view of whether the system is spending time blocked around I/O:

Pay particular attention to:

A rising b column, high load average, and mostly idle CPUs are consistent with tasks waiting for I/O. They are not sufficient to identify the device or cause.

I/O wait is frequently misinterpreted. %iowait means CPUs were idle while the kernel had outstanding disk I/O. It is not the percentage of time that application processes waited for storage. A busy system can have painful storage latency and little I/O wait because other runnable work keeps the CPUs occupied. Conversely, one slow request on an otherwise idle machine can produce a high I/O-wait percentage.

Pressure Stall Information, when available, offers another system-wide view:

Example:

some indicates time when at least one non-idle task was stalled on I/O. full indicates time when all non-idle tasks were stalled simultaneously. The averages describe recent pressure windows; total is cumulative stall time. PSI helps establish that I/O is delaying useful work, but it still does not identify the responsible device or process.

For a live extended device view, use iostat from the sysstat package:

Here:

Interval reports are critical during an incident. A since-boot average can hide a sharp slowdown occurring now.

The exact output columns vary somewhat by sysstat version, but these are the central fields:

Older versions may show a combined await rather than separate read and write latency. Always read the column headings from the installed version rather than relying on a memorized layout.

Reading the Metrics Together

Suppose a device shows the following pattern over several intervals:

That combination is strong evidence that incoming read demand has exceeded the useful service rate of the storage path. More work is waiting, requests take longer, and completed throughput has reached a plateau.

No single latency value is universally “bad.” One millisecond might be excellent for a remote volume and unacceptable for an application designed around persistent-memory-like storage. Compare the current value with the device's normal baseline and with the application's latency budget.

%util also needs context. On a device that services one request at a time, a value near 100% is a useful saturation signal. Modern NVMe devices, RAID arrays, and virtual volumes can service many requests in parallel. Their %util can remain near 100% while still accepting more useful concurrency, or their performance can hit an external throughput or IOPS limit before the local metric tells the complete story. Use %util as supporting evidence, not as a universal saturation test.

Read and write averages can hide important variation. A small number of extremely slow requests may hurt an application's tail latency while await remains moderate. If averages do not explain the application symptom, preserve that mismatch as evidence; do not force a conclusion from iostat.

When logical and physical devices both appear, follow the device map created earlier. Metrics at the logical layer describe what applications submit, while lower layers describe how that work is dispatched. Do not sum the same operation across the stack.

Application file I/O does not always correspond immediately to block-device I/O.

A read can be satisfied from the page cache, in which case no device request occurs. A buffered write usually modifies pages in memory first. The kernel marks those pages dirty and writes them to storage later. As a result:

Inspect the current amount of dirty data and writeback in progress with:

One high sample is not enough to diagnose a problem. Look for dirty data accumulating while device throughput has plateaued, followed by writer stalls or a writeback burst. This pattern suggests that the kernel can no longer flush dirty pages as quickly as applications create them.

This timing difference is also why “the process with the largest write rate” is not always easy to determine from one snapshot. Logical bytes written by a process, bytes attributed during writeback, and physical bytes sent to the device describe related but different boundaries.

Once a device or filesystem looks suspicious, move from system-wide evidence to processes.

pidstat can report per-process disk activity over intervals:

Important fields include:

For a continuously updating view, iotop is another option:

This command reports only processes performing I/O, groups threads by process, runs in batch mode, and takes five one-second samples. Depending on the kernel and system configuration, complete delay accounting may require privileges and kernel task-delay accounting. Do not enable additional accounting on a busy production system without understanding its overhead and operational policy.

For one process, /proc exposes cumulative I/O counters:

Example fields include:

rchar and wchar count bytes passed through read-like and write-like system calls. They describe logical application I/O and may include data served from cache. read_bytes and write_bytes represent storage-layer activity attributed to the process. These values are cumulative, so take two samples separated by a known interval and compute the difference.

Permission rules may prevent reading another process's counters. Attribution is also imperfect for asynchronous buffered writeback and shared files. Use process counters to narrow the investigation, then correlate the timing with the device and application.

After identifying a process, connect it to actual files:

The block layer knows about requests to device sectors, not application filenames. These file-descriptor views help bridge that gap, although memory-mapped files, rapidly opened files, and already-closed files can make a single snapshot incomplete.

A workload can experience severe I/O delay even when the host device has spare capacity. In containerized environments, the relevant cgroup or one of its ancestors may impose a bandwidth or IOPS limit.

Find the process's cgroup:

On a cgroup v2 system, enter the corresponding directory under /sys/fs/cgroup and inspect:

io.stat contains cumulative per-device counters such as read bytes, write bytes, read operations, and write operations. Devices are identified by major and minor numbers, which can be mapped with:

Take deltas from io.stat; cumulative totals alone do not show the current rate.

io.max describes configured limits such as read bytes per second, write bytes per second, read IOPS, or write IOPS. A process repeatedly reaching such a limit can be throttled even though iostat shows headroom. Remember that a parent cgroup can constrain its descendants, so inspect the effective hierarchy rather than only the leaf directory.

io.pressure provides the same style of stall accounting as system-wide PSI, scoped to the cgroup. High cgroup I/O pressure combined with a configured limit and an underutilized host device is strong evidence of throttling.

Buffered-write attribution is not perfect in every filesystem and workload. When multiple cgroups dirty the same file, ownership can move, and some filesystems have limited writeback attribution support. Treat cgroup counters as one correlated source rather than unquestionable byte-for-byte billing.

Performance counters do not replace error inspection. A device that repeatedly times out and retries may look slow before it fails outright.

Search recent kernel messages:

On systems without a persistent journal, use:

Look for events involving I/O errors, command timeouts, device resets, filesystem errors, aborted requests, or a filesystem remounting read-only. Pay attention to timestamps and the complete sequence. A filesystem error may be a consequence of an earlier device problem rather than the initial cause.

Useful read-only device-health commands include:

Use the tool appropriate for the device, and follow the infrastructure team's access policy. In virtual machines and cloud environments, the guest may not receive meaningful physical-device health data. Provider volume metrics and platform events may be the relevant lower-level evidence.

If errors are present, preserve logs and escalate according to the system's recovery procedures. Repairing filesystems, replacing devices, detaching volumes, or forcing mounts are state-changing recovery actions, not exploratory diagnostics.

Live commands tell you what is happening now. They cannot reconstruct a five-minute storage stall that ended before you logged in.

If sysstat collection is enabled, historical sar data can show earlier device activity. Monitoring systems may retain device latency, throughput, queue depth, capacity, and application latency over much longer windows. Align these time series by timestamp.

The strongest incident analysis often comes from correlation:

Timing alone does not prove causality, but it provides a testable hypothesis. If pausing or rate-limiting the batch workload reduces the queue and restores application latency, the explanation becomes much stronger.

Consider a database whose query latency rises from 20 milliseconds to more than one second on one host. CPU utilization is only 25%, memory is not under pressure, and load average has increased.

First, identify the database path:

The result maps it to an XFS filesystem on /dev/mapper/data-db. lsblk shows that this logical device sits on a virtual NVMe volume.

Capacity checks are normal:

The filesystem is 62% full and inode usage is low. There are no quota or read-only errors, so this is not a capacity incident.

Next, sample the system:

Several processes appear in the b column while CPUs remain mostly idle. That is consistent with blocked work, but it does not yet identify storage as the cause.

Device sampling provides stronger evidence:

During the slowdown, read throughput on the NVMe volume plateaus around 430 MiB/s. Average read latency has risen from its usual 1.2 milliseconds to 28 milliseconds, and aqu-sz is near 55. %util is close to 100%, but the conclusion does not rest on that number alone. The rising queue, rising latency, and flat completed throughput together indicate that the storage path is saturated for this workload.

Now inspect process activity:

An export worker is reading roughly 400 MiB/s. It started at the same time as the database latency increase. The database itself performs a much smaller byte rate, but its latency-sensitive reads are waiting behind the export workload.

The operator pauses the export according to the service runbook. Within the next intervals:

The root cause is not simply “high disk usage.” A co-located export workload consumed the available read bandwidth and queueing capacity of the shared volume, delaying the database's smaller reads. Durable remedies might include rate-limiting the export, scheduling it outside peak hours, or isolating it onto different storage. Which remedy is appropriate depends on cost and reliability requirements, but the diagnostic evidence is now specific enough to guide that decision.

The following sequence works well when an application's filesystem operations are slow or failing:

Stop when the evidence is sufficient to state which resource is constrained, which workload is contributing, and why the application symptom follows from those measurements.

A reliable disk investigation begins by distinguishing capacity failures, performance degradation, and storage errors. Map the affected application path to its filesystem and complete device stack before interpreting metrics.

For capacity, check both blocks and inodes, understand why df and du can disagree, and look for quotas, read-only mounts, or deleted files that remain open. For performance, interpret latency, IOPS, throughput, request size, and queue depth together; neither I/O wait nor %util is sufficient alone.

Finally, connect the device evidence to the workload using per-process and cgroup measurements, account for page-cache writeback, and inspect kernel error events. The goal is a causal statement that identifies the constrained storage boundary, the work creating pressure, and the application operation being delayed.