Investigating Disk

Suppose an API that normally responds in 80 ms suddenly takes several seconds. CPU usage is low, memory looks healthy, and the process is still running, but the host’s load average keeps rising.

It is tempting to blame “slow disk I/O,” but that is only a symptom.

The affected data may live on a local SSD, logical volume, network filesystem, or container filesystem. The real problem could be a full filesystem, exhausted inodes, a growing device queue, writeback pressure, an I/O limit, or failing hardware.

A useful investigation should lead to a precise conclusion, such as:

Writes to the application’s data filesystem are queued on the underlying volume, while a backup process is generating most of the competing I/O.

This chapter develops a practical method for reaching that level of diagnosis using standard Linux tools.