Investigating System Calls

A service can consume little CPU, perform almost no disk I/O, and still take seconds to answer a request. Its logs may say only “operation timed out,” leaving an important question unanswered:

What was the process asking the kernel to do while the request was slow?

System-call tracing observes the boundary between a program and the Linux kernel. It can reveal a failed file lookup, a thread waiting on a synchronization primitive, a retry loop issuing thousands of tiny operations, or a deliberate sleep between attempts. This evidence is especially valuable when source code is unavailable or application logs omit the low-level operation.

System-call tracing is also intrusive. A tracer can slow the target, change timing, and expose sensitive arguments. The goal is therefore not to collect everything. It is to capture the smallest trace that can confirm or reject a specific hypothesis.

This chapter explains when and how to trace new and running processes safely, how to filter and time bounded captures, and how to interpret descriptors, errors, interrupted and concurrent calls, summaries, diagnostic patterns, and tracing blind spots.