A service can consume little CPU, perform almost no disk I/O, and still take seconds to answer a request. Its logs may say only “operation timed out,” leaving an important question unanswered:

What was the process asking the kernel to do while the request was slow?

System-call tracing observes the boundary between a program and the Linux kernel. It can reveal a failed file lookup, a thread waiting on a synchronization primitive, a retry loop issuing thousands of tiny operations, or a deliberate sleep between attempts. This evidence is especially valuable when source code is unavailable or application logs omit the low-level operation.

System-call tracing is also intrusive. A tracer can slow the target, change timing, and expose sensitive arguments. The goal is therefore not to collect everything. It is to capture the smallest trace that can confirm or reject a specific hypothesis.

A system call is one transition through the user/kernel interface. strace records that transition and normally displays:

The relationship to an application request looks like this:

The trace does not automatically show which source-code function initiated the call, which customer request it belongs to, or the complete path taken inside the kernel. It observes inputs and outputs at one boundary.

This distinction prevents several incorrect conclusions. If a thread spends time computing entirely in user space, there may be no system calls to show. If a system call lasts 500 milliseconds, that does not mean the CPU executed kernel instructions for the entire interval. The thread may have slept, waited for a resource, or been descheduled.

System-call evidence is strongest when it is correlated with an exact process, thread, operation, and time window.

System-call tracing is useful when the symptom involves an operating-system service:

It is less useful when the suspected work occurs entirely in user space. A CPU-intensive algorithm, a runtime garbage collector, a user-space spin lock, or incorrect application data may produce little meaningful syscall activity. A trace can still establish that the process is not crossing the kernel boundary often, but another type of evidence is then needed to locate the user-space work.

Prefer a trace of an isolated reproduction when possible. Startup failures, command-line tools, failing test cases, and one-off batch jobs are ideal because the tracer can launch the program and observe the complete operation. Attaching to a busy production service is riskier and produces a mixed stream from unrelated requests.

Before attaching a tracer, use lower-overhead state to see whether the process is running or waiting:

-L lists threads. The state column distinguishes runnable tasks from tasks that are sleeping, and wchan shows the symbolic kernel wait location when it is available. A snapshot cannot tell you how long the thread has been there, but it helps select the thread and syscall family worth investigating.

Linux also exposes the currently executing system call for a task:

This file may contain a system-call number, raw argument-register values, a stack pointer, and a program counter. It may instead report running, or use -1 when a blocked task is not currently inside a system call.

This is a momentary, architecture-specific view. The call can change immediately after the file is read, and a raw syscall number must be interpreted using the target process's architecture and ABI. It is useful for a quick clue, not for reconstructing a timeline.

To trace a command from its beginning:

The options provide a practical diagnostic starting point:

Tracing from launch captures dynamic-loader activity before main() begins. A normal dynamically linked program may open libraries, map files, inspect system configuration, and query locale data during startup. This can create hundreds of lines unrelated to the operation being investigated.

For a startup failure, that early activity is exactly what you need. For steady-state request latency, either attach after startup or align the trace with the known request time.

To observe a running process:

When -f is combined with -p, strace attaches to all threads in the target thread group and follows newly created descendants. Without -f, attaching to the process ID may trace only the thread whose ID was supplied. That can miss the worker actually handling the slow operation.

Pressing Ctrl-C normally tells strace to detach, leaving an attached service running. Verify that the tracer has exited and that the service remains healthy.

For a multithreaded process, separate files can make the output easier to inspect:

-ff follows the same process and thread activity but writes each traced process to a file whose name ends in its PID. Separate files simplify per-thread reading, while a combined file makes chronological relationships across threads more visible. Choose based on the question being investigated.

Permissions Are a Security Boundary

strace uses Linux tracing controls associated with ptrace. Attaching may be denied because:

Do not respond to a permission failure by globally weakening kernel.yama.ptrace_scope or granting broad capabilities without authorization. Those restrictions help prevent one compromised process from reading another process's memory and credentials. Use the service's approved debugging path, a privileged diagnostic container with the required scope, or an isolated reproduction.

Traditional strace stops a traced task at syscall entry and exit so the tracer can inspect it. A syscall-heavy process can therefore slow significantly, and tracing may perturb the race or latency being investigated. Attaching can also interrupt a small class of non-restartable calls on some systems.

Reduce risk by making four decisions before tracing:

Avoid an unfiltered, indefinite trace of a high-throughput production service. Start with a short capture during a known request, and stop as soon as the necessary event is recorded. Monitor the service while tracing and abandon the capture if impact increases.

Trace files can contain filenames, command arguments, portions of file content, process IDs, and decoded descriptors. Larger -s values reveal more string data. Full read or write buffer dumps reveal much more. Store traces with restrictive access, avoid capturing secrets unnecessarily, and follow the system's data-retention policy.

Tracing every syscall often hides the useful lines in loader activity, time queries, memory management, and background threads. -e trace= selects a smaller set.

For a missing or inaccessible file:

%file includes syscalls that take a pathname, such as open and status operations. Using the group is safer than remembering every modern variant such as openat() and newfstatat().

For a suspected wait or retry loop:

Other useful groups include %desc for file-descriptor operations, %process for process lifecycle, %memory for memory mappings, and %signal for signal-related operations. Support for individual syscall names and newer filter features depends on the installed strace version and the target ABI. Check strace -V and strace --help on the system being investigated.

A path filter can narrow file activity further:

Filters are not neutral when interpreting absence. If the trace contains no writes because only %file calls were selected, it does not prove the process performed no writes. Record the exact command with the trace so later readers know what was excluded.

Consider this simplified line:

Read it from left to right:

14:07:18.441203 is the wall-clock entry time added by -tt. Use it to align the call with application logs and request traces.

openat is the kernel interface invoked. The application source may have called open(), a runtime helper, or a higher-level configuration function. Library wrappers do not always map one-to-one to syscall names.

The values in parentheses are decoded arguments. Here the process asks to open a path for reading and to close the resulting descriptor automatically across a successful program execution.

= -1 ENOENT is the result. The negative result indicates failure, and ENOENT names the error. Successful calls can return zero, a byte count, a new descriptor, a process ID, or another nonnegative result depending on the interface.

<0.000052> is elapsed wall time between entry and return, added by -T. The failed lookup completed in 52 microseconds. It did not cause a one-second delay by itself.

When an argument points to user memory, strace may dereference and format the data for readability. The quoted string is a decoded representation; the kernel interface actually received an address and other arguments.

The duration printed by -T is elapsed time, not pure kernel CPU time. It can include:

Suppose a trace contains:

The thread spent about five seconds inside poll(), which returned because its timeout expired. This does not show that poll() itself is inefficient. It shows that none of the requested events became ready within the supplied interval. The next question is why the program expected readiness and what should have produced it.

A long wait is not automatically pathological. An idle event loop can spend almost all its time in epoll_wait(), and an unused worker can wait indefinitely in futex(). Those are efficient ways to sleep. A long call matters when the affected request or thread needed it to finish sooner.

For a short, reproducible operation, summary mode can identify high-count or error-heavy syscall families:

-c reports call counts, error counts, and summarized time per syscall instead of printing every event. By default, the time portion represents system CPU time, not elapsed wait time.

When the installed version supports it, add -w to summarize entry-to-return wall time:

This is more useful for finding calls that block or sleep. It still needs careful interpretation. In a multithreaded program, calls overlap, so adding their wall durations does not reconstruct end-to-end request latency. The tracing overhead also affects short, frequent calls.

Use the summary to choose a focused detailed trace. It can suggest that openat() fails thousands of times or that most observed wall time appears in futex(), but it cannot show the sequence of events, the arguments, or which particular call belongs to the symptom.

Many syscalls operate on numeric file descriptors rather than paths. To understand a later read(7, ...), find how descriptor 7 was created:

The return value from openat() establishes the descriptor. Later operations use the number until it is closed. After close(7), the process may reuse 7 for a completely different object.

The -y option helps by annotating known descriptor paths:

Do not assume that one descriptor number has one identity for the process's entire lifetime. Follow creation, duplication, inheritance, and close operations within the relevant interval.

Return counts matter as well. read(..., 4096) = 128 is a successful short read of 128 bytes, not a failed request for 4,096 bytes. read(...) = 0 normally indicates end-of-file for a regular file or that the peer has closed a byte stream. Interpret the result according to the specific syscall's manual page.

The error column is visually prominent, but not every failed syscall represents an application bug.

Programs routinely probe several paths until one succeeds. Dynamic loaders search for libraries. Nonblocking operations use EAGAIN to say that progress cannot be made immediately. A process may intentionally test for a file and treat ENOENT as “feature not configured.”

The important questions are:

For example:

This can be normal fallback behavior. In contrast, hundreds of EACCES results on the required configuration path followed by process exit form a meaningful failure sequence.

The exact meaning of an error depends on the interface. Use the relevant manual page:

Do not infer a high-level cause from an errno name alone. ENOMEM, for example, does not always mean the host has no free RAM; the operation and its narrower limits determine the meaning.

In a combined trace, one thread can enter a syscall while another thread produces output. strace represents this with unfinished and resumed lines:

The first and third lines are two parts of one futex() call. Do not count them as separate calls or treat the unfinished marker as a failure.

Signal delivery can interrupt a restartable syscall:

ERESTARTSYS in a trace is an internal restart indication, not normally an errno returned directly to application code. The kernel and runtime may restart the operation after the signal handler. In other cases the application can observe EINTR and must decide whether to retry.

Signals matter when their timing or frequency explains the symptom. A single routine SIGCHLD line is not a diagnosis.

A Slow Blocking Call

A request-correlated read(), fsync(), poll(), or futex() has a large -T duration. This identifies where elapsed time became visible at the system-call boundary. It does not by itself identify the lower resource, event producer, or lock owner.

Fast Calls Repeated Too Often

No individual call is slow, but summary mode shows hundreds of thousands of calls. Repeated one-byte reads, tiny writes, metadata probes, or time queries can make boundary-crossing overhead material. Compare the count per useful operation with a healthy run rather than assuming every high count is waste.

A Busy Retry Loop

The same syscall returns EAGAIN or another transient condition repeatedly with no blocking wait or backoff between attempts. This pattern can consume CPU while making little progress. Verify that timestamps show a tight loop and that the application is expected to wait for readiness.

Repeated Failure Followed by Sleep

A lookup or operation fails quickly, followed by nanosleep() or clock_nanosleep(). The externally visible delay comes mostly from retry policy, while the failed syscall explains what condition triggered the policy.

Waiting as Healthy Behavior

An idle worker blocks in futex() and an event loop blocks in epoll_wait(). Long durations here can indicate that the program is efficiently idle. Tie the thread to active work before labeling the wait a problem.

The absence of a syscall does not prove that the corresponding high-level operation never occurred.

Some library operations are completed entirely in user space. Linux can provide selected time and CPU information through the vDSO without a traditional system-call transition. A cached file read may still call read(), but it can complete without physical storage I/O. Buffered writes can return before data is durable.

Asynchronous interfaces can also separate submission from completion. With an interface such as io_uring, many operations are described through shared memory and a smaller number of syscalls. A syscall-only trace may therefore show ring management rather than one conventional call per I/O operation.

strace also does not inherently provide:

State the conclusion at the boundary the tool observes. “Thread 2518 waited 800 milliseconds in this futex() call” is supported. “Function refreshCache() held the lock for 800 milliseconds” requires additional evidence.

An internal report API begins taking slightly more than two seconds for its first request after a deployment. CPU and device metrics remain normal. Application logs report only:

The issue reproduces in a staging instance with the same configuration, so the operator traces one request there instead of attaching to the production process.

A wall-time summary shows that most observed elapsed time is associated with clock_nanosleep(). A focused trace of file and sleep operations contains this repeating sequence:

The pattern repeats eight times before the request fails.

The trace separates trigger from delay:

Inspection of the deployment configuration shows that the catalog volume is now mounted at /run/acme-state, while the service still checks /run/acme/catalog.ready. Updating the configured path causes the first openat() to succeed. The sleeps disappear and request latency returns to normal.

A precise root-cause statement is:

The deployment changed the catalog mount path without updating the readiness-file setting. Each first request performed eight failed readiness checks separated by 250-millisecond backoffs, producing the two-second delay.

The system-call trace did not merely expose ENOENT; it revealed the ordered mechanism connecting a configuration error to the user-visible latency.

Use the following sequence when syscall evidence is likely to reduce uncertainty:

The investigation is complete when the trace supports a causal mechanism, not when it merely produces an unfamiliar syscall name.

System-call tracing reveals what a process requests from the Linux kernel, the arguments supplied, the results returned, and the elapsed time across each call. It is most useful for investigating failed resource access, blocking operations, retry behavior, signals, and excessive syscall frequency.

Use strace deliberately. Prefer isolated reproduction, trace all relevant threads, filter around a hypothesis, capture a short time window, and protect sensitive output. Interpret durations as elapsed time rather than pure kernel execution, and interpret errors within the surrounding control flow.

A valuable trace connects a precise application symptom to an ordered mechanism: which call occurred, what it returned, what the program did next, and how that sequence produced the observed delay or failure.