Traditional metrics can show that something is wrong without revealing why. A service might pause for 400 milliseconds even though CPU usage is low and everything else looks normal. Tracing every system call could expose the delay, but on a busy service it would generate enormous output and add unnecessary overhead.

A better approach is to capture only the relevant events and summarize them as they happen, perhaps as a latency distribution.

This is a core use case for eBPF. Small programs attach to selected events in the kernel or user-space applications. They can filter, count, time, and aggregate data before sending the results to user space. bpftrace makes this capability accessible through a compact tracing language.

Many diagnostic tools collect every event in user space and analyze it afterward. This becomes expensive when the event is frequent. A server might make hundreds of thousands of system calls per second, but the investigator may need only a histogram of read() latency for one PID.

An eBPF program can perform that filtering and aggregation at the event source:

The target does not need to be recompiled for many dynamic probes. The eBPF program is loaded into the kernel and attached to an existing hook, such as a system-call tracepoint, kernel function, user-space function, or periodic sampling event.

This does not make observation free. The attached program still executes whenever its probe fires. Its cost depends on the event rate, the work performed for each event, map contention, and how much data is copied to user space. eBPF makes efficient aggregation possible; it does not guarantee that every eBPF tool is efficient.

eBPF is a Linux kernel execution mechanism. It defines an instruction set, program types, maps, helper functions, loading rules, and attachment mechanisms. Networking, security, scheduling, and observability tools can all use it.

bpftrace is a user-space tool and high-level language for dynamic tracing. It performs several jobs:

A bpftrace script is therefore not interpreted inside the kernel as text. It is compiled into constrained programs, while output formatting and some other operations happen in the bpftrace process.

The kernel verifier analyzes each program before accepting it. Among other checks, it reasons about control flow, memory access, pointer use, helper calls, and whether execution is bounded. Accepted bytecode may be interpreted or translated into native machine instructions by a just-in-time compiler.

Verifier acceptance establishes that a program satisfies the kernel's safety rules. It does not establish that the program asks the right diagnostic question, has low overhead, protects sensitive data, or interprets kernel fields correctly.

An eBPF program is invoked for one event and then returns. Persistent state lives in BPF maps. A map can hold counters, timestamps, histograms, stack identifiers, or other key-value data that is shared across invocations and accessible from user space.

In bpftrace, map names begin with @:

The values can be updated where the event occurs and printed later. This is the important performance difference between these two approaches:

The first emits one record for every event. The second updates an aggregate and normally produces output only when the map is printed. For high-frequency events, aggregation is usually safer and more useful.

bpftrace also has scratch variables whose names begin with $. A scratch variable exists only while one action block is executing:

Use $ for a temporary calculation and @ when state must survive across events or be shared between probes.

A bpftrace action block has three parts:

For example, this program counts openat() calls made by process 2471 and stops after ten seconds:

The probe identifies the event. The predicate limits the action to the selected process. The action updates a map. The interval probe bounds the recording.

Maps that contain data are normally printed when bpftrace exits, including after Ctrl-C. A possible result is:

That count is meaningful only with its scope and duration: process 2471 entered openat() 1,842 times during the ten-second observation.

Useful built-in values include:

comm is a short task name, not a globally unique process identity. Different processes can share it, and long names may be truncated. Prefer a verified PID, cgroup, or another unambiguous scope when the conclusion concerns one service.

Probe names, fields, and features vary with the kernel, architecture, bpftrace build, and target executable. Discover them on the system being observed:

--info reports features supported by the current kernel and bpftrace build. Modern installations can use BTF, or BPF Type Format, to obtain type information about the running kernel:

The absence of that file does not mean all tracing is impossible, but typed probes and portable access to kernel structures may be limited.

List matching probes rather than guessing their names:

Inspect a probe's available fields:

Typical fields include a directory file descriptor, filename pointer, flags, and mode. The actual listing is authoritative for that host. A script copied from a different kernel may refer to a field that does not exist locally.

Probe discovery is also a safety step. A wildcard such as kprobe:* may match tens of thousands of functions. Listing the expansion reveals the true attachment scope before the script runs.

Different probe providers expose different stability, context, and overhead characteristics.

Tracepoints

Kernel tracepoints are static instrumentation sites compiled into the kernel. System calls, scheduling, block I/O, networking, and many other subsystems expose them.

Tracepoints generally provide a more stable interface than internal kernel functions, and their named fields are discoverable with -lv. Prefer a suitable tracepoint when one exists. “More stable” does not mean identical across every kernel version, so validate the fields on the target.

Kprobes and Kretprobes

A kprobe dynamically instruments the entry to a kernel function; a kretprobe observes its return.

These probes reach internal implementation details when no tracepoint exposes the required event. That flexibility comes with portability risk. Kernel functions can be renamed, inlined, optimized away, or have their arguments changed. A kprobe script should be tied to a known kernel build and tested again after upgrades.

Function arguments for traditional kprobes are commonly accessed through arg0, arg1, and similar built-ins. Their meaning comes from the exact function prototype and architecture calling convention. Guessing an argument's type can produce invalid data or verifier rejection.

Fentry and Fexit

On systems with suitable kernel and BTF support, fentry and fexit probes attach to kernel function entry and exit with typed argument information. They can be more direct and efficient than traditional kprobes:

They still observe internal kernel functions rather than a stable application interface. BTF helps interpret the current build; it does not promise that the same function will exist in a future build.

Uprobes and Uretprobes

Uprobes dynamically instrument functions in user-space executables or shared libraries:

They are useful when an application has no built-in observability for a specific function. Their reliability depends on the exact executable, symbols, debug information, compiler optimization, and language runtime. An inlined function may have no independently attachable entry. A stripped or just-in-time compiled program may require runtime-specific support.

USDT Probes

User Statically Defined Tracing probes are intentional instrumentation points compiled into an application or runtime. When available, they provide application-level events such as request start, garbage collection, or query execution without depending on an internal function name.

USDT is often preferable to an arbitrary uprobe because the application explicitly defines the event and its arguments. Availability and semantics still depend on the application build.

Profile and Interval Probes

A profile probe fires at a requested frequency on each CPU. It can build a stack profile:

An interval probe is different: it fires periodically for script control, such as printing a map or stopping a bounded observation. It does not sample every target CPU in the same way as a profile probe.

Start with an aggregate rather than a stream of raw events. This program counts all system-call entries by process name for ten seconds:

The raw syscall tracepoint fires very frequently, so even this small program should be used for a deliberate, bounded observation. Grouping by comm can identify candidates, after which a PID-scoped script can answer a narrower question.

For process 2471, summarize successful read() return sizes:

hist() creates power-of-two buckets. A distribution preserves structure that an average hides. Thousands of 64-byte reads mixed with a few one-megabyte reads may have an unremarkable average but very different performance implications.

Other aggregation functions include sum(), min(), max(), avg(), and lhist(). lhist() creates fixed-width linear buckets when meaningful thresholds are known. Choose the representation from the question rather than collecting every statistic.

Use count() instead of a raw @counter++ for concurrent events. count() uses a map representation designed for safe, efficient updates across CPUs, while an ordinary increment of a shared value can lose updates.

Many latency questions require two events: record a start time at function or syscall entry, then calculate elapsed time at exit. The thread ID is a useful correlation key because one thread cannot execute two system calls simultaneously.

Save the following as read-latency.bt:

Run it with the access required by the system:

@start and @fd preserve entry data for each thread. The exit probe ignores calls whose entry was not observed, calculates microseconds, adds the value to a histogram keyed by file descriptor, and deletes the temporary state. END clears any unfinished state so those internal maps are not printed.

The measured interval is wall-clock time from syscall entry to return. It can include time sleeping for data and time descheduled from the CPU. It is not a measurement of pure kernel CPU execution.

Correlation state must always be removed. A map keyed by transient thread IDs, request IDs, or pointers can grow until it reaches its configured capacity if completed operations are never deleted.

First discover whether the expected symbol exists in the exact deployed executable:

If serializeResponse is present, count its calls for one process:

The binary path identifies the object being instrumented, while the predicate prevents calls from other processes using that object from entering the aggregate.

Do not infer source-level behavior from the symbol name alone. Compiler inlining, tail calls, function cloning, language wrappers, and runtime-generated code can change the relationship between source functions and executable instructions. Return probes can also interact poorly with runtimes that manage unusual stacks or return addresses. Prefer a supported runtime profiler or USDT probe when it expresses the same question more reliably.

A useful cost model is:

Apply that model before attaching:

Predicates reduce the work after a probe fires, but the attached program must still be invoked to evaluate the predicate. A PID filter on every system call is much cheaper than printing every system call, yet it is not equivalent to having no probe.

printf() is asynchronous. The eBPF side sends an event to a buffer, and the bpftrace process formats it later. If events arrive faster than user space drains the buffer, records can be lost. More buffering delays the problem and consumes more memory; it does not repair an unnecessarily noisy script.

exit() is also handled asynchronously, so an interval-bounded script can process a small number of additional events while termination reaches user space. Treat the requested duration as a tight operational bound, not a nanosecond-precise cutoff.

Maps avoid per-event output but introduce their own limits. A map keyed by PID has bounded practical cardinality on a host. A map keyed by arbitrary filenames, stack traces, or network addresses can grow rapidly. Use deliberate keys, clear maps when producing periodic windows, and pay attention to allocation or update failures.

Where the installed version supports it, test attachment without collecting a full observation:

This checks parsing, loading, verification, and attachment on that host. It does not exercise the script under real event volume, so overhead and output behavior still require a representative test.

Loading eBPF programs and attaching tracing probes is privileged on most production systems. The exact requirements depend on kernel version, configuration, lockdown mode, probe type, and distribution policy. Modern kernels separate some authority into capabilities such as CAP_BPF and CAP_PERFMON; older or more restrictive systems may require broader privilege.

Do not respond to an authorization failure by globally enabling unprivileged BPF or granting a container unrestricted host access. Use an approved observability role, a controlled diagnostic environment, or an isolated reproduction.

Running bpftrace inside a container does not create a private eBPF kernel. Containers share the host kernel. PID namespaces may change the number used to identify a process, cgroups may be a better workload boundary, and the tracer still needs host-kernel tracing authority.

Observability programs can expose filenames, process activity, memory-derived strings, stack symbols, and proprietary implementation details. Some eBPF program types and explicitly unsafe bpftrace operations can also affect system behavior. The verifier is not a privacy policy or an authorization system.

In the normal unpinned bpftrace workflow, exiting the process closes its BPF object references, detaches the probes, and releases their maps. Verify that the tracer exits after the capture. Persistent pinned programs and maps have different lifetimes and should not be introduced during an ad hoc investigation.

Record the script, operator, target, host, kernel version, bpftrace version, start time, and duration. This makes the result auditable and helps another engineer reproduce it.

A verifier error means the kernel could not prove that the generated program obeys its rules. Common causes include unsafe pointer access, unsupported helpers, excessive complexity, incompatible types, or features missing from that kernel. Read the complete error rather than repeatedly adding privilege.

An attachment error often means the probe does not exist, is not traceable, or requires unsupported kernel facilities. Re-run bpftrace -l for the exact attach point and inspect bpftrace --info.

A script that attaches but prints nothing may be correct. The event might not have occurred during the window. It can also indicate the wrong PID namespace, an incorrect predicate, a wrapper that uses a different syscall, or a user-space symbol from a different binary build.

Incorrect fields and implausible values usually indicate a probe/argument mismatch. Prefer typed tracepoint or fentry arguments when available, and never treat arg0 as a particular structure merely because a script for another kernel did so.

Lost-event warnings mean the output path could not keep up. Reduce event volume and aggregate first. A larger buffer is a secondary measure, not the first fix.

An API has periodic latency spikes while CPU and disk utilization remain low. Thread snapshots show workers sleeping, but application logs identify no slow dependency.

A 15-second read() latency script produces:

File descriptor 18 repeatedly spends hundreds of milliseconds inside read(), while descriptor 24 returns in tens of microseconds. The histogram does not yet identify either resource.

The investigator checks the live descriptor:

It is a socket. Socket inspection maps it to a connection from the API to one cache shard. A recent configuration change directed most traffic to that shard, whose request queue is saturated.

Descriptor numbers can be closed and reused, so this lookup is performed during the same capture window and against the same verified process instance.

The eBPF result did not prove that the kernel's read() implementation was slow. It showed that calls on one descriptor had long entry-to-return latency. Descriptor and socket evidence identified the remote dependency, and correcting the cache routing removed both the long histogram mode and the request-latency spikes.

This is the right role for dynamic tracing: measure a narrowly defined gap, then combine the result with independent evidence.

Use this sequence when a normal metric cannot answer the question:

If an existing low-overhead tool already answers the question, use it. Dynamic tracing is most valuable when the required event or correlation is absent from standard metrics, not simply when eBPF is available.

eBPF executes verified programs at selected Linux events. BPF maps allow those programs to preserve state and aggregate observations before user space reads them. bpftrace provides a high-level language that compiles action blocks into eBPF, attaches them to probes, and formats their results.

Tracepoints and USDT probes are usually preferable when they expose the needed event. Kprobes, fentry/fexit probes, and uprobes reach deeper implementation details but require careful validation against the exact kernel or binary.

Efficient tracing comes from narrow scope, bounded duration, early predicates, in-kernel aggregation, controlled map cardinality, and minimal event output. A verifier-approved script can still be expensive or misleading, so its result must be interpreted from the precise hook, fields, and time interval it measured.