An API deployment is followed by a 40% increase in CPU time per request. Host and container metrics confirm real CPU pressure, and thread-level inspection identifies the busy workers. The remaining question is more specific:

Logs rarely answer this. They describe events the application chose to record, not every function that executed between those events. Adding logs around hundreds of functions would be slow, intrusive, and difficult to interpret.

A CPU profiler answers the question statistically. It observes the running program at intervals, records the active call stacks, and aggregates repeated observations. A flame graph then turns those stacks into a compact picture of where sampled CPU execution occurred.

A sampling profiler periodically records the instruction being executed and, when call-chain collection is enabled, the chain of function calls that led to it.

Suppose a profiler collects 10,000 on-CPU samples from a service. If 3,700 samples contain escapeJsonString, then approximately 37% of the sampled CPU execution passed through that function during the recording:

This is an estimate, not an exact accounting of every call or CPU cycle. Sampling may catch one invocation several times and another not at all. A larger set of representative samples reduces random noise, while a badly chosen target or time window produces a precise-looking answer to the wrong question.

Sampling is different from tracing. A function tracer may record each selected call, its arguments, or its duration. A sampling profiler records occasional snapshots. Sampling usually has much lower overhead and naturally emphasizes code that consumes more of the sampled event.

The sample count for a caller includes samples in its descendants. In this example, observations in escapeJsonString also contribute to serializeResponse, handleRequest, and workerLoop. Inclusive percentages are therefore nested, not additive.

Linux perf can sample many hardware and software events. For a first CPU hotspot profile, the useful choices are usually:

Choose the event explicitly so that the recording is reproducible:

Available hardware events depend on the processor, kernel, virtualization environment, and security policy. A virtual machine may expose fewer counters than its host. If cycles is unavailable, cpu-clock is a practical starting point for locating on-CPU code.

An on-CPU profile only samples execution while the target is running on a CPU. It does not explain time spent asleep, blocked on a lock, waiting for storage, queued for a remote service, or otherwise off CPU. If requests are slow while the service uses little CPU and has little runnable delay, an on-CPU flame graph is unlikely to reveal the latency source.

The event can also be restricted by privilege level:

The :u form samples user-space execution, and :k samples kernel execution. An unrestricted cycles event normally attempts to include both when permissions allow it. Start with the scope required by the question. A user-only profile is useful for application code, but it deliberately excludes CPU consumed on the application's behalf inside the kernel.

Every percentage in a profile is conditional on four things:

“Function X is 30%” is incomplete. A meaningful statement is: “Function X appeared in 30% of the cycles samples collected from process 2471 during the 30-second request peak.”

perf record can profile one process, one thread, a newly launched command, selected CPUs, or the entire system.

Attach to a running process with:

This selects the running process rather than launching it. If earlier evidence identifies one hot thread, target its thread ID:

To profile a program from its beginning, place the command after --:

System-wide recording uses -a. It is appropriate when the question genuinely spans the host, such as CPU work moving across many processes and kernel threads. It also collects much more unrelated and potentially sensitive information. Do not begin with -a merely because the relevant PID has not been identified.

Confirm the target immediately before recording:

A stale PID can be reused by an unrelated process after a restart. In a container, the PID visible inside the container may differ from the host PID used by a host-level perf. Resolve the actual process in the namespace where the profiler runs, and keep its executable version or build identifier with the recording notes.

Performance counters reveal detailed behavior, so Linux restricts access to them. The current policy is visible at:

The exact operations allowed at each value vary across kernel and distribution policy. Systems can also authorize a profiling tool through capabilities such as CAP_PERFMON. Follow the environment's approved operational path rather than weakening a host-wide security setting for convenience. If elevated access is required, use it only for the bounded command and target already identified.

Verify that perf is installed and usable:

Some distributions package kernel-specific performance tools separately. A tool that cannot interpret the running kernel's data or locate its symbols may produce incomplete results even when recording succeeds.

Treat profiles as operational data. They can expose process names, function names, executable paths, loaded libraries, and details about proprietary code. Store perf.data, exported stacks, and SVG files according to the same access rules as other production diagnostic artifacts.

The following command attaches to process 2471, requests 99 samples per second, collects call chains using DWARF unwind information, and stops after 30 seconds:

Use sudo only when the approved access model requires it. The important properties are the explicit event, target, frequency, call-graph method, output path, and duration.

At 99 Hz, a thread that remained continuously on CPU for 30 seconds could contribute roughly 2,970 samples. A thread that used one-tenth of a CPU would contribute far fewer. The actual count depends on scheduling, counter availability, throttling, and lost samples.

A non-round frequency such as 99 Hz reduces the chance of repeatedly sampling at the same point in periodic application work. It is a reasonable starting point, not a universal setting. Raising the frequency increases detail but also increases profiler interrupts, data volume, and unwind work. Measure profiling overhead on a representative environment before using an aggressive rate in production.

To profile a short-lived command:

Choose a recording interval that contains the symptom. A two-minute profile that includes five seconds of regression and 115 seconds of normal operation will mostly describe normal execution. Several short recordings around distinct states are often easier to interpret than one long, mixed recording.

perf reports collection statistics when it exits. Warnings about lost samples, event throttling, or unavailable counters weaken the result and should be preserved with the profile. Sampling overhead is never exactly zero, so also watch whether the application changes behavior during collection.

The sampled instruction identifies a flat hotspot, but most useful diagnoses require its ancestry. For example, a JSON encoder may be hot because one request path invokes it excessively, not because the encoder itself recently changed. Call chains preserve that context.

Linux perf commonly supports three call-graph methods:

Frame-pointer unwinding (--call-graph fp) follows a linked chain of stack frames. It has relatively low collection overhead and works well when every relevant binary and library preserves frame pointers. Optimized builds that omit them can produce shallow or broken stacks.

DWARF unwinding (--call-graph dwarf) records stack data and uses compiler-generated unwind information. It can reconstruct stacks when frame pointers are unavailable, but it creates larger recordings and adds collection and analysis cost. Missing unwind metadata, unusual runtime stacks, and insufficient captured stack data can still break the chain.

Last Branch Record (--call-graph lbr) uses processor branch-history hardware. Availability, maximum depth, and interactions with other events depend on the architecture and processor. It is useful in suitable environments but is not the portable default.

Prefer frame pointers when the complete deployed stack is known to preserve them. Otherwise, DWARF is a common starting point. Do not decide solely from the absence of an error message: inspect the resulting call chains. If most samples stop after one or two frames, the profiler did not collect enough ancestry to support a call-path conclusion.

Different language runtimes add their own considerations. Native ahead-of-time compiled programs often work with frame pointers or unwind metadata. Just-in-time compiled runtimes may need runtime-specific symbol export or JIT integration. Managed runtimes, interpreters, coroutines, and user-space schedulers can make a kernel-visible stack differ from the application's logical stack.

Open the interactive report:

perf report distinguishes Self and Children overhead. Self overhead counts samples assigned directly to a symbol. Children overhead accumulates samples from descendant functions into their callers.

For a flat, direct-cost view:

A flat report is useful for locating leaf functions that were frequently executing when sampled. The call-chain view explains which paths reached those functions. Use both; they answer different questions.

Suppose a report contains:

If escapeJsonString has 37% self overhead, it was directly executing for a large share of samples. If serializeResponse has 48% children overhead but little self overhead, its cost lies mostly in callees. If handleRequest has 62% inclusive overhead, that 62% already includes both descendants. Adding 62%, 48%, and 37% would count many of the same samples repeatedly.

Before drawing a flame graph, check that:

This validation prevents a polished visualization from hiding weak input data.

A profiler records instruction addresses. Symbolization maps those addresses back to function names, shared libraries, and sometimes source lines. Missing or mismatched symbol information commonly produces hexadecimal addresses, [unknown] frames, or misleading names.

Useful analysis requires the exact executable and libraries used during recording. Optimized production binaries can be stripped as long as matching separate debug information or suitable symbol data is available through the organization's symbol workflow. Build IDs help perf associate recorded mappings with the correct artifacts.

Analyzing a profile on another host is risky if that host has a different build. Preserve the build identity and obtain the matching binaries and debug symbols rather than substituting a newer executable with the same filename. Any archive containing production binaries or symbols must be handled as sensitive code.

Even correctly symbolized optimized code is not a literal record of source calls. Compilers inline small functions, merge equivalent code, remove unused paths, and replace some calls with tail jumps. An inlined function may appear as part of its caller, while a source-level call frame may not exist at runtime.

Repeated [unknown] blocks are a data-quality signal, not a function to optimize. Common causes include missing symbols, broken unwinding, JIT-generated code without profiler integration, inaccessible kernel symbols, or mappings that changed before analysis. Resolve enough unknown space to support the conclusion, especially if it occupies a large share of the graph.

The standard FlameGraph scripts are separate from perf. Once they are installed in an approved location, the transformation has three stages:

perf script emits each sampled stack. stackcollapse-perf.pl combines identical stacks into folded form:

The semicolon-separated functions describe one call path from root to leaf, and the final number is its aggregated sample count. flamegraph.pl converts those counts into an interactive SVG.

Keep the raw perf.data file. Rendering options can be changed later, but information discarded before or during stack conversion cannot be recovered from the SVG.

A flame graph arranges aggregated stacks according to a few rules:

A wide box near the top often identifies direct CPU work. A wide box lower in the graph identifies an expensive path including its descendants. The lower function is not necessarily the code performing the work; it may be a dispatcher or request handler through which many expensive paths pass.

The default warm colors make stack shapes easy to distinguish. They do not mean that red functions are hotter than yellow functions. Width carries the sample magnitude in a normal flame graph. Color gains quantitative meaning only in specialized renderings, such as a differential flame graph, whose legend and generation options must be understood.

Most generated SVGs support searching for a function and clicking a frame to zoom into its subtree. Zooming changes the displayed denominator. A function shown as 60% after zooming may represent 60% of that subtree, not 60% of the original process profile.

Recursion can place the same function in several vertical frames. Asynchronous runtimes may expose scheduler or executor frames as broad bases. Shared library routines such as allocation, copying, compression, or encryption may appear under several unrelated callers. The caller path determines which application behavior should be changed.

A single flame graph shows where samples occurred, but it does not establish what changed. For a regression, collect a baseline and degraded profile with comparable:

Generate a folded stack file for each recording. A differential flame graph can then color stacks by the change between them:

With the standard differential workflow and the files ordered as above, frame widths show their population in the second, degraded profile. Red indicates growth from baseline to degraded, while blue indicates reduction. Color represents the change in a function's direct contribution, not the sum of its descendants. Rendering options such as --negate can reverse the color convention, so preserve the command and read the generated legend.

Width and color therefore communicate different properties: a narrow frame can have a large relative change, while a very wide frame can remain nearly unchanged. A path that disappeared entirely from the second profile has no width in this view, so retain and inspect both ordinary flame graphs as well.

If the recordings contain different total sample counts because of an unrelated load difference, difffolded.pl -n can normalize the first profile to the second before calculating colors. Normalization compares proportions; it deliberately removes the total-count difference. Decide whether that is appropriate from the question rather than applying it automatically.

Profile percentages are ratios. If a function falls from 30% to 20% while total service CPU doubles, its absolute CPU use may still have increased. Compare the visualization with an external measure such as CPU seconds per request, total process CPU, throughput, and request mix.

Statistical noise also matters. A difference supported by twelve samples is much less convincing than one supported by thousands. Repeat the recording when the apparent regression is close to the profiler's expected variation.

Consider an API whose throughput falls after a deployment. Traffic is unchanged, four worker threads stay busy, and CPU seconds per request increase by 35%. This establishes CPU work, not merely elapsed latency, as a meaningful part of the regression.

An engineer records two 30-second profiles at the same request rate and with the same cycles event, frequency, and call-graph method. The baseline profile comes from the previous build; the degraded profile comes from the new build. Both have complete application stacks and matching symbols.

In the degraded flame graph, the following path occupies a large region:

These numbers must not be added. The 71% for handleRequest includes the samples attributed to both descendants. The important shape is a broad escapeJsonString block at the top of the stack: the CPU was directly executing that leaf for 38% of the process samples.

In the baseline, the same leaf accounts for only 8%. A differential graph highlights growth on this exact call path. Source inspection then finds that a newly added metadata object is serialized inside a loop for every response item, even though the object is identical for the entire response.

The team moves the serialization outside the loop and repeats the same workload. CPU seconds per request return close to the baseline, the wide leaf shrinks, and throughput recovers.

The profile did not prove the source-level bug by itself. It narrowed the search to a code path, the code explained why that path expanded, and a controlled change verified causality.

The most damaging mistake is profiling the wrong state. Confirm workload symptoms, process identity, and recording timestamps before interpreting individual functions.

Other recurring errors include:

A flame graph is persuasive because it is visual. Its validity still comes from experimental discipline: a scoped question, representative recording, sufficient samples, trustworthy stacks, and a repeatable comparison.

Use the following sequence for a CPU code-path investigation:

This workflow turns a flame graph from an attractive picture into evidence.

Sampling profilers estimate where a target spends a selected on-CPU event by periodically recording instructions and call stacks. A useful profile must define its event, process or thread, workload, time window, frequency, and unwind method.

In a flame graph, width represents sample count, vertical position represents call depth, and horizontal position is not time. Wide top frames suggest direct CPU work; wide lower frames show inclusive paths through their descendants. Symbol quality and complete call chains are prerequisites for trustworthy interpretation.

For a regression, compare profiles collected under equivalent conditions and relate their percentages to absolute measures such as CPU seconds per request. Profiling locates expensive execution paths; source inspection and repeated measurement establish the cause and verify the fix.