A 16-CPU host reports only 35% total CPU usage, yet one API instance has severe latency spikes. Is CPU capacity ruled out?

Not necessarily. One thread may be saturating one CPU. The service may be restricted to a small CPU set. A container may consume its CPU quota early in each enforcement period and then be throttled. Host averages can hide all three conditions.

CPU investigation therefore requires more than finding a large %CPU value. It must answer:

CPU utilization measures activity. CPU saturation measures unmet demand. A CPU bottleneck exists when limited CPU service delays useful work enough to cause the observed problem.

Define the affected operation before opening a CPU tool. Record its latency, throughput, error rate, request rate, and time window.

CPU demand comes from two factors:

A service can reach the same CPU limit in different ways. Traffic may increase while CPU cost per request remains constant, or a code change may make each request more expensive while traffic remains constant. These lead to different long-term fixes.

For one request, elapsed time can be divided conceptually into:

From arrival to completion, a request's time is spent in these states:

Only states 2 and 5 consume CPU. A request can be slow because of any of the others, and CPU utilization will not show it.

Only the first, second, fourth, and fifth phases involve CPU demand or CPU scheduling. Time spent sleeping for a database response, storage operation, lock, or timer is off-CPU time. A slow request with little CPU time and little run-queue waiting is unlikely to be fixed by adding CPU capacity.

Before diagnosing a CPU bottleneck, look for two relationships:

High utilization without service impact may simply mean the machine is doing useful work efficiently.

Begin with the machine's CPU topology:

lscpu summarizes the CPUs visible to Linux. Its extended view maps logical CPU IDs to physical cores, sockets, and NUMA nodes when the platform exposes that information.

nproc reports processing units available to the current process. This can be smaller than the number installed or online because of affinity or other restrictions. It still does not express every kind of limit, particularly a time-based cgroup CPU quota.

A logical CPU is a scheduling unit, not necessarily a complete physical core. Simultaneous multithreading can expose two logical CPUs backed by one physical core. Two such CPUs do not always provide twice the throughput of one, because they share core resources.

The relevant denominator is the capacity available to the affected workload, not automatically the host's logical CPU count. Check the target process's allowed CPU list:

The query form of taskset displays affinity without changing it. Affinity is maintained per thread, so inspect individual thread IDs if they may have different masks.

A process allowed on CPUs 0-15 can still have a quota equivalent to one CPU. Conversely, a process restricted to CPUs 2-3 cannot use the other fourteen CPUs even if they remain idle. Capacity must account for both placement and quota.

Linux accounts CPU time in several categories. mpstat, usually provided by the sysstat package, reports those categories at intervals:

-P ALL shows an aggregate row and one row per logical CPU. The interval-and-count form above produces five reports one second apart; without an interval, mpstat reports activity since boot. Concentrate on rows that overlap the incident.

The most useful fields are:

Some versions also display guest categories. Guest time may already be included in user time, so do not blindly add every printed percentage.

The category mix guides the next question.

High %usr suggests that application or runtime code is consuming the CPU. This can be useful computation, an unexpectedly expensive algorithm, garbage collection, serialization, compression, encryption, or a busy loop.

High %sys means the kernel is doing substantial work. It may be handling frequent system calls, virtual-memory operations, networking, filesystem activity, or process and scheduling operations. The number identifies a broad execution context, not the exact kernel path.

High %irq or %soft points toward interrupt-related work. Network receive processing is a common source of softirq CPU time. This work may not appear as CPU charged directly to the application process that ultimately consumes the data.

High %steal in a virtual machine means the guest wanted CPU time that the hypervisor did not provide because the physical host was doing other work. From inside the guest, CPUs can appear busy or delayed even though no guest process owns that time.

%iowait needs special care. It is not CPU execution and is not the percentage of time every process spent waiting for I/O. It is an idle-CPU accounting category observed while block I/O is outstanding. High iowait suggests investigating I/O, while low iowait does not prove that no process waits for I/O. The accounting is also affected by task migration and platform behavior.

The aggregate CPU-time counters come from the first line of /proc/stat:

A representative line looks like:

The values are cumulative units of CPU time in fields such as user, nice, system, idle, iowait, interrupt, softirq, and steal. Their absolute totals since boot rarely diagnose a current incident.

Utilization over an interval requires two readings:

Do not add guest fields a second time when they are already included in user categories. In practice, established tools such as mpstat handle these accounting details and are safer than an improvised parser.

The important principle applies to every cumulative counter:

Subtract two observations taken at known times. Diagnose the interval, not the lifetime total.

An average across a long interval can still hide a short saturation burst. Choose a sampling period short enough to include the duration of the latency spike.

Utilization describes how much available CPU time was busy. Saturation describes runnable work that could not execute immediately because CPU service was unavailable.

A CPU can be highly utilized without a queue:

Add more runnable workers:

The utilization is similar, but the second case has scheduling delay.

vmstat provides a compact system-wide view:

Relevant fields include:

The first vmstat report commonly represents averages since boot. Subsequent rows represent the requested intervals.

Because r includes tasks currently executing as well as tasks waiting to execute, compare it with effective CPU capacity over several samples. An r value persistently larger than the available CPUs suggests run-queue waiting. A one-second spike does not necessarily indicate harmful saturation.

The b column is different. Those tasks are blocked rather than runnable, so adding CPU capacity does not directly make them execute. A high load average accompanied by many blocked tasks and idle CPU can therefore reflect I/O or another uninterruptible wait rather than CPU saturation.

Linux Pressure Stall Information, or PSI, reports time lost because tasks cannot obtain a resource.

Read system-wide CPU pressure with:

An illustrative some line is:

For CPU, some measures periods during which at least one runnable task was delayed waiting for CPU service. avg10, avg60, and avg300 summarize recent pressure, while total is cumulative stall time in microseconds.

CPU PSI answers a question that utilization alone cannot:

How much time are runnable tasks losing because they cannot get CPU service?

On a cgroup v2 system, a workload can have its own pressure file:

Replace PATH with the cgroup path associated with the service or container. Per-cgroup pressure is often more actionable on a shared host because it focuses on the workload rather than mixing every runnable task on the machine.

PSI still needs correlation. Background batch work may experience pressure without harming the API. Compare the pressure interval with the affected workload's latency and throughput.

If the pressure file does not exist, the kernel may not provide PSI in that environment. Missing telemetry must not be interpreted as zero pressure.

An aggregate CPU average can hide severe imbalance.

Suppose a host has eight logical CPUs and one single-threaded process fully occupies CPU 3 while every other CPU is idle:

The host has spare aggregate capacity, but the single thread cannot execute on several CPUs simultaneously. If that thread is on the request's critical path, CPU is still the bottleneck for that operation.

Use the per-CPU rows from:

Persistent imbalance can result from:

Remember that logical CPUs may share a physical core. If sibling hardware threads are both busy, they can compete for core execution resources even though Linux reports them as separate CPUs. lscpu -e=CPU,CORE,SOCKET reveals the topology needed to recognize such pairs.

After confirming CPU demand at the relevant scope, identify the process responsible.

For a live system-wide view, top sorts processes interactively by CPU usage. A bounded batch snapshot is:

pidstat provides interval-based process measurements:

This is usually more useful than a one-time ps listing because it reports activity during defined intervals. Output varies by version, but commonly separates user and system CPU time and identifies the CPU on which the task ran.

For a known process:

ps can still provide a quick ranking:

Its %CPU value is CPU time divided by process lifetime. A newly busy old process can rank lower than expected because earlier idle time remains in the average. Use interval tools to confirm current activity.

Also account for work that is not attributed neatly to an application process. High interrupt or softirq time may be distributed through interrupt context or kernel threads. Per-CPU category data, /proc/interrupts, and /proc/softirqs can reveal such work:

These files contain cumulative per-CPU counters. Compare deltas rather than selecting the largest lifetime number.

A multithreaded process can consume 400% CPU because four threads each keep one CPU busy. It can also show approximately 100% because only one thread is active while dozens sleep.

Inspect current CPU activity by thread:

Or use an interactive view:

Thread-level data distinguishes several patterns:

CPU percentage conventions vary by tool. In common Linux top configurations, one fully busy thread appears near 100%, so a multithreaded process can exceed 100%. Other views normalize total machine capacity to 100%. Always determine the denominator before comparing values from different tools.

For one thread, Linux exposes cumulative scheduler statistics:

The first fields report time executing on a CPU, time waiting runnable on a run queue, and the number of times the task ran. Read them twice and subtract to measure an interval. Run-queue waiting is particularly useful for separating “the thread needed CPU but could not get it” from “the thread slept waiting for something else.”

Once a consistently CPU-consuming thread is identified, a sampling profiler can determine which user or kernel stacks account for its on-CPU time. Profiling should be bounded and selected only after the broader evidence establishes that CPU execution is the relevant path.

CPU contention is sometimes accompanied by high context-switch rates, but a high rate is not a root cause by itself.

System-wide context switches appear in vmstat under cs. Per-process and per-thread switch rates can be measured with:

Voluntary context switches occur when a task blocks or yields. Involuntary switches commonly occur when the scheduler preempts it.

Many switches can be normal for an event-driven server handling a large number of short operations. They become relevant when the rate changes with the regression and a credible mechanism connects switching overhead, runnable waiting, or poor locality to the service impact.

Low context-switch activity also does not rule out a CPU problem. One compute-bound thread can occupy a CPU continuously with relatively few switches.

Containers and services often run inside cgroups. A cgroup CPU quota can constrain a workload even while the host has idle CPUs.

First locate the target process's cgroup:

On cgroup v2, the output commonly contains one unified path:

Use that path under /sys/fs/cgroup:

The actual path varies by service manager and container runtime.

cpu.max contains a quota and period in microseconds:

This grants 100,000 microseconds of CPU time per 100,000-microsecond period, equivalent to an average of one CPU. A value such as:

means no hard CPU-time quota is configured by this cgroup.

These cgroup v2 bandwidth controls apply to the fair scheduling class used by ordinary services; specialized real-time policies have different controls.

Quota can be consumed in parallel. Two runnable threads can each use a CPU and spend a one-CPU quota in roughly half of the wall-clock period. The entire cgroup may then be throttled until the next period even though other host CPUs remain idle. This creates periodic latency spikes that host-wide average CPU usage can hide.

cpu.stat exposes cumulative accounting such as:

The most useful evidence is the change during the incident:

A historical nonzero counter does not prove current throttling. Sample before and after a latency spike, and compare the deltas with request behavior.

cpu.weight is different from cpu.max. Weight influences proportional sharing when cgroups compete for CPU; it is not a fixed CPU-time ceiling. A low-weight group may run freely on an idle host and receive a smaller share only when contention appears.

On cgroup v1 systems, equivalent quota information is exposed through files such as cpu.cfs_quota_us, cpu.cfs_period_us, and cpu.stat. Do not mix v1 and v2 paths in the same investigation.

A workload can suffer CPU waiting because it is permitted to run on too few CPUs.

At process level:

At cgroup v2 level:

The effective set matters because a child cgroup cannot use CPUs excluded by its ancestors.

Inspect threads individually when affinity may differ:

If all request workers are pinned to one CPU, that CPU can develop a run queue while others remain idle. Changing affinity in production is a mutation, not a diagnostic read. First determine why the restriction exists; it may protect cache locality, isolate latency-sensitive work, or satisfy a real-time policy.

Quota and CPU sets constrain different dimensions:

A workload allowed on eight CPUs but limited to one CPU of quota can use several CPUs briefly and still average only one CPU of service.

Inside a virtual machine, guest-visible CPU capacity depends on host scheduling. Sustained %steal indicates that a runnable virtual CPU waited while the hypervisor used the physical CPU elsewhere. The owning guest process may appear ready to run but cannot control the missing host capacity.

CPU percentages also measure time, not work completed per cycle. Processor frequency changes with power management, thermal conditions, turbo behavior, and platform policy. Two intervals with the same 100% utilization can complete different amounts of work.

Investigate effective CPU speed when:

Hardware-counter tools and platform utilities can provide deeper evidence, but begin by comparing host model, topology, virtualization metrics, and workload CPU time. A code change that increases CPU seconds per request is different from a processor that completes fewer instructions per second.

Suppose an API container on a 16-CPU host normally has p99 latency of 120 milliseconds. After a deployment, p99 rises to 900 milliseconds in a repeating pattern. Request rate is unchanged, host-wide CPU averages 35%, and no individual CPU remains fully busy for an entire one-second sample.

The host average initially makes CPU saturation look unlikely. The investigation checks the service's effective boundary.

The API process belongs to:

Its cpu.max is:

The container is limited to an average of one CPU. It has four runnable request workers, so they can consume that quota in parallel before the period ends.

Two snapshots of cpu.stat, taken 60 seconds apart during the incident, show that nr_throttled and throttled_usec rise rapidly during the same interval as request p99. Host-level idle CPU remains available.

Thread-level pidstat shows that all four request workers perform user-space CPU work. Application measurements show that CPU time per request increased immediately after the deployment, while request rate stayed constant. The changed code serializes the same response data multiple times.

The causal finding is:

Temporarily restoring the previous deployment reduces CPU time per request, throttling deltas, and p99 latency together. That is stronger evidence than changing the quota alone because it connects the regression to the new computational cost.

A short-term alternative might raise the quota if spare host capacity and operational policy allow it. The long-term fix removes duplicate serialization and sets CPU requests and limits from measured CPU cost, traffic, and latency requirements.

The following sequence captures a broad CPU view without attaching a profiler or changing the workload:

After identifying a target PID:

If the target is cgroup-limited, capture cpu.max, cpu.stat, cpu.pressure, and cpuset.cpus.effective from its actual cgroup before and after a representative interval.

Keep the application symptom visible throughout the capture. A perfect CPU diagnosis still requires showing that CPU pressure and recovery align with the operation users care about.

CPU investigation separates activity from scarcity. Establish the workload's effective capacity, including logical CPUs, affinity, CPU sets, quotas, and virtualization constraints.

Use interval measurements to classify CPU time as user, system, interrupt, iowait, steal, or idle. Examine run-queue length and CPU pressure to determine whether runnable work is actually waiting, and inspect individual CPUs so aggregate averages do not hide a hot core.

Narrow CPU demand from the system to a process and then to individual threads. Compare execution time with run-queue waiting, and account for interrupt work that may not be charged directly to the application.

For cgroup workloads, changes in throttle counters can explain latency spikes even when the host is mostly idle, while per-cgroup CPU pressure can reveal runnable delay within that workload. A complete diagnosis connects workload or code changes to CPU demand, constrained service, runnable waiting, and the original latency or throughput symptom.