A host has idle CPU cores and gigabytes of available memory, yet one container develops severe latency and another is killed.

This is not contradictory. A container shares the host's kernel, but its processes can run inside a much smaller resource boundary. The kernel enforces that boundary even when resources exist elsewhere on the machine.

Different boundaries also fail in different ways. A CPU limit usually delays work. A memory limit can end a process with SIGKILL. A PID limit makes process or thread creation fail. A full writable filesystem makes writes fail with ENOSPC.

Host headroom does not imply container headroom. Diagnose the resource scope that contains the failing workload.

“The container ran out of resources” is not a complete diagnosis. Start by classifying what actually happened.

The workload becomes slow. CPU quota exhaustion, I/O throttling, or reclaim above memory.high can suspend or delay otherwise runnable work. The process stays alive, but latency rises and throughput falls.

An operation fails. Reaching pids.max can make fork() or thread creation return an error. A full filesystem can make a write fail. A file-descriptor limit can make open() or accept() fail. Existing work may continue while one class of operation stops succeeding.

A process is killed. If a cgroup cannot reclaim enough memory below memory.max, the kernel can select a process in that cgroup and send SIGKILL. There is no cleanup handler for the victim to run.

Only part of the workload fails. The killed process might be one worker rather than PID 1. The container can remain “running” while the service is degraded or internally broken.

The symptom narrows the search. A process that is alive but periodically frozen suggests a different mechanism from one that disappeared without a shutdown log.

Container configuration is translated into kernel mechanisms. For CPU, memory, I/O, and task counts, that usually means cgroups.

These examples use cgroup v2. Confirm the mounted version:

cgroup2fs identifies a cgroup v2 mount. Inside a cgroup namespace, /proc/self/cgroup may report:

In that case, /sys/fs/cgroup commonly represents the container's cgroup root, and controller files can be read directly from it:

The mount is often read-only inside a container. Read access is enough for diagnosis; policy changes should go through the runtime or orchestrator.

Cgroup limits are hierarchical. A container can be below the values visible at its own level and still be constrained by a tighter ancestor shared with other workloads. A cgroup namespace can also hide the host-side path and ancestors. When local files do not explain the behavior, inspect the container through platform telemetry or resolve its cgroup from the host.

Configuration describes what should happen. Counters describe what the kernel actually enforced. Reliable diagnosis needs both.

CPU quota creates a maximum amount of CPU time that a cgroup can consume during an enforcement period.

For cgroup v2:

This grants 50 milliseconds of aggregate CPU time per 100-millisecond period, equivalent to half of one CPU's sustained capacity.

The word aggregate matters. If four worker threads run on four cores, they can consume that 50-millisecond budget in roughly 12.5 milliseconds of wall-clock time. The kernel then throttles the cgroup until the next period.

Runnable and throttled at the same time is the state that confuses monitoring. The threads are ready and the scheduler will not run them.

The host can have idle CPUs throughout the throttled interval. The unavailable resource is not a physical core; it is permission for this cgroup to consume more CPU time during the current period.

This creates a recognizable failure pattern:

A CPU limit is therefore a fail-slow boundary, not normally a process-kill boundary.

Read the policy and counters from the workload's actual cgroup:

An example cpu.stat might contain:

The counters are cumulative. A nonzero nr_throttled proves that throttling occurred sometime during the cgroup's lifetime, not necessarily during the current incident.

Take two snapshots across a representative interval:

An increase in nr_throttled shows that the cgroup exhausted its quota during that interval. An increase in throttled_usec quantifies accumulated throttled time. Correlate those deltas with request latency and throughput; throttling without user-visible impact may not require action.

Do not confuse quota throttling with ordinary CPU contention. cpu.weight changes how competing cgroups share CPU, but it does not create a fixed ceiling. A low-weight workload can run freely when the host is idle and lose CPU when siblings become busy, without quota-throttling counters increasing.

CPU placement can also be the constraint. cpuset.cpus.effective may expose fewer CPUs than the application assumes:

The application must size worker pools using effective container capacity, not merely the number of processors visible on the host.

CPU percentages need a denominator. A monitoring system might report usage relative to one CPU, the container quota, all host CPUs, or a requested amount. The same workload can therefore display very different percentages in different tools.

Consider a container limited to half a CPU on a 32-CPU host. Fully consuming its quota is only about 1.6% of total host capacity. A host dashboard can look almost idle while the container is continuously constrained.

Burst timing can hide the problem further. A request fan-out may consume the quota early in each period, then wait. An average over one minute smooths the run-wait cycle into an innocent-looking number, while p99 latency records every forced pause.

The important comparison is not simply “CPU usage is below 100%.” It is:

Did the workload need CPU while cgroup policy made it ineligible to run?

Throttle-counter deltas provide direct evidence for that question.

Raising or removing the quota may be correct when the workload has legitimate demand and the host or cluster has capacity. It is not the only response.

A service can also reduce unnecessary work, bound request concurrency, eliminate busy polling, or resize thread pools. For latency-sensitive services, a very small CPU quota combined with many runnable workers often produces poor tail behavior even if average throughput appears acceptable.

Adding replicas helps only when traffic can be distributed and the underlying placement has capacity. More replicas on the same saturated or tightly constrained node can move the contention rather than remove it.

Also distinguish a limit from a reservation. A quota caps how much CPU a workload may consume; it does not guarantee that the workload will receive that amount under all host conditions. Placement policy, CPU weight, and available node capacity determine whether the promised service level is realistic.

Container memory failure is not a single jump from “healthy” to “OOM-killed.”

The cgroup memory charge can include anonymous application memory, file cache, socket buffers, page tables, shared memory, and kernel allocations made on the workload's behalf. An application heap metric therefore does not describe the whole boundary.

Key cgroup v2 files include:

memory.high is a reclaim and throttling boundary. Crossing it can force allocation paths to spend time reclaiming memory, producing latency before any process is killed.

memory.max is the hard boundary. When a new charge cannot be satisfied while keeping usage within the limit, and reclaim cannot recover enough memory, the cgroup can enter an out-of-memory condition.

The host may still have abundant free memory. The kernel is enforcing the container's local allocation domain, not offering it unbounded access to every byte in the machine.

memory.events exposes cumulative counters:

The fields describe different stages:

An oom event does not guarantee that a process was killed. The kernel may recover without selecting a victim. A contemporaneous increase in oom_kill is much stronger evidence that cgroup OOM handling killed a process.

As with CPU counters, use deltas. A counter left at 1 after last week's incident does not explain today's restart.

Memory composition helps explain the charge:

These fields are not all independent totals; some are subcategories. Use them to identify whether growth is mainly anonymous memory, cache, sockets, or kernel-owned data rather than adding every line.

By default, a memory-cgroup OOM can select one eligible process as the victim. That process is not necessarily PID 1.

Suppose a server uses a supervisor and four workers. If one worker is killed, the container's main process remains alive. A basic “container is running” check can stay green even though capacity has fallen or the supervisor is repeatedly recreating the worker and triggering another OOM.

For tightly coupled multi-process workloads, cgroup v2 offers memory.oom.group so policy can request that the workload be treated as an indivisible unit during cgroup OOM handling. That is an integrity decision, not a universal default. Independent jobs sharing a cgroup may prefer individual victim selection.

Health checks should verify that the service can perform useful work, not merely that PID 1 exists.

Shells and container runtimes commonly encode signal termination as 128 + signal number. SIGKILL is signal 9, so a process killed by it commonly appears with status 137:

The kernel OOM killer uses SIGKILL, but so do administrators, supervisors, and container runtimes after a shutdown timeout.

Treat exit 137 as evidence of SIGKILL, then establish the cause. Useful independent evidence includes:

With Docker, runtime state can be inspected using:

An application-level out-of-memory exception is a different event. A managed runtime can exhaust its configured heap and log an error without the kernel killing it. Conversely, a cgroup OOM kill can end the process without giving the runtime any chance to emit a final message.

Setting an application heap equal to memory.max leaves no room for everything outside that heap.

A realistic budget must include native runtime structures, thread stacks, resident executable and library pages, page tables, socket buffers, file cache, helper processes charged to the same cgroup, and temporary bursts during operations such as compaction or request fan-out.

The correct response to OOM is not always “raise the limit.” First decide whether normal demand is underprovisioned or usage is unbounded.

For normal demand, choose a limit above the measured working set and credible bursts. Configure the language runtime using effective container memory, not host memory. Leave explicit non-heap headroom.

For unbounded growth, increasing the limit only postpones failure. Bound queues and caches, cap concurrency, investigate leaks, and reduce duplication. If memory.high is available, it can provide an observable pressure boundary before the hard limit, but sustained reclaim there is itself a latency problem.

Swap can add headroom in configurations that allow it, but it trades abrupt failure for possible latency. A service whose active working set is repeatedly swapped may remain alive while becoming unusably slow.

The cgroup PID controller limits kernel tasks, not just user-visible processes. Every thread counts toward pids.current.

Inspect:

When pids.current reaches a finite pids.max, the kernel rejects additional task creation. fork(), clone(), or a language runtime's thread-creation operation commonly fails with EAGAIN, often rendered as “Resource temporarily unavailable.”

Existing tasks are not killed merely because the boundary was reached. This produces partial failures:

pids.events includes a max counter for attempts rejected by the limit. Compare its change with the application's errors.

If an interactive command cannot be started inside the container, collect cgroup counters from the host or platform. The failed diagnostic command may be evidence of the same PID exhaustion, not a separate tooling problem.

Bound worker pools and subprocess creation, remove thread leaks, and leave operational headroom for lifecycle hooks and diagnostics. Raising pids.max without understanding growth can turn a contained failure into broader host task exhaustion.

The cgroup I/O controller can cap bandwidth or operations per second for a backing block device. When a workload reaches io.max, I/O is normally queued and delayed rather than rejected with a “quota exceeded” error.

Inspect the configured limits and activity:

io.stat reports device-level byte and operation counts. io.pressure, when available, reports time lost because tasks are stalled on I/O.

Buffered writes complicate timing. An application can copy data into the page cache quickly and encounter the delay later when dirty data is written back or when a durability operation such as fsync() waits. Correlate cgroup evidence with the exact application operation that became slow.

An I/O cap is distinct from device saturation. A container can be throttled below the storage device's physical capacity while the device itself looks lightly used. Conversely, slow I/O with no cgroup cap may come from contention, a network filesystem, writeback, or the underlying storage system.

The I/O controller limits rate; it does not limit how many filesystem blocks or inodes a container may own.

A container's writable layer, volume, or temporary filesystem can run out of capacity independently. Writes may then fail with errors such as:

Start with the exact path the application is writing:

df -h checks byte capacity. df -i checks inode capacity. Millions of tiny files can exhaust inodes while considerable byte space remains.

The mount identity matters. /var/lib/api might be in the ephemeral writable layer, a persistent volume, a bind mount, or tmpfs. Each has a different capacity source and lifecycle. An orchestrator may also evict a container after platform-level ephemeral-storage usage crosses a policy threshold; that is an external lifecycle action rather than an io.max throttle.

Deleting arbitrary files during an incident is risky. Identify ownership and retention policy first. Durable fixes include bounding temporary data, rotating logs, using an appropriately sized volume, and preventing deleted-but-open files from continuing to consume backing storage.

Containers can also inherit per-process resource limits. A common example is the open-file descriptor limit.

Inspect PID 1's effective limits:

If the process reaches RLIMIT_NOFILE, operations such as open(), socket(), or accept() can fail with EMFILE, usually reported as “Too many open files.” The container can have free memory, available CPU, and unused PID capacity at the same time.

Count PID 1's currently open descriptors:

This is one process's count, not necessarily the total for every process in the container.

Shared-memory mounts provide another distinct boundary. /dev/shm is commonly a size-limited tmpfs; filling it can produce ENOSPC even when the container has not reached a disk quota. Because tmpfs consumes memory, its pages can also contribute to the cgroup memory charge.

The error and kernel interface identify the mechanism. Do not label every container resource incident “a cgroup problem.”

Resource counters often belong to the container's cgroup. When the container exits and its cgroup is removed, those files disappear. A restarted container receives new processes and usually a new cgroup lifecycle, so its counters do not reconstruct the previous failure.

Monitoring should record at least:

Application metrics complete the picture: request latency, queue depth, active workers, heap use, cache size, and file-descriptor counts explain the effect of kernel enforcement.

Sampling must be frequent enough to retain short bursts. A one-minute average can miss a CPU budget exhausted every 100 milliseconds or a memory spike that ends the container in seconds.

When a container becomes slow, unhealthy, or restarts, use a bounded sequence.

First, record the exact symptom and time window. Distinguish delay, operation failure, process death, and platform eviction.

Second, identify the affected container, its main process, exit status, restart count, and current cgroup. Do not assume a new container instance has the same evidence as the one that failed.

Third, read the configured CPU, memory, PID, and I/O boundaries. Include ancestor or platform policy when the local cgroup view is incomplete.

Fourth, capture counters at two times and calculate changes. Cumulative totals without a time interval are historical facts, not incident correlation.

Fifth, match kernel enforcement to application impact. CPU throttling should line up with latency or throughput loss. An oom_kill increment should line up with a process disappearance. A pids.events:max increment should line up with failed creation calls.

Finally, choose a mitigation that addresses the mechanism. More memory does not fix CPU throttling; more CPU does not fix an inode leak; restarting may clear symptoms while destroying the most useful evidence.

An API has a CPU limit equivalent to half a CPU. It uses eight worker threads. During a traffic burst, p99 latency jumps from 40 milliseconds to roughly 200 milliseconds, while the 32-CPU host remains more than 80% idle.

Two cpu.stat snapshots taken over 30 seconds show:

The cgroup was throttled during 275 of 300 enforcement periods in the observation window. The throttle deltas rise during the same interval as p99 latency, while host CPU remains available.

The evidence supports a local quota mechanism. The eight workers consume a small aggregate budget quickly, then wait for replenishment. Host idle percentage does not disprove the diagnosis because host capacity lies outside the container's permitted budget.

The team tests a higher quota and a smaller, bounded worker pool. Throttle time and p99 latency both fall without changing memory or storage. That result verifies the mechanism rather than merely masking the incident with a restart.