An API container disappears during a traffic spike. The last application log looks normal, but the platform reports:

It is tempting to summarize the incident as “the process used too much memory.” That leaves the important questions unanswered:

The Linux out-of-memory killer is a last-resort recovery mechanism. Before invoking it, the kernel normally tries to recover reusable memory. Those attempts can stall applications and degrade throughput long before anything is killed.

This chapter follows that progression from healthy memory use through reclaim, severe pressure, and an OOM kill. It then develops a practical workflow for diagnosing and preventing production incidents.

Memory usage describes how much of a memory resource is occupied or charged.

Memory pressure describes the work and delay caused by competing demand for memory that is difficult to supply.

A machine can have little completely unused RAM without being under serious pressure. Linux deliberately uses otherwise idle memory for file cache and other useful caches. If enough of those pages can be reclaimed cheaply, a new allocation can still proceed quickly.

Conversely, a system can experience pressure before every byte is occupied. The remaining memory may be unavailable to a particular cgroup, unsuitable for a constrained allocation, or expensive to reclaim.

The practical distinction is:

This is why a dashboard showing “95% memory used” is not enough to diagnose an incident. Operators also need to know what the memory contains, whether useful work is stalling, and which limit or allocation domain is constrained.

The simplified path looks like this:

This is intentionally simplified. Real allocation behavior depends on the requested size, kernel context, memory zone, NUMA policy, cgroup, reclaim options, and other constraints.

Two points matter immediately:

Some callers are allowed to fail without destructive recovery. Certain physically constrained requests can fail even though other memory exists. A strict commit policy or process resource limit can also reject an allocation directly.

The OOM killer is used when the kernel concludes that an eligible allocation domain cannot make sufficient progress through less destructive means.

When readily available pages run low, Linux attempts to reclaim memory whose contents can be discarded or preserved elsewhere.

The main categories behave differently.

Clean File-Backed Pages

Clean pages from executable files, shared libraries, and memory-mapped files already have an authoritative copy in storage. The kernel can remove such a page from RAM and read it again later if a process accesses it.

Reclaiming the page frees memory, but a later access can incur storage I/O and latency.

Dirty File-Backed Pages

A dirty page contains modifications not yet written to its backing file. Its contents must be written before the physical memory can be safely reused.

If storage is slow or writeback is already congested, reclaiming dirty pages can stall allocation paths.

Anonymous Memory

Heap and stack memory is commonly anonymous: there is no ordinary file from which its current contents can be reconstructed.

If swap is available and permitted, the kernel can move eligible anonymous contents to swap and reuse the physical page. Without usable swap, anonymous memory generally must remain resident until the owning process releases it or exits.

Reclaimable Kernel Caches

Some kernel caches contain filesystem metadata or other data that can be reconstructed. The kernel can shrink these caches under pressure.

Other kernel memory is not readily reclaimable. Pinned pages, active device buffers, page tables, socket memory, and subsystem allocations can all reduce the memory available for new work.

The broad lesson is:

Two systems with the same used-memory total can behave very differently because the contents have different reclaim costs.

Linux uses both background and synchronous reclaim.

As available memory passes internal watermarks, the kernel wakes background reclaim workers such as kswapd. These workers scan for pages that can be freed, written back, or moved to swap. If they restore enough headroom, application allocations continue with little visible disruption.

Under heavier pressure, a task requesting memory may have to participate in direct reclaim. Its allocation path pauses while the kernel searches for memory.

From the application's perspective, an ordinary operation suddenly takes much longer:

No OOM kill is required for this to damage service quality. Direct reclaim can produce:

If reclaimed pages are needed again immediately, the system repeatedly evicts and reloads useful data. This state is often called thrashing. The machine remains busy, but little useful application work completes.

Memory pressure is therefore a performance incident before it becomes an availability incident.

Swap provides backing storage for eligible anonymous memory. It gives the kernel another way to reclaim physical RAM and can absorb short-lived peaks.

Swap does not make memory demand free. Moving pages to and from storage is much slower than accessing RAM. If an application's active working data exceeds available physical memory for a sustained period, additional swap can turn an immediate kill into prolonged latency and thrashing.

With no swap:

With swap:

The right swap policy depends on workload latency requirements and failure strategy. Its behavior should be measured under load rather than inferred from the configured swap size alone.

Containers can also have their own swap accounting or limit. Host swap capacity does not guarantee that a constrained cgroup is allowed to use it.

A successful allocation reserves a usable address range for the process. Depending on the allocation and system policy, the physical resources needed for every byte may not be committed to RAM at that moment.

Linux supports memory overcommit because many programs reserve more address space than they actually touch. The policy is controlled by:

The common modes are:

In strict mode, commitments are limited using configured RAM and swap policy. The relevant system-wide fields are:

CommitLimit is the limit used by strict overcommit accounting. Committed_AS is the amount of memory the system has promised under that accounting model, including allocations that may not yet be resident.

These are commitment metrics, not current physical-memory usage. Their ratio is particularly meaningful in strict mode and should not be mistaken for RSS.

With permissive overcommit, this sequence is possible:

An allocation can also fail normally and return NULL. Strict overcommit, an address-space limit, an allocation constraint, or a caller that forbids aggressive recovery can produce a failure without an OOM kill.

Applications must still check allocation results. The existence of the OOM killer is not an error-handling strategy.

“The machine ran out of memory” is only one kind of OOM event. Linux chooses victims from the scope in which the failure occurred.

System-Wide OOM

A global OOM occurs when the system cannot satisfy an eligible allocation and cannot reclaim enough memory from the system-wide resources available to that allocation.

The kernel can select an eligible victim from the broader system. This is dangerous for a shared host because an unrelated service may be sacrificed to keep the machine operating.

Global OOM evidence normally appears in kernel logs and includes information about the allocation context, memory state, candidate processes, and selected victim. The exact log format varies by kernel version.

Cgroup OOM

A memory-cgroup OOM occurs when a cgroup reaches its hard memory boundary and cannot reduce its charged usage enough to satisfy a new charge.

The host can still have abundant memory:

In this case, the OOM killer is confined to the cgroup. It does not select a victim from an unrelated cgroup just because that process has a higher global memory footprint.

This isolation is one reason container memory limits protect the rest of a host. It also explains why checking only host-level free output can completely miss the cause of a container termination.

Other Constrained Scopes

An allocation can be constrained by eligible memory nodes, CPU-set memory placement, or a process memory policy. The kernel's OOM decision considers the memory allowed in that context.

For production diagnosis, the first question should be:

Which allocation domain was exhausted?

The answer determines which processes were eligible victims and which memory totals matter.

Cgroup v2 exposes several controls with different purposes:

memory.high is valuable as an early control point. Processes exceeding it are throttled and placed under heavy reclaim pressure. A management agent can observe the event and reduce load, raise the boundary, or terminate work deliberately.

memory.max is the hard backstop. A cgroup may exceed it briefly in some circumstances, but if charged usage cannot be reduced, the kernel can invoke OOM handling within that cgroup.

By default, a cgroup OOM may kill one selected process. That can leave a multi-process service partially alive but internally broken. Setting:

asks the kernel to treat the cgroup and its descendants as one indivisible workload for OOM killing. Tasks explicitly protected with oom_score_adj = -1000 remain exceptions.

Group killing is a workload-integrity choice, not a universal default. A batch cgroup containing independent jobs may prefer individual victims, while a tightly coupled database process group may be unusable after any one member disappears.

The goal is not to punish the process that made the final allocation request. The goal is to free enough memory while preserving overall system operation.

Linux assigns eligible tasks an OOM “badness” score. Memory use relative to the allowed allocation scope is central to the score, and user-space policy can adjust it.

Inspect a process with:

oom_score is the kernel's current exported score for that process. A higher value generally means the process is a more likely victim within the relevant OOM scope.

oom_score_adj is a policy adjustment from -1000 to 1000:

For example, an operator could make a disposable batch worker easier to sacrifice:

Changing protection in the opposite direction may require additional privilege. Production services should normally configure this policy through their service manager or container platform rather than modifying live processes by hand.

The score is not a stable ranking that can be recorded once. It changes with memory use, allowed scope, process lifetime, and policy. A process with the largest RSS is not guaranteed to have the highest relevant score.

The Allocating Task Is Not Necessarily the Victim

Suppose process A makes the allocation that finally exposes exhaustion, while process B owns much more killable memory. The kernel may kill B.

Therefore:

The root cause might be an unbounded queue spread across many workers, an oversized cache, a cgroup limit set below normal demand, or another process that consumed the available headroom.

Do Not Protect Everything

Setting critical processes to -1000 sounds safe, but protected memory is still consumed. If every large process is protected, the kernel has fewer useful victims and may be unable to recover cleanly.

OOM adjustments should express an intentional sacrifice order:

Protection does not create memory. It transfers risk to other processes.

The kernel terminates a selected user-space victim with SIGKILL. The process cannot catch or ignore this signal, so it has no opportunity to flush application buffers, run shutdown hooks, or write a final diagnostic message.

That explains the abrupt symptom:

Memory release may not be instantaneous. The victim still has to exit and its resources must be dismantled. The kernel can accelerate recovery, but tasks stuck in certain kernel operations may delay complete cleanup.

If the killed process is a worker rather than the service's supervisor, the remaining service may continue in a degraded or inconsistent state. A supervisor may restart the worker, immediately recreate the same memory demand, and enter a crash loop.

An OOM kill also differs from an application-level out-of-memory error:

The symptom alone does not prove which path occurred.

Shells and container runtimes commonly represent signal termination as:

For SIGKILL, signal number 9:

An OOM kill can therefore produce exit code 137. So can:

Treat 137 as a reason to investigate SIGKILL. Confirm OOM using cgroup event counters, platform state, or kernel logs.

Similarly, a shell printing:

means the process received a killing signal. It does not identify who sent the signal or why.

Start with a compact view:

Do not focus only on the free column. Linux uses spare RAM for caches. The available estimate is more useful because it accounts for memory the kernel expects it can make available without relying on swap.

Inspect selected /proc/meminfo fields:

These fields help separate:

Watch activity over time with:

Pay attention to sustained swap-in and swap-out activity, blocked work, I/O wait, and a falling available-memory trend. One sample is rarely enough; pressure is a time-dependent behavior.

High swap usage alone does not prove current pressure. Cold pages can remain in swap even after the pressure episode ends. Active swap traffic and application stalls are stronger evidence.

Linux Pressure Stall Information, or PSI, measures time lost because tasks cannot make progress due to resource contention.

Read system-wide memory pressure with:

Example:

The some line measures periods when at least some tasks are stalled on memory.

The full line measures periods when all non-idle tasks in the measured scope are simultaneously stalled on memory. Sustained full pressure means the workload is spending time without productive progress and may be thrashing.

avg10, avg60, and avg300 are recent percentages over 10-, 60-, and 300-second windows. total is cumulative stall time in microseconds since the counter began.

PSI answers a question that memory-use percentages cannot:

How much execution time is memory scarcity taking away from the workload?

A service can be unhealthy with moderate RSS if it repeatedly waits on reclaim. Another can have high RSS and near-zero memory PSI because its working data fits and reclaim is not interfering.

On cgroup v2, each workload cgroup can expose:

Per-cgroup PSI is usually more actionable for a shared host because it identifies the workload experiencing the stalls.

For a cgroup v2 workload, define its actual cgroup directory and inspect:

memory.current is current charged usage. memory.peak is the recorded peak. The total can include anonymous memory, file cache, kernel memory, page tables, socket buffers, and other categories charged to the cgroup; it is not simply one process's RSS.

memory.events contains cumulative counters such as:

Interpret them carefully:

An oom event does not imply that a process was killed. Compare it with oom_kill. Because the counters are cumulative, monitoring should record changes rather than alert forever on a nonzero historical value.

memory.events includes events from descendant cgroups. Use:

when only events occurring directly at that cgroup level are required.

The cgroup path shown to a process depends on how the system, service manager, or container runtime organizes the hierarchy. Inside a container, /sys/fs/cgroup may already represent the container's delegated root.

On a Linux host using the system journal:

On systems where the kernel ring buffer is directly accessible:

An OOM record can reveal:

Log formats and available fields differ across kernel versions. Read the whole event rather than matching only the phrase Killed process.

Containers commonly cannot read host kernel logs. Managed platforms may expose the event through node logs, workload status, or a monitoring service instead. If logs have already rotated, persistent cgroup counters and platform termination state may be the remaining evidence.

Never run an unlimited memory-pressure program directly on a development machine. The experiment must be placed inside a verified memory limit so the OOM decision remains local to the disposable workload.

The following Python program allocates 8 MiB chunks, explicitly writes through each chunk, and reports its charged progress:

Save it as memory_pressure_demo.py.

The following Docker workflow creates a container with a 128 MiB memory limit. Setting --memory-swap to the same value prevents the container from gaining additional capacity from swap:

Before starting it, verify that both limits are nonzero and equal:

Expected values for 128 MiB are:

Start and attach to the container:

In another terminal, observe its memory use while it runs:

The program should print increasing totals and then stop abruptly when the cgroup cannot satisfy another charge.

Inspect the final state:

A typical result is:

On a native Linux Docker host, inspect the kernel logs using the commands from the previous section. Docker Desktop runs containers inside a Linux virtual machine, so the desktop host's kernel log may not contain the container's OOM record.

Remove the stopped demonstration container:

The precise last allocation reported will be below 128 MiB. The Python interpreter, loaded libraries, allocator metadata, page tables, and other charged memory also consume part of the cgroup budget.

This experiment demonstrates three distinct facts:

If the platform reports that memory limits are unsupported or the inspection output shows a zero limit, do not run the workload. Use a Linux environment with working cgroup memory enforcement.

When a service disappears or becomes extremely slow, investigate in a consistent order.

1. Confirm the Termination Mechanism

Look for:

Do not declare OOM from exit code 137 alone.

2. Identify the Exhausted Scope

Determine whether pressure occurred at:

Compare the limit and usage from that same scope. Host MemAvailable does not disprove a cgroup OOM.

3. Reconstruct the Timeline

Align:

Pressure often begins before the kill. The first rise in PSI or memory.events:high can be more informative than the final termination time.

4. Break Down the Memory

Inspect:

Then connect the dominant category to process and application data:

These arrows are investigation directions, not proof. Confirm with process mappings, allocator data, and workload behavior.

5. Separate Victim from Cause

The kernel log names the victim. It does not automatically explain why total demand exceeded the limit.

Ask what grew:

A restart can erase the evidence. Persist cgroup counters, application metrics, and periodic process breakdowns outside the failing workload.

6. Reproduce Under a Deliberate Limit

Use a disposable environment and a representative workload. Observe when pressure begins, not only when the kill occurs.

The useful threshold is the point at which latency and PSI become unacceptable. A hard OOM boundary is too late to serve as the primary operating target.

OOM prevention is primarily capacity and overload design.

Keep Bounded State

Queues, caches, request bodies, retry buffers, batches, and per-connection state need explicit limits. A service with bounded concurrency but an unbounded input queue is still unbounded in memory.

When a bound is reached, choose a deliberate policy such as rejecting work, applying backpressure, shedding optional features, or spilling appropriate data to durable storage.

Budget the Whole Process

A managed heap limit is only one line in the budget:

Set the container limit above normal steady-state demand and measured bursts. Setting a 1 GiB cgroup limit around a runtime configured to grow its heap to 1 GiB leaves no room for anything else.

Use a Warning Boundary Before the Hard Limit

Where supported, memory.high can create an observable throttling boundary below memory.max. PSI and memory.events:high then provide a chance to reduce demand before hard-limit OOM.

The warning boundary must leave enough headroom for in-flight requests, monitoring, cleanup, and control-plane activity. A signal arriving one allocation before memory.max is not an early warning.

Make Sacrifice Policy Explicit

If several workloads share a host, decide which can be restarted and which must be protected. Use OOM score adjustments sparingly and test the resulting behavior.

For a tightly coupled multi-process service, decide whether partial survival is useful. If not, cgroup group-kill policy can make failure atomic so a supervisor restarts the complete workload.

Capture Evidence Before Failure

Because SIGKILL cannot run cleanup hooks, an OOM victim cannot reliably create a heap dump at the moment it is killed.

Collect continuous low-cost metrics and trigger deeper diagnostics before the hard limit:

Good observability turns an abrupt kill into the final point of an already visible trend.

Memory pressure begins when competing demand makes memory expensive to supply, not simply when a used-memory percentage becomes high. Linux first tries to reclaim clean file-backed pages, write back dirty data, move eligible anonymous memory to swap, and shrink suitable kernel caches. Direct reclaim stalls application allocations and can damage latency long before an OOM kill.

If recovery cannot satisfy an eligible allocation, the kernel may invoke the OOM killer within the exhausted scope. A global OOM can select from the broader system, while a cgroup OOM selects only from the constrained cgroup. memory.high creates throttling and reclaim pressure; memory.max is the hard cgroup boundary.

Victim selection uses an OOM badness score influenced by memory use, the allowed scope, and oom_score_adj. The process making the final allocation is not necessarily the victim, and the victim is not necessarily the root cause.

Exit code 137 indicates SIGKILL, not OOM by itself. Confirm an incident through kernel logs, platform state, and cgroup memory.events. Use PSI to measure time lost to memory stalls and correlate it with application latency, swap activity, cgroup usage, and memory-category growth.

Reliable services preserve headroom, bound queues and caches, budget native and kernel-charged memory in addition to the managed heap, and react before the hard limit. The OOM killer is an emergency mechanism for restoring system progress, not a substitute for resource control.