A Linux host can report very little completely free memory and still be healthy. Another host can report gigabytes of available memory while one container repeatedly dies from an out-of-memory condition.

Both observations are possible because “memory usage” is not one number. Linux uses RAM for application pages, shared libraries, the page cache, kernel objects, page tables, socket buffers, and many other purposes. Some memory is easy to reclaim, some is expensive to reclaim, and some belongs to a limit smaller than the host.

A useful memory investigation asks:

Memory usage describes occupancy. Memory pressure describes the work and delay required to obtain memory. Investigate both.

Memory incidents usually begin with one of four external symptoms.

Latency or throughput degrades. The workload may be stalling in direct reclaim, faulting pages back into memory, or spending CPU on garbage collection. The relevant evidence is time lost and activity during the slowdown.

A process or container is killed. Determine whether the kernel OOM killer acted, which memory scope was exhausted, and which process it selected. An abrupt exit alone is insufficient.

Memory usage grows over time. Identify whether anonymous memory, file cache, shared memory, kernel memory, or mappings are growing. Growth may be an intentional cache, an unbounded queue, allocator retention, or a leak.

An allocation fails. The cause can be a process limit, cgroup limit, virtual address-space constraint, commit policy, unavailable huge pages, local NUMA policy, or physical memory pressure. Host-wide free RAM does not rule out narrower constraints.

Record the time window, affected workload, load level, recent changes, and the metric that triggered concern. “Memory is at 90%” is not a complete symptom until its scope, formula, and operational effect are known.

Memory can be measured at several boundaries:

Each level has its own memory accounting, and the numbers do not add up between them. "Out of memory" means something different at every level, so the first question is which one is actually exhausted.

These totals do not have to match.

A runtime heap metric excludes native allocations, thread stacks, mapped files, shared libraries, page tables, and much of the kernel memory charged on the process's behalf. Process RSS excludes some cgroup-charged kernel memory and double-counts shared pages when RSS values are added across processes. A container limit can be far below host capacity.

Start at the boundary where failure occurred. For a host alert, examine host memory and then find contributors. For a container kill, begin with the container's cgroup accounting. For one growing process, begin with process residency and mapping categories.

The process's cgroup can be found through:

On cgroup v2, the returned unified path can be resolved under /sys/fs/cgroup. Inside a container, the cgroup filesystem may already present the container's delegated root rather than the host's complete hierarchy.

Begin with:

A typical display separates total, used, free, shared, buffer/cache, and available memory. The most commonly misunderstood values are free and available.

Free memory is RAM not currently holding useful content. Linux deliberately uses otherwise idle RAM for caches, so a low free value is normal on a busy machine.

Available memory estimates how much memory can be provided to new applications without swapping. It considers free pages, reclaimable page cache, and the fact that not every apparently reclaimable object can actually be discarded.

This makes available a better initial headroom indicator than free, but it is still an estimate. It does not reveal a container limit, local NUMA constraint, allocation order, or latency caused by reclaim.

Inspect the source fields directly:

These fields help divide memory into operational categories:

Do not add every field. Some are subsets of others. Use them to explain a total, not to construct a new total by guesswork.

A machine with high usage and low pressure can be healthy. A machine with reclaim stalls can be unhealthy before it runs out of memory.

Watch activity with:

The first row commonly contains averages since boot; later rows describe the requested intervals. Relevant fields include:

One low free sample is weak evidence. Sustained swap traffic, blocked work, falling MemAvailable, and degraded service behavior during the same interval form a much stronger case.

Linux also exposes cumulative reclaim counters in /proc/vmstat:

Counter names vary somewhat by kernel configuration and version. The pgscan families count pages examined by reclaim, while pgsteal families count pages reclaimed. pswpin and pswpout count swap activity. allocstall families indicate allocation paths that stalled for direct reclaim.

These are lifetime counters. Read them at the beginning and end of a known interval. Rapid scanning with relatively little reclaimed memory suggests the kernel is working hard to find eligible pages.

Pressure Stall Information, or PSI, measures time tasks lose because of resource scarcity.

Read system-wide memory PSI with:

An illustrative result is:

For memory:

Sustained full pressure means the scope is spending intervals without productive work and may be thrashing. The avg10, avg60, and avg300 values show recent trends; total is a cumulative microsecond counter that can be differenced over custom intervals.

On cgroup v2, inspect the workload rather than only the host:

A host can show negligible memory PSI while one tightly limited cgroup shows severe pressure. If a pressure file is unavailable, record the missing capability rather than interpreting it as zero.

PSI measures impact but does not identify the pages being reclaimed or the code retaining memory. Combine it with memory composition and owner-level evidence.

Check swap capacity with:

The amount of swap currently used is historical state. Cold anonymous pages can remain in swap after a pressure episode ends, so nonzero swap use does not prove current thrashing.

Current swap traffic is more informative:

Sustained nonzero si means pages are being read from swap, while so means pages are being written to swap. Correlate the rates with major faults, storage activity, memory PSI, and application latency.

Swap changes the choices available to reclaim. It can preserve anonymous pages that would otherwise remain in RAM or contribute to an OOM condition. It also introduces storage latency when those pages are needed again.

No swap does not guarantee good latency, and adding swap does not fix an oversized working set. A workload that repeatedly swaps its actively used pages can spend more time moving data than performing useful work.

For a host-level incident, start with a process ranking:

RSS is current resident memory associated with the process. VSZ is mapped virtual address space, including ranges that may not be resident or even physically allocated.

A large VSZ can result from reserved address space, sparse mappings, runtime heaps, or mapped files without consuming equivalent RAM. A large RSS proves residency, but not exclusive ownership or active use.

Do not sum RSS across processes to estimate host or service memory. Shared libraries, shared memory, and shared file-backed pages can appear in the RSS of every process that maps them.

For current rates and page faults:

For one known process:

Common fields include VSZ, RSS, minor faults per second, and major faults per second.

A minor fault does not require loading the page from storage. It may establish a mapping to a page already available in memory, allocate a zero-filled page, or handle copy-on-write.

A major fault requires loading a page from storage. Major faults can be normal during startup or a cold-cache scan. A sustained increase during steady-state traffic, especially with latency and reclaim pressure, can show that the working set no longer remains resident.

Page-fault rate is workload-dependent. Compare faults per unit of work and against a healthy baseline rather than applying one universal threshold.

Use /proc/PID/status for a fast, low-cost overview:

The fields distinguish:

The RSS values in status are optimized for inexpensive reporting and can be approximate. Use them for trends and triage, not exact shared-memory attribution.

For process-wide proportional accounting:

smaps_rollup aggregates information across the process's mappings:

Some fields depend on kernel version and configuration. Access can also be restricted.

When one category grows, inspect mappings:

Or read /proc/PID/smaps for per-mapping detail. This can distinguish a growing anonymous region from a mapped data file or shared-memory object. Reading every mapping and page-table statistic is more expensive than reading status, so do it for selected processes rather than polling an entire host frequently.

An upward RSS graph does not prove a leak.

A service may intentionally warm a cache, load indexes, increase a worker pool, or retain allocator arenas for reuse. File-backed pages may enter the process RSS as data is read. A garbage-collected runtime may grow its heap toward a configured target before collecting.

Even after the application frees objects, RSS may remain high because the allocator keeps free chunks in its arenas, fragmentation prevents returning complete ranges, or the runtime retains committed pages for future allocations.

A stronger leak investigation uses a controlled timeline:

If private anonymous PSS grows without a stable bound, inspect runtime heap metrics and native allocations. If file PSS grows, identify the mappings and whether cache growth is expected. If cgroup memory grows while every process RSS remains stable, inspect kernel, socket, shared-memory, and page-cache charges.

A heap dump explains only memory managed by that runtime. Native libraries, direct buffers, thread stacks, JIT code, mapped files, and kernel-charged resources may remain outside it.

The term leak should describe a mechanism: memory that the system no longer needs remains retained or cannot be released. “RSS increased” is evidence to investigate, not the mechanism.

Linux uses RAM to cache file data because memory access is faster than rereading storage. Cache growth can push MemFree down while MemAvailable remains healthy.

Clean file-backed pages are often reclaimable, but reclaim is not free. If the workload needs those pages again, dropping them forces new faults and storage reads. A cache can therefore be reclaimable and still be valuable to performance.

Evidence of healthy cache use includes:

Evidence of harmful pressure includes falling headroom, active reclaim, repeated major faults, swap traffic, and memory stalls that track the service regression.

Do not routinely write to /proc/sys/vm/drop_caches during an incident. Dropping caches changes the workload, discards useful state, can create a large I/O burst, and does not identify why memory became constrained.

If host memory is disappearing but process residency does not explain it, inspect kernel categories:

Large SReclaimable may represent useful dentry, inode, or other object caches that can shrink under pressure. Growing SUnreclaim deserves closer attention because those slab objects are not considered reclaimable.

Inspect slab caches with:

Access to detailed slab information may require additional permission. A large cache name is not automatically a kernel leak. Compare its object count and bytes over time, then connect it to workload events such as process creation, filesystem traversal, connection growth, or a driver change.

KernelStack can grow with task and thread count. PageTables can grow with the number and shape of address spaces. Socket buffers and other kernel allocations may be charged to a workload cgroup without appearing in its runtime heap.

Kernel memory diagnosis follows the same method as process memory: identify a changing category, find the workload or event that drives it, and verify that controlling the driver changes both memory and the original symptom.

For a cgroup v2 workload, inspect its usage, boundaries, events, composition, and pressure:

The amounts in these cgroup files are bytes.

memory.current reports memory charged to the cgroup and its descendants. Charges can include anonymous memory, file cache, kernel memory, page tables, socket buffers, and shared memory. It is not the sum of process RSS.

memory.peak records a high-water mark when supported by the kernel.

memory.low is best-effort protection from reclaim. It does not reserve new physical memory or create capacity.

memory.high is a throttling and reclaim boundary. Crossing it can force tasks in the cgroup into expensive direct reclaim, but the boundary itself does not invoke the OOM killer.

memory.max is a hard usage boundary. If reclaim cannot keep usage within it, the cgroup can enter OOM and the kernel selects a victim from that cgroup.

Break down the current charge:

Entries in memory.stat are not all additive; several are subcategories. Use the keys to explain where the charge comes from. Do not depend on their order because kernels can add new fields.

memory.events contains cumulative counters:

The important meanings are:

An oom event does not guarantee that a process was killed, so compare it with oom_kill. A nonzero value from last week does not explain today's incident. Capture the counters before and after the event or use monitoring that records their rate of change.

memory.events includes descendant cgroup events. When only events local to that cgroup level are needed, use:

Cgroup directories can disappear when a short-lived container is deleted. Persistent monitoring or the container platform may be the only way to preserve these counters after termination.

Exit status 137 commonly represents termination by SIGKILL:

The kernel OOM killer uses SIGKILL, but so do administrators, supervisors, and container runtimes. Treat 137 as evidence of the signal, not proof of OOM.

For a cgroup workload, a contemporaneous increase in memory.events:oom_kill is strong confirmation. Platform termination state can provide another independent source.

For host or node events, inspect kernel logs when permitted:

Or:

Read the entire OOM record, not only the line naming the killed process. It can identify the exhausted scope, the task that triggered OOM handling, and the selected victim. The victim is not necessarily the workload that caused the long-term growth.

An application-level allocation failure is also different from an OOM kill. A runtime may report its own heap exhaustion, or a native allocator may return failure while the process remains alive. Correlate application logs, process exit state, cgroup events, and kernel logs.

On a NUMA machine, memory is divided among nodes. A restrictive memory policy or CPU placement can make one node important even while other nodes have free memory.

numastat, commonly provided with NUMA utilities, shows node-level memory:

Linux also exposes per-node data:

Consider NUMA evidence when a problem follows CPU placement, one node has poor headroom, remote-access counters change, or a process uses an explicit memory policy. Do not begin every memory incident here; host and cgroup pressure explain far more common failures.

Global free memory also does not satisfy every specialized allocation. Large physically contiguous allocations and pre-reserved huge-page pools have constraints that ordinary MemAvailable does not describe.

Suppose a Go API container restarts during a downstream-service outage. Its runtime reports exit code 137. The 64 GiB host still has more than 20 GiB available, and peer containers remain healthy.

The host is not the failed accounting boundary. The API's cgroup reports:

The container has a 1 GiB hard limit. Monitoring shows memory.current rising steadily from about 420 MiB toward that boundary after the downstream outage begins.

The memory.events deltas over the incident are:

The oom_kill increase confirms a cgroup OOM kill. Exit 137 is consistent with that evidence but did not prove it alone. high remains unchanged because memory.high is configured as max, so there was no lower warning boundary forcing reclaim before the hard limit.

memory.stat shows anonymous memory as the dominant growing category. Before the kill, smaps_rollup for the main process shows that Pss_Anon and private dirty memory account for most of its footprint. File cache and kernel memory remain comparatively stable.

Application metrics reveal an in-memory retry queue that grows whenever the downstream service is unavailable. A new retry policy stores complete request payloads without a count or byte limit. Queue length, private anonymous memory, cgroup usage, and memory pressure rise together.

The causal finding is:

The immediate mitigation is to disable or tightly cap retries and reduce incoming work until the downstream service recovers. Raising the limit may buy time but does not bound the failure.

The durable fix caps the queue by count and bytes, applies backpressure or rejects excess work, limits retry lifetime, and alerts on queue growth, memory.current headroom, memory pressure, and OOM events. A deliberate memory.high below memory.max can provide earlier pressure evidence, but it is not a substitute for bounded application state.

Verification must show that the queue remains bounded during another controlled downstream failure, cgroup memory plateaus with safe headroom, pressure stays acceptable, and no OOM event counter increases.

For a host-level symptom:

For a selected process:

For a cgroup v2 workload, capture memory.current, memory.peak, memory.high, memory.max, memory.swap.current, memory.swap.max, memory.events, memory.stat, and memory.pressure from its actual cgroup path.

Repeat the changing counters over a representative interval and align every capture with request load and latency. A collection of memory snapshots without a workload timeline cannot distinguish warm-up from a leak or historical swap use from active thrashing.

Memory investigation separates occupancy, pressure, ownership, and enforcement scope. Low free memory is often normal; falling available memory, reclaim work, swap traffic, fault rates, and PSI reveal whether memory scarcity is disrupting useful work.

At process level, distinguish virtual size from residency and use anonymous, file, shared, private, and proportional metrics to identify the growing category. Do not sum RSS or assume that a runtime heap explains the complete process.

At cgroup level, compare charged usage with memory.high and memory.max, break the charge down with memory.stat, and interpret event counters as interval deltas. A container can be reclaimed, throttled, or OOM-killed while the host remains healthy.

A complete diagnosis connects a workload event to a specific memory category, pressure or limit, and the original latency, allocation failure, or termination symptom.