A backend service runs normally for several days, then suddenly stops accepting connections. Existing requests continue for a while, but new database connections fail, log rotation breaks, and even reading a configuration file reports:

These failures appear unrelated until one shared dependency is recognized: every operation needs a new file descriptor.

File-descriptor exhaustion is a capacity failure in an allocation table. The immediate task is to determine which table is full. The deeper task is to explain why descriptors accumulated, whether through expected concurrency, an undersized limit, or a lifecycle bug that failed to release them.

A file descriptor is a process-local integer that refers to an open kernel object. Regular files, directories, sockets, pipes, terminals, and Linux event objects can all consume descriptors.

Exhaustion therefore causes cascading symptoms:

A service may remain alive because operations using existing descriptors can continue. Health checks that require a new connection or file can fail while an established connection still works. Logging may become incomplete if the logger itself cannot open a destination.

Do not diagnose this incident from the text “too many open files” alone. Preserve the exact failing operation and errno.

Linux exposes two important exhaustion scopes.

EMFILE means the calling process has no descriptor number available under its RLIMIT_NOFILE limit. Other processes may continue opening resources normally.

ENFILE means the system-wide open-file-handle limit has been reached. Allocations can fail across unrelated processes.

The distinction matters operationally. Raising a kernel-wide limit cannot repair a process that has reached its own soft limit. Raising one process's limit cannot repair system-wide ENFILE.

Some application frameworks discard errno and emit only a generic message. Search structured logs, exception chains, and kernel-facing diagnostics for the actual error. A focused trace of a safe reproduction can show it directly:

The failing syscall identifies what the application could no longer acquire. It does not identify what consumed the existing descriptors.

Resolve the exact service process before counting anything. A supervisor may own one PID while several worker processes handle requests, and a restart can replace the failing process.

Capture identity and start time:

Read the limit that the running process actually has:

Example:

The first number is the soft limit currently enforced. The second is the hard limit, which is the ceiling an unprivileged process can use when raising its own soft limit.

RLIMIT_NOFILE is defined as one greater than the largest descriptor number the kernel may allocate to the process. With a limit of 8,192, newly allocated descriptor numbers must be below 8,192. Operations that cannot allocate within the limit fail with EMFILE.

prlimit provides another read-only view when used without a new value:

Permissions and installed tools vary, but /proc/2471/limits remains the simplest source for the live process.

Do Not Use the Investigator's ulimit

These commands describe the current shell:

They do not describe an already running service. Resource limits are normally inherited when a process is created, and a program may adjust its own soft limit afterward.

For a systemd service, inspect the manager's values:

This helps explain how the service was started, but the target's /proc/PID/limits is the final verification. A unit-file change does not retroactively alter an already running process unless an authorized mechanism explicitly changes that process's limit.

Container runtimes can also apply resource limits when launching a container. Host shell limits and container-process limits may differ, so inspect the actual PID in the relevant PID namespace.

Linux represents each open descriptor as a numeric entry in /proc/PID/fd:

If the count is close to the process's soft limit while allocation calls return EMFILE, the immediate scope is confirmed.

This is a live, racy snapshot. Descriptors can open and close while the directory is being read. That does not make the count useless; it means small differences should not be treated as exact accounting.

Inspect the entries:

Typical targets include:

The numeric name is the descriptor in that process. A target such as socket:[4112804] identifies a socket inode, pipe:[...] identifies a pipe endpoint, and anon_inode:[eventpoll] identifies a kernel object without a normal filesystem inode.

Access to another process's descriptor links can be restricted. A disappearing entry usually means the descriptor closed during inspection, not that /proc is corrupt.

The count confirms pressure; the descriptor types identify the likely ownership path.

Use lsof for a human-readable inventory:

-nP avoids hostname and service-name resolution, which keeps the command faster and prevents diagnostics from depending on working DNS.

Do not use the number of lsof output lines as the descriptor count. lsof can include non-descriptor references such as the current working directory, root directory, executable text, and memory mappings. Numeric entries under /proc/2471/fd are the direct representation of the descriptor table.

To focus on network sockets owned by the process:

Socket state and endpoints can then be inspected with:

Filter these results to the target process or service endpoint on a busy host.

For a quick approximate category count using /proc:

Run these close together and remember that the process is changing underneath them.

The category suggests the next question:

One epoll descriptor can monitor many other descriptors without duplicating them. Similarly, one inotify descriptor can contain many watches. Do not assume that every logical watch consumes a separate process descriptor.

A high descriptor count is not necessarily a leak. A server with 20,000 active client connections may legitimately hold more than 20,000 descriptors.

A leak is a lifecycle pattern:

Measure the count repeatedly over a bounded interval:

Interpret the trend alongside workload:

Normalize when possible. Descriptors per active connection, per worker, per queued job, or per completed request are more meaningful than a raw count.

Historical metrics are much stronger than an emergency snapshot. Monitor both open-descriptor count and maximum descriptors for long-running services. Alert on sustained headroom loss and growth rate, not merely on one universal count.

Resource limits and descriptor tables are process-scoped. Linux threads in a normal multithreaded process share one descriptor table, so counting every thread would duplicate the same table.

Multi-process services are different. A prefork server may have one supervisor and many workers, each with its own table and limit. Identify the process tree:

Inspect the workers individually. One worker may leak while its siblings remain healthy, or every worker may grow at the same rate because they execute the same faulty code.

Forked processes inherit descriptors that remain open unless explicitly closed or marked close-on-exec for a subsequent program execution. A parent can have a stable count while children retain inherited references. The service-wide resource lifetime therefore may not be visible from the main PID alone.

Linux exposes system-wide open-file-handle accounting through:

file-nr contains three values:

Linux 2.6 and later normally reports zero for the second value. file-max exposes the system-wide maximum separately. When the system reaches that limit, allocations can fail with ENFILE, and the kernel may log:

Search recent kernel messages when ENFILE is suspected:

nr_open is a different ceiling: it limits how high a process's RLIMIT_NOFILE hard limit can be raised. It is not the current number of open descriptors and not the system-wide file-handle limit.

Why the Counts Do Not Add Up Directly

The process and system values measure different objects:

The three descriptors are three process-table entries but can refer to one shared open file description after duplication or inheritance. /proc/PID/fd counts descriptor entries for one process; file-nr counts system file handles. Summing all process descriptor counts does not have to equal file-nr.

System-wide exhaustion is less common than one-process exhaustion, but its impact is broader. Finding the owner may require examining several high-usage services rather than one PID. Avoid an unrestricted lsof scan on a heavily loaded system unless necessary; walking every process and descriptor can itself be expensive, and diagnostic tools may fail when the system has almost no allocation headroom.

Different descriptor populations point toward different cleanup mistakes.

Network sockets

A rising socket count can result from accepted client connections, outbound connection creation, or connections retained in a pool. Group them by state and peer.

A sustained increase in CLOSE-WAIT means peers have closed their sending side while the local application has not completed local cleanup. This is strong evidence about the missing close direction, though it does not identify the source-code path by itself.

Large numbers of established sockets may be healthy concurrency. Compare them with active requests, pool configuration, and expected connection reuse.

Regular files and directories

Repeated paths suggest that the application opens the same resource without closing earlier descriptors. Unique temporary paths can indicate per-job files retained after completion. Directory iteration APIs can also own descriptors until their handles are closed.

A path ending in (deleted) means the directory entry is gone while a descriptor still refers to the open object. It consumes a descriptor whether or not it also retains filesystem blocks.

Pipes and local IPC

Pipes have two ends, and multiple processes can inherit copies. A pipeline may remain open because one process kept an unused write end, preventing readers from observing end-of-file. Map both endpoint inodes and the processes holding them before blaming the process with the largest count.

Anonymous kernel objects

Descriptors for epoll, eventfd, timerfd, signalfd, and inotify appear as anon_inode targets. Repeated creation of event loops, timers, or notification instances without teardown can leak these descriptors just as files or sockets can.

When inventory and trends identify a descriptor family but not the code path, trace a small reproducible workload:

This trace can reveal a repeating sequence such as:

If descriptor 41 is never closed and the pattern repeats, the timeout path is a strong leak candidate.

Do not mechanically compare the number of acquisition calls with the number of close() calls. pipe() and socketpair() create two descriptors, dup() creates a new reference, descriptors can be inherited or transferred, close_range() can close many entries, and process exit closes all remaining descriptors. Follow ownership for the operation being reproduced.

Tracing is intrusive and can generate large output on a high-throughput service. Prefer staging, a test case, or a narrow low-rate reproduction over an indefinite production trace.

There are three common conclusions.

The workload is healthy but the limit is too low. Descriptor count tracks expected concurrency, returns toward baseline, and the service's capacity model requires more descriptors than its configured soft limit.

The limit is reasonable but descriptors leak. Count grows independently of useful concurrency, a descriptor category accumulates, and completed operations do not release their resources.

Both are true. A low limit causes an early incident, while a slower leak guarantees that any finite replacement limit will eventually be reached.

Estimate expected capacity before changing a limit:

Do not size exactly to steady-state demand. Restarts, deployments, failover, bursts, and diagnostic operations require headroom.

A higher RLIMIT_NOFILE does not preallocate every possible descriptor, but it permits the process to retain more kernel and application resources. A server using the legacy select() interface can also be unable to represent descriptor numbers at or above its fixed descriptor-set size even when the operating-system limit is higher. Verify the application's I/O model before assuming an arbitrarily large limit is safe.

If impact permits, capture:

A controlled restart closes the process's descriptors and can restore service quickly, but it destroys the live table that proves what accumulated. Capture a bounded snapshot first when possible.

Raising the soft limit can be a valid short-term mitigation for expected load or to buy time during a leak investigation. It is not evidence that the previous limit was the root cause. If count continues growing, the process will eventually reach the new limit.

Do not attempt to close arbitrary descriptors inside another live process. Descriptor numbers can be reused immediately, shared state may have other owners, and forced closure can corrupt application protocols or data. Remediation should use the application's normal cleanup path, controlled traffic reduction, or a runbook-approved restart.

An API begins rejecting new clients roughly four days after each deployment. Existing connections continue briefly, and logs contain:

The live process shows:

The soft and hard limits are both 8,192. Repeated descriptor counts fluctuate between 8,185 and 8,192 while accept4() returns EMFILE.

System-wide accounting shows ample headroom:

The first file-nr value is far below file-max, and unrelated processes can still open files. This confirms per-process exhaustion rather than ENFILE.

Descriptor classification shows that approximately 7,700 entries are TCP sockets. Filtering socket state by PID reveals more than 6,900 connections in CLOSE-WAIT, almost all from the same upstream service. The count grows by about twenty per minute even though request rate is stable.

A staging reproduction forces the upstream to close during a cancelled request. The syscall trace shows that the client socket is created and receives the peer's close, but one cancellation path returns without closing the descriptor. The newly deployed request-timeout code introduced that path.

A rolling restart restores capacity after the evidence is captured. The code is then changed so socket ownership is released on every success, error, timeout, and cancellation path. Under the same test, the descriptor count now varies between 350 and 600 and returns toward baseline after traffic subsides. CLOSE-WAIT no longer grows.

The root cause is:

The request-cancellation path failed to close upstream sockets after the peer ended the connection. Those descriptors accumulated in CLOSE-WAIT until the API reached its per-process limit of 8,192, causing accept4() to fail with EMFILE.

Raising the limit would have delayed the outage without removing this mechanism.

Use this sequence when a service reports “too many open files”:

Stop when the evidence explains which table filled, which descriptor class grew, which lifecycle retained it, and why the observed operation could no longer allocate.

File-descriptor exhaustion is either per-process EMFILE or system-wide ENFILE. Confirm the exact errno, then compare the target process's live RLIMIT_NOFILE with numeric entries under /proc/PID/fd. Use file-nr and file-max for the separate system-wide boundary.

Classify descriptors by object type and measure how the dominant category changes with workload. High occupancy may be valid capacity; persistent growth without release indicates a lifecycle problem. Inspect every service process, account for inheritance, and use a narrow reproduction to trace acquisition and cleanup when necessary.

Limits provide capacity, not ownership correctness. A complete diagnosis explains which table filled, which resources accumulated, why they were not released, and whether remediation restores stable descriptor headroom.