A service can be “up” according to its service manager and still be unable to do useful work. Its main process may exist while worker processes have died, every request thread may be waiting, or the program may be restarting so quickly that a casual ps command misses the pattern.

Process investigation answers a more precise set of questions:

Linux exposes this information through tools such as ps, pstree, top, pidstat, service-manager commands, and the /proc filesystem. The commands are useful, but the investigation depends on interpreting them together.

A process listing is not a diagnosis. It is a time-stamped view of kernel state that must be connected to the reported symptom.

Before inspecting a PID, identify the operational object that users are experiencing. It may be a system service, container, scheduled job, shell command, or one member of a larger application.

Searching by name is a reasonable starting point:

-x asks for an exact process name, while -a prints the command line. If the executable name is not known exactly, searching the full command line can help:

Full-command-line matching is broader. It can match a wrapper, a helper, or an unrelated command containing the same text. Treat the result as a set of candidates rather than proof of identity.

On a systemd-based host, ask the service manager which process it considers the main process:

MainPID may identify a supervisor rather than the worker that handles requests. A service may also use several cooperating processes. The manager's active state means that its definition of the unit is satisfied; it does not prove that the application can serve traffic correctly.

Once a candidate PID is known, capture its basic identity:

Replace PID with the numeric process ID. The selected fields show:

Use lstart or another full timestamp when correlating the process with an incident. The compact etime field changes format as the lifetime grows, to [[days-]hours:]minutes:seconds, and a PID alone contains no lifetime information.

The %CPU value from ps is CPU time divided by the process's elapsed lifetime, not an instantaneous sample. %MEM compares the process's resident set with the machine's physical memory. Neither value by itself proves that the corresponding resource is constrained.

PIDs are reusable integers. After one process exits, Linux may assign the same number to an unrelated process. A note saying only “PID 4821 was unhealthy” may identify the wrong process by the time someone investigates.

Cross-check a target using several properties:

/proc/PID/exe points to the executable associated with the process. /proc/PID/cwd points to its current working directory. The owner, start time, executable, command line, cgroup, and place in the process tree together provide a much stronger identity than the PID or command name alone.

The comm and args columns are different:

Neither should be treated as a cryptographic identity. A process can change its short name, and a program may overwrite the memory used for its displayed command line. Script interpreters and runtime launchers can also make the visible executable different from the application being investigated.

The executable link may end with (deleted):

This means the process still refers to an executable whose directory entry has been removed or replaced. The running process does not automatically switch to the new file. This frequently explains why a deployment appears updated on disk while an old process continues executing the previous image.

For automation that must act on a process safely, a numeric PID followed by a delay creates a PID-reuse race. Prefer the service manager that owns the process or a programmatic interface using a stable process handle such as a Linux pidfd.

Applications rarely consist of one isolated process. Launchers create workers, shells start pipelines, service managers supervise daemons, and containers use an init-like process to manage descendants.

Show the ancestry and descendants of a known process with:

The options include process IDs and show the target's ancestors. A service might look conceptually like:

The supervisor, not systemd, owns the workers. A worker that misbehaves is the supervisor's responsibility to restart, and killing the supervisor takes all four children with it.

If pstree is unavailable, ps can display a system-wide hierarchy:

The parent-child relationship answers operational questions. If workers are zombies, their parent is responsible for reaping them. If a worker repeatedly disappears, its supervisor may create replacements. If the main service process is a shell script, the child process may be the actual server.

Do not infer communication or resource ownership solely from the tree. A child can communicate mostly with a remote service, and two unrelated processes can share files or sockets. The process tree describes creation and current parentage, not the complete flow of work.

Parentage can also change. When a parent exits, its children are reparented to an appropriate reaper, which may be PID 1 or a configured subreaper. A PPID of 1 therefore does not prove that PID 1 originally launched the process.

The one-character state column from ps summarizes whether a task can currently execute:

Most healthy processes spend much of their time in S. A server waiting for a connection, timer, work item, or lock should not consume a CPU continuously. “Sleeping” means it is waiting, not that it is broken.

R also requires context. It combines a task actually executing with one that is ready but waiting for a CPU. One R observation does not tell which case applies or how long it lasts.

D means the task is in an uninterruptible wait. Storage is a common reason, but it is not the only one. Kernel synchronization, some network filesystems, drivers, and other kernel paths can produce the same state. A brief D state during normal I/O is not automatically a problem. Many tasks remaining in D while latency rises is stronger evidence of a stuck or saturated operation.

T and t indicate that execution has been deliberately stopped. They are different from a sleeping task that is waiting for normal work. A service can remain present yet make no progress if its threads are stopped.

Z is a lifecycle record, not a running program. A zombie does not execute code and has already released most resources. Sending a signal to it cannot make it exit again. Persistent zombie accumulation points to a parent that is not collecting child exit status.

The stat column begins with the same state letter and may append modifiers. For example, Sl commonly means an interruptibly sleeping, multithreaded process. Do not mistake the extra characters for additional primary states.

If a task is sleeping, the wait channel can show the kernel function or wait point at which it sleeps:

The wchan value is a symbolic clue, not a stable user-facing API. Names depend on the kernel version and build, and access controls may hide them. A tool may display -, 0, or an address instead of a useful symbol.

Common categories include waiting for:

Interpret the channel in application context. An event-loop thread waiting for events is often healthy when there is no work. A request worker waiting on the same channel for several seconds during active traffic may be important.

One sample shows where the task was at one instant. Repeated samples showing the same request threads at the same wait point during the entire slowdown are more persuasive. Even then, a wait channel usually identifies where waiting is visible, not why the required event never occurred.

Linux schedules tasks. Each user-space thread has its own task ID and scheduling state.

For a conventional multithreaded process:

A process-level ps row can hide the behavior of its worker threads. Inspect them explicitly:

Here, PID is the thread-group ID and TID identifies each thread. PSR is the CPU on which the thread most recently executed; it does not guarantee that the thread is executing there at the moment the output is read.

For a live interactive view:

Thread names can reveal roles such as an acceptor, garbage collector, event loop, or worker. Names are only labels, however, and may be missing, truncated, or reused.

Linux also exposes each thread under:

For example:

Thread-level inspection is essential when one thread consumes a CPU while the rest sleep, one worker is stuck in D, or all workers wait on the same condition. A process-wide average can dilute each of these patterns.

The Threads value in /proc/PID/status and the NLWP column from ps provide a quick count. A growing count can indicate new workload or an application that keeps creating threads without retiring them. The count alone does not distinguish useful concurrency from a leak.

/proc is a virtual filesystem backed by kernel state. A numeric directory exists while the corresponding process exists:

The entries are not a frozen report. A process can change state between two reads, create a thread while its status is being inspected, or exit halfway through a command. Treat every value as a snapshot.

Identity and status

Start with a bounded selection from status:

This view combines identity, credentials, state, thread count, resident-memory information, and lifetime context-switch counters. The counters in one task's status file should not automatically be treated as totals for every thread in the process.

NSpid may contain multiple PID values when nested PID namespaces are involved. This helps relate the host-visible PID to the number shown inside a container.

Executable and filesystem context

Several symbolic links describe the process's view of files:

exe identifies the associated executable, cwd is the current working directory, and root is the process's filesystem root. The last value can differ from the host's root because of containers or other isolation mechanisms.

These links are useful when a service reads relative paths, starts in an unexpected directory, uses a replaced binary, or sees a different filesystem tree from the investigator.

Command line

The cmdline entry separates arguments with null bytes rather than spaces. Render one argument per line with:

An empty result can be legitimate for a zombie or a kernel thread. A user-space process may also alter the displayed argument area.

The process environment is exposed through /proc/PID/environ when permissions allow. Do not dump it casually: environments often contain credentials, access tokens, database URLs, and customer-specific configuration. Inspect only the required variable and handle captured output as sensitive data.

Limits and open resources

Per-process resource limits appear in:

This is more relevant than running ulimit -a in an investigator's shell. ulimit reports limits for that shell and its future children, which may differ from the service's actual limits.

Open descriptors are represented under:

The directory can quickly confirm that the process has pipes, sockets, files, devices, or deleted files open. A large listing should be counted or filtered rather than copied indiscriminately, and its contents can change during inspection.

Cgroup and namespace membership

Check the process's control-group membership with:

This connects a host PID to its service or container resource boundary. A process may be throttled or limited by that boundary even when host-wide resources look healthy.

Namespace links help compare execution contexts:

Processes with different namespace identifiers may see different PID sets, mount trees, or network stacks. This is why a PID, path, or socket observed inside a container may not match the host's view directly.

ps and most /proc reads are snapshots. Performance diagnosis usually needs rates and repeated samples.

pidstat, commonly installed through the sysstat package, can follow one process at one-second intervals:

The command takes five interval samples. Thread-level reporting is available with:

Depending on its options, pidstat can report CPU activity, scheduling activity, memory-related counters, or I/O. Select only the category needed to test the current hypothesis.

For an interactive process view, top -p PID keeps attention on one process. A single batch snapshot can be captured with:

Repeated sampling answers questions a single row cannot:

If a service restarts, a command pinned to the old PID will not automatically follow the replacement. Re-read the service manager's MainPID, start time, and restart count. A graph that silently switches from one process lifetime to another can otherwise produce misleading rates.

The process is absent

An absent PID does not explain why the process exited. Check the service manager's state, exit result, restart policy, and logs around the disappearance:

The process may have exited normally, crashed, been terminated, failed during startup, or been replaced by a new instance. Use timestamps to distinguish these cases.

The process exists but the service is unresponsive

Confirm that the PID belongs to the expected process, then inspect all threads rather than only the group leader. A mostly idle server may be waiting normally, have no runnable workers, be stopped, or be blocked on a dependency. Process state narrows the question but does not replace application-level health checks.

The process keeps restarting

Repeatedly short elapsed times, changing PIDs, and an increasing service-manager restart count indicate a restart loop. A one-time ps snapshot may make the process look healthy because it happens to catch the brief interval between startup and failure.

Threads accumulate

A rising NLWP or Threads count is evidence that creation outpaces exit. Correlate the increase with traffic, latency, memory use, and application configuration. A large stable pool may be intentional; unbounded growth is the suspicious pattern.

Tasks remain in D

Capture thread-level state and wait channels over several intervals. Determine whether the same tasks remain blocked and whether the count tracks the incident. Do not assume the cause is a local disk solely from the state letter.

Tasks are stopped

T or t explains why a present task is not running. Determine whether a debugger, job-control action, supervisor, or operator intentionally stopped it before attempting to continue or restart it.

Zombies accumulate

Find zombies and their parents:

The parent process, not the zombie, must collect the exit status. One transient zombie can be normal; a growing population under the same parent is an application or supervision defect.

The ability to see a PID does not guarantee access to all its details.

Linux applies ownership checks and security policies to sensitive /proc entries and diagnostic operations. A system may mount /proc with options that hide other users' processes. Container boundaries, Linux security modules, and debugging restrictions can further limit access.

Missing command lines, unreadable symbolic links, or hidden wait channels may therefore mean “permission denied,” not “no data exists.” Record the error instead of converting it into an empty fact.

Do not automatically run every command with elevated privilege. Higher privilege can expose secrets belonging to other services and expands the impact of a mistake. Use the minimum access approved for the investigation and capture only data relevant to the incident.

Inspection can also affect the target. Reading a few /proc files is usually inexpensive, while attaching a debugger, tracing every operation, or stopping a process can change timing or availability. Begin with low-overhead, read-only evidence.

Suppose a service manager reports that image-api.service is active, but the load balancer marks the instance unhealthy. CPU usage is low, and the main PID has existed for several hours.

Start with the service boundary:

The service manager reports:

The state is accurate from systemd's perspective: the declared main process exists. It does not yet explain whether request workers exist.

The process tree shows a supervisor and four children:

The worker PIDs have changed from the earlier listing while the supervisor's PID has not. New PIDs under the same parent mean the workers were replaced, not that they have been running all along.

A state-oriented listing reveals:

The supervisor is alive and sleeping in its event loop. Every worker has exited and remains as a zombie. The service manager has not restarted the unit because the declared main process never exited.

Logs aligned with 16:12 show that all workers rejected a newly generated configuration and terminated. The supervisor neither reaped the children nor created replacements.

The causal finding is:

Sending signals to the zombie PIDs would not restore service. A safe mitigation is to restore a valid configuration and restart the unit so it creates new workers. Verification must confirm that healthy worker processes exist and that the load balancer's original health check succeeds.

The durable fixes address both failure paths: validate configuration before replacing the known-good version, and make the supervisor reap and replace failed workers or exit so the service manager can recover the complete service.

For a known target PID, the following sequence captures a useful initial case file without tracing or changing the process:

Choose additional inspection based on what this snapshot reveals. Do not collect every available /proc entry by default, and do not forget to record the service symptom over the same interval.

If any command reports that the process no longer exists, check whether it exited or restarted. Do not continue assuming that the same numeric PID still identifies the same process.

Process investigation begins by identifying the correct service and process lifetime. Confirm a PID with its owner, start time, executable, command line, cgroup, and position in the process tree.

Use scheduling states as clues. R means running or runnable, S is commonly a normal interruptible wait, D is an uninterruptible kernel wait, T and t are stopped states, and Z identifies an exited child awaiting reaping. A wait channel can show where a task sleeps but does not by itself reveal the root cause.

Inspect multithreaded programs at thread level because each Linux task has its own state and activity. Use /proc/PID as a live case file for identity, filesystem context, limits, resources, cgroups, and namespaces, while remembering that every read is a race-prone snapshot.

Correlate repeated process observations with the original service symptom. The presence of a process, or an active service-manager state, does not prove that the application can perform useful work.