Operating-system concepts are easier to understand when you can observe them directly.

Instead of only reading about processes, memory, files, and system calls, you should be able to answer questions such as:

Linux provides several tools for answering these questions.

This chapter introduces the small set of tools we will use throughout the course:

A useful way to organize them is by which part of a running program they let you see:

ps and top observe a process from the outside (as a row in a process table) while gdb can pause it mid-instruction and inspect its variables. /proc spans several of these layers.

You do not need to memorize every option. The goal is to understand what each tool is for and how to use it when investigating a program.

We will use the following program throughout the chapter to perform the investigation:

Save it as observe.c.

The program opens a file, writes its process ID, and then waits for you to press Enter. Because it remains running, we have time to inspect it from another terminal.

The man command opens the system’s manual pages.

For example:

This shows documentation for the ps command, including its options and examples.

You can search inside a manual page by pressing / and entering a word:

Press n to move to the next match and q to exit.

Manual pages are also available for C library functions and system interfaces:

Sometimes the same name exists in multiple sections of the manual. For example, printf is a shell command on some systems and also a C library function.

You can specify the section explicitly:

Common manual sections include:

For example:

The first opens the documentation for the write system call. The second opens the documentation for the fopen library function.

A useful habit is:

Before searching randomly online, check whether the command or function has a manual page.

Compile the sample program using:

This creates an executable named observe.

The compiler flags are important:

Run the program:

You should see output similar to:

Do not press Enter yet.

Warnings are especially valuable in systems programming because C allows many unsafe operations. The compiler may detect uninitialized values, suspicious conversions, incorrect function arguments, and unused variables.

For example:

Compiling with warnings may report that count is used without being initialized.

Treat warnings as problems to investigate rather than harmless messages to ignore.

Keep the sample program running and open another terminal.

Use ps to find it:

You may see output similar to:

The exact columns vary, but the important values include the process ID, parent process ID, and command.

A cleaner command is:

Example output:

Here:

The state S means the process is sleeping. In this case, it is waiting for keyboard input rather than actively using the CPU.

Some commonly observed states are:

These states will make more sense after we study processes in detail.

You can also inspect one known process directly:

Replace 4821 with the process ID printed by your program.

While ps gives you a snapshot, top continuously updates process information.

Run:

The top section shows system-wide information such as CPU usage, memory usage, load average, and the number of running processes.

The lower section displays individual processes.

To watch only the sample program, run:

Replace the PID with your program’s actual process ID.

Because the program is waiting for input, its CPU usage should remain close to zero.

Inside top, press:

to sort processes by memory usage, or:

to sort by CPU usage.

Press q to exit.

top is useful when a machine feels slow and you need an immediate overview. It can quickly show whether a process is consuming excessive CPU or memory.

However, it usually tells you which process is expensive, not why it is expensive. Other tools help with the deeper investigation.

Linux exposes information about running processes through a virtual file system called /proc.

Each running process has a directory named after its PID.

For process 4821, the directory is:

This directory does not exist as ordinary data stored on disk. The kernel generates its contents dynamically.

Inspect the process status:

This includes information such as the process name, state, parent PID, thread count, and memory usage.

You can display the command used to start the process:

Inspect its current working directory:

Inspect the executable being run:

View its memory mappings:

This shows regions used for the executable, shared libraries, heap, stack, and other mappings.

You do not need to understand every line yet. The important idea is that /proc provides a detailed view of how the kernel sees a process.

Open File Descriptors

Every process has a file descriptor directory:

You may see something similar to:

The first three file descriptors usually represent:

File descriptor 3 refers to output.txt, which our program opened using fopen.

This demonstrates an important operating-system idea: a process interacts with files and input/output resources through small integer identifiers called file descriptors.

We will study them in detail later.

A C program can calculate values and modify its own memory without asking the operating system for help.

However, opening a file, writing to the terminal, reading input, and exiting all require operating-system services.

strace shows these requests. It sits at the boundary between the program and the kernel, and records each crossing:

strace does not read your program's variables or step through its logic. It only sees the requests that leave the process, which is why arithmetic and memory updates produce no output at all.

Run the program through strace:

The output may appear overwhelming because program startup involves many operations.

Near the end, you should find calls related to opening the file, writing data, displaying text, and waiting for input.

A simplified example might look like:

The value after = is the result returned by the system call.

For example:

means the file was opened successfully and assigned file descriptor 3.

You can limit the trace to a few relevant calls:

To save the trace to a file:

You can then inspect it using:

strace is especially useful when a program fails because of a missing file, incorrect permission, unavailable resource, or unexpected system interaction.

For example, a failed file operation may appear as:

ENOENT means that the requested file or directory does not exist.

Instead of only seeing that the application failed, you can see the exact request that failed.

The name lsof stands for list open files.

On Linux, the term “file” includes more than regular files. It can also include directories, devices, pipes, and network sockets.

To inspect the files opened by the sample program, run:

You should find entries for the executable, shared libraries, terminal, and output.txt.

To see which process has opened a specific file:

This is useful when a file cannot be deleted, unmounted, or modified because another process is using it.

lsof can also inspect network activity.

For example, to show processes listening for TCP connections:

Later, when we build network servers, lsof will help us confirm which ports they have opened.

The main distinction between /proc/<PID>/fd and lsof is convenience. /proc exposes the kernel’s raw process information, while lsof collects and presents open resources in a more readable format.

gdb allows you to pause a program, inspect variables, move through code one line at a time, and investigate crashes.

Because we compiled the program with -g, debugging information is available.

Start the debugger:

You will enter the GDB prompt:

Set a breakpoint at main:

Start the program:

The program pauses before executing the first line of main.

Move to the next source line:

Inspect a variable:

Continue execution:

Exit GDB:

Investigating a Crash

Consider this broken program:

Compile it:

Run it inside GDB:

Then:

The program should stop when it attempts to dereference the null pointer.

Use:

to display the sequence of function calls that led to the crash.

For this small program, the backtrace may contain only main. In larger programs, it helps identify which chain of function calls caused the failure.

You can inspect the current line and nearby source code using:

gdb is most valuable when the program’s internal state is the problem. strace shows interactions with the operating system, while gdb shows what is happening inside the program itself.

Different tools answer different questions.

These tools are often used together rather than independently.

Suppose a service becomes unresponsive.

You might first use ps to confirm that it is still running. Then use top to check whether it is consuming CPU. You could inspect /proc to understand its state, use lsof to examine its files and sockets, and finally use strace or gdb to investigate what it is doing.

This creates a repeatable workflow:

The order matters more than the individual commands. The early steps are cheap and answer broad questions, so they often make the later steps unnecessary. Reaching for gdb first usually means examining a program in detail before knowing whether it was the right program to examine.

Start the sample program:

Keep it running and note its PID.

From another terminal, perform the following investigation:

Replace <PID> with the process ID printed by the program.

Now attach strace to the running process:

Return to the first terminal and press Enter.

The trace should show the program reading from standard input, closing its file, and terminating.

This single exercise reveals several operating-system concepts:

You will study each of these ideas in much greater depth later.

Linux provides tools that allow us to observe operating-system behavior instead of treating it as a hidden black box.

man explains commands and programming interfaces. gcc compiles programs and identifies suspicious code. ps and top show running processes and resource usage.

The /proc file system exposes detailed kernel information about each process. strace reveals system calls, while lsof shows open files and sockets. gdb lets you pause a program, inspect its state, and investigate crashes.

You do not need to memorize every command. Remember the purpose of each tool and return to its documentation when you need a specific option.

The central idea is:

Systems tools help you move from guessing what a program is doing to observing what it is actually doing.