Operating systems enforce boundaries between users, processes, memory regions, files, devices, and kernel code. Those boundaries are strong only when software uses them correctly.

A privileged program can check one file and accidentally open another. A service can leak an administrative socket into a child process. A kernel driver can trust a length supplied by an unprivileged caller. In each case, the operating system may faithfully perform every requested operation while the program violates its own security intent.

These are OS-level vulnerabilities: flaws at or near operating-system interfaces that allow an attacker to cross a protection boundary, gain unintended authority, disclose data, corrupt state, or deny service.

The recurring causes are remarkably consistent:

Learning to recognize these broken assumptions is more useful than memorizing a list of famous exploits.

A bug is incorrect behavior. A vulnerability is a bug or design flaw that can be used to violate a security requirement.

The same defect can have very different consequences depending on where it runs. An out-of-bounds write in an isolated utility may crash only that utility. In a network-facing service, remote data may influence the process. In the kernel, one process may be able to affect the entire machine.

A vulnerability does not automatically imply a reliable exploit. Exploitability depends on what an attacker can control, what the corrupted state affects, which mitigations are active, and which privileges the vulnerable component holds.

A useful analysis starts with four questions:

Suppose a root-owned service accepts a pathname from an ordinary user and writes a report there. The string itself is merely input. The important boundary is that unprivileged input is controlling a filesystem operation performed with root authority.

That authority difference is what makes an ordinary pathname bug a privilege-escalation risk.

Programs written in memory-unsafe languages can access memory outside the lifetime or bounds of an intended object. Common forms include:

Consider a message containing a count and fixed-size records:

If count * sizeof(struct record) overflows size_t, bytes can become much smaller than intended. The allocation succeeds for the smaller size, but read_records() still processes count records and writes beyond the allocation.

The calculation must be checked before multiplication:

Every length conversion deserves similar attention. A negative signed length converted to an unsigned type can become a very large positive value. Adding a header size can wrap even if the payload length itself is valid.

Why kernel memory bugs are especially serious

An ordinary process is normally confined to its own virtual address space. Corrupting that process may expose only the process's memory and authority.

Kernel code shares one privileged address space and manages the protection mechanisms themselves. A memory error in a system call, filesystem, network stack, or device driver can therefore have system-wide impact:

One memory-safety failure produces three quite different outcomes. Which one occurs depends on what the corrupted data was used for, not on the bug itself.

Kernel interfaces must treat pointers, lengths, command numbers, and nested structures supplied by user space as hostile input. Good implementations copy data across the user–kernel boundary through the designated access functions, validate it, use checked arithmetic, and avoid repeatedly reading security-sensitive values from memory another thread can change.

Prevent the bug before relying on mitigation

The strongest defense is to avoid invalid memory access:

Production hardening adds valuable obstacles. Non-executable data pages make injected data harder to execute. Address-space randomization makes useful addresses less predictable. Stack canaries can detect certain stack overwrites before a function returns.

These mechanisms reduce exploitability; they do not make an out-of-bounds write correct. The underlying memory bug can still cause corruption, disclosure, or a denial of service.

A race condition exists when correctness depends on the timing or ordering of concurrent events.

Not every race is security-sensitive. A lost update to a display counter may be an ordinary correctness bug. A race becomes a security vulnerability when an attacker can change state between a security decision and the operation that depends on it.

The classic case is time of check to time of use, abbreviated TOCTOU:

The program checks and uses the same name twice, and the name means different things each time. Holding a descriptor from the first lookup removes the gap the attacker needs.

The check and use each succeed. The vulnerability is that they act on different filesystem objects.

The dangerous access() then open() pattern

The following pattern is especially risky in a privileged program:

On Linux, access() normally checks permissions using the process's real user and group IDs. An ordinary open() checks using the credentials effective for filesystem access. This distinction lets a set-user-ID program ask whether its invoking user would have access.

It does not bind the answer to a later open().

After access() returns, another process can rename an entry, replace it, or redirect lookup with a symbolic link. The later open() performs a new pathname resolution and may act on a different object using the program's elevated authority.

This is a general rule:

A successful permission or metadata check on a pathname says what was true during that lookup. It is not a reservation of the name or the object.

For ordinary operations, let the operation perform its own authorization:

If the operation must be authorized as the invoking user rather than as the privileged program, perform the open with the intended credentials. Do not check with one identity and reopen with another. Privilege changes are subtle, so a small unprivileged helper or a design that avoids elevated credentials is often easier to verify than temporary credential switching.

Open once, inspect once, use the descriptor

Sometimes the program must verify properties beyond normal access permissions. It may accept only regular files owned by a particular user.

The safer shape is:

The descriptor refers to the opened object even if its directory entry is later renamed or removed. fstat() and subsequent I/O therefore operate on the same object.

The check and use are still separate instructions, but the attacker can no longer substitute a different object merely by replacing the pathname. Any property that can itself change concurrently must still be evaluated according to the operation's exact security requirements.

A lock prevents a race only when every party that can change the relevant state obeys that lock. An attacker modifying a shared directory will not cooperate with an application-level mutex.

The reliable solution is usually an atomic kernel operation.

To create a new name only if it does not already exist, do not check that the name is absent and then create it in a separate operation. Request both conditions in one operation:

With O_CREAT | O_EXCL, creation fails if the name already exists. There is no successful gap in which another process can create the same name first.

Other interfaces encode similar atomic state transitions:

Atomicity must match the property being protected. An atomic create prevents a name-collision race, but it does not automatically make an attacker-controlled parent directory trustworthy.

A pathname is not just a string label. It instructs the kernel to walk directories, interpret . and .., follow symbolic links, cross mount points, and apply permissions along the way.

Input such as:

can escape an intended directory if the program simply joins it to a trusted prefix:

Rejecting literal .. components is not a complete solution. Symbolic links can redirect lookup, mount points can lead to different trees, and concurrent renames can invalidate a sequence of string-based checks.

Anchor resolution to an opened directory

Directory-relative interfaces let a program use a directory descriptor as a stable starting point:

This avoids dependence on the process's current working directory. It does not, by itself, guarantee that lookup remains below base. An absolute path can ignore the directory descriptor, .. can move upward, and symbolic links can redirect resolution.

Linux provides openat2() for stronger, kernel-enforced resolution policies. For example, a caller can combine a directory descriptor with:

RESOLVE_BENEATH rejects a resolution that escapes the directory represented by the starting descriptor. RESOLVE_NO_SYMLINKS rejects symbolic links in any pathname component.

These flags should reflect the application policy. Some valid directory trees intentionally use symbolic links, so rejecting all of them is not a universal default.

O_NOFOLLOW is narrower: for a normal open, it prevents following a symbolic link in the final pathname component. It does not prevent an intermediate component from being a symbolic link. Treating it as protection for the entire path leaves an important gap.

The parent directory controls the name

Checking only the target file's owner and mode misses another source of authority: the directory containing its name.

If an attacker can write to the parent directory, the attacker may be able to remove or replace an entry even without permission to modify the opened file's contents. A privileged program that validates a file, closes it, and later reopens the same name is vulnerable to that replacement.

This is why secure designs care about the complete resolution path, directory ownership and permissions, and stable descriptors, not just the final file's mode bits.

Shared temporary directories are intentionally writable by many users. Predictable names are therefore dangerous:

Another user can create the name between the check and the open. They may also create it before the program starts, possibly as a symbolic link to another object.

The sticky bit commonly used on /tmp restricts who may remove or rename existing entries. It does not stop another user from creating an available predictable name first.

Use an interface that chooses a difficult-to-predict name and atomically creates it:

mkstemp() creates and opens the file with exclusive creation semantics and mode 0600. The returned descriptor is the important result; continue using it instead of closing it and reopening the generated pathname.

On Linux systems using the GNU interface, mkostemp() can request O_CLOEXEC during creation:

When no other process needs the name, the program can unlink the entry after opening it and continue using the descriptor. The object then disappears automatically after the final descriptor is closed.

An even cleaner design is often to use a private runtime directory owned by the service rather than a globally writable directory. Secure creation and a trusted directory solve related but different problems.

Privileged code turns small bugs into large security failures. The safest privileged operation is one the process never has authority to perform.

Common mistakes include:

Credential transitions differ across operating systems and depend on real, effective, and saved IDs. They should not be improvised from a short sequence copied without its assumptions.

A robust architecture reduces the amount of code that ever holds extra authority:

The privileged component is deliberately the small one. Everything that handles untrusted input runs in the component that holds no privilege.

The unprivileged component can perform parsing, protocol handling, and most business logic. If it is compromised, it does not automatically acquire the launcher's full authority.

If a process must discard privilege permanently, it must account for all relevant credentials, supplementary groups, and retained capabilities; check every transition for failure; and verify the resulting state. A failed privilege drop must terminate the privileged path rather than continue under the false assumption that the drop succeeded.

A confused deputy is a component that has legitimate authority but is tricked into using that authority on behalf of a caller who should not have it.

Imagine a backup helper that may read every customer's files. A user asks it to back up /srv/customers/alice/data.

If the helper checks only that it is capable of reading the path, the request succeeds, because the helper itself has broad authority. The missing question is whether the caller is authorized to ask for that specific object.

The helper always has the authority to perform the action. The question it must answer is whether the caller does.

The helper must authenticate the caller through a trustworthy channel, authorize the requested operation for that caller, and bind the authorization decision to the object actually used.

Narrow interfaces help. “Write these bytes to the log descriptor” exposes less authority than “open any pathname and write these bytes.” Passing an already authorized file descriptor or opaque handle can avoid reinterpreting an attacker-controlled global name.

Least privilege limits the deputy's maximum reach, but it does not replace per-request authorization.

Creating a new program with execve() replaces code and memory mappings, but several pieces of process state survive unless deliberately changed. File descriptors are among the most security-sensitive.

Suppose a service opens:

It then executes an image-conversion utility. If those descriptors are inherited, the utility may use resources it could never open under its own credentials.

Set close-on-exec at descriptor creation:

In a multithreaded program, opening a descriptor and later setting FD_CLOEXEC with fcntl() creates a race. Another thread can fork and execute a new program between the two operations. O_CLOEXEC, SOCK_CLOEXEC, pipe2(O_CLOEXEC), and similar interfaces make descriptor creation and the inheritance policy one atomic step.

Other inherited state also deserves review:

A privileged process should not execute a shell command assembled from untrusted text. Shell metacharacters transform data into program syntax. Prefer a direct execution interface with a fixed executable, a separate argument vector, an explicit environment, and no search through an attacker-influenced PATH.

Some runtimes and loaders apply special restrictions during privileged execution, but security should not depend on every inherited variable being harmless by accident.

Kernel and privileged code often reads input from memory that another thread or process can modify.

Consider code that reads and validates a user-controlled length, allocates a buffer, reads the length again, and then copies that many bytes.

If the input changes between validation and the second read, the value used for the copy may not be the value that was validated. This is sometimes called a double-fetch vulnerability.

The safe pattern is to capture the security-sensitive input into trusted memory once, validate that snapshot, and use the validated value consistently:

Validating the caller's copy and then reading it again allows the value to change in between.

Nested pointers and variable-length structures require particular care. Validation must cover not only each individual field but also relationships such as:

Device-control interfaces are a frequent risk because one command can expose a large, driver-specific parser inside the kernel. Keeping these interfaces small, versioned, and rigorously validated reduces both mistakes and reachable attack surface.

A service that consumes unbounded memory, descriptors, tasks, CPU time, disk space, or queue entries can deny service to legitimate users.

The vulnerable pattern is often a cheap client request that causes an expensive or persistent server allocation. Repetition eventually exhausts the resource.

A memory leak caused only by rare internal behavior may be a reliability defect. If an unauthenticated client can trigger the leak repeatedly, it becomes a denial-of-service vulnerability.

Defenses need to match the accounting scope:

Simply raising a limit can increase the damage an attacker can cause. A large descriptor ceiling supports legitimate concurrency only when connection admission and descriptor lifetimes are controlled.

Security-sensitive failures must have an explicit safe outcome.

Suppose a service tries to drop privilege:

If setuid() fails, the process continues serving requests with elevated authority. The code's intended state and actual kernel state have diverged.

The safe structure checks the transition and stops if the secure state cannot be established:

This fragment illustrates fail-closed error handling, not a complete, platform-independent privilege-drop procedure. A complete implementation must handle all relevant credentials and verify the resulting state.

The same principle applies to:

“Continue with defaults” is safe only when the defaults deny the sensitive action.

Logging also needs care. Error messages should identify enough context for diagnosis without copying secrets, authentication tokens, private memory, or untrusted control characters into trusted logs.

No single operating-system defense covers every vulnerability.

Each layer assumes another may fail.

Memory protections can make exploitation harder but do not fix memory corruption. A system-call filter can limit what compromised code requests but cannot make allowed calls safe. Resource limits can contain exhaustion but do not repair a leak. Least privilege reduces the consequences of a confused deputy but does not correct missing authorization.

Timely updates remain necessary because the kernel, runtime, libraries, and privileged utilities are part of the trusted computing base. Hardening lowers risk; it is not a substitute for removing known vulnerabilities.

When reviewing code near an OS boundary, trace authority and object identity rather than only control flow.

For every input, identify who can change it and whether it remains mutable during use. For every pathname, ask where resolution begins, whether links or .. can redirect it, who controls each parent directory, and whether the program reopens the name after validation.

For every privileged operation, identify the exact authority required and how much code holds it. Check that credential changes, sandbox installation, and permission changes are verified. Review every descriptor and environment value that can cross an execution boundary.

Finally, examine failure paths. Security code is often well tested when every system call succeeds and least tested when allocation fails, a limit is reached, an object is replaced concurrently, or the kernel rejects a requested protection.

The goal is to make the secure operation the simplest possible state transition: authorize the request, acquire the object once, operate through a stable handle, and release unneeded authority promptly.

OS-level vulnerabilities usually arise when software makes a false assumption about memory, object identity, concurrency, privilege, inherited state, or resource use.

Memory-safety errors are especially dangerous in kernels and privileged services. Prevent them with safe abstractions, checked sizes, careful boundary validation, and testing; treat address randomization and non-executable memory as mitigation rather than a fix.

A TOCTOU vulnerability occurs when an attacker can change security-relevant state between a check and its use. Pathnames are mutable names, not stable handles. Prefer atomic operations, open an object once, inspect it through its descriptor, and continue using that descriptor.

Secure temporary files require atomic creation and safe naming. Directory-relative and constrained-resolution APIs help keep untrusted paths within an intended tree, while close-on-exec flags prevent descriptors from leaking authority into executed programs.

Keep privileged components small, authorize requests for the caller rather than for the helper, verify every security transition, and fail closed. Combine bug prevention with least privilege, interface reduction, containment, monitoring, and timely updates so that one defect does not become a system-wide compromise.