AlgoMaster Logo

Crash Consistency

29 min readUpdated August 7, 2026
Listen to this chapter
Unlock Audio

A process creates:

The operation looks like one action to the application. Inside the file system, it can require several persistent updates:

Now imagine power fails after only some of those updates reach storage.

The directory might contain a name whose inode was never initialized. A block might be marked free even though an inode points to it. An allocated inode might become unreachable, leaking space.

Preventing or repairing such partial states is the problem of crash consistency.

A crash-consistent file system preserves its structural invariants across an abrupt stop, even when one logical operation required several persistent writes.

Crash consistency does not necessarily preserve the newest application data. A file system can recover to an older but valid state.

Process Crashes vs. Machine Crashes

The word crash can describe different failures.

If one application process terminates unexpectedly, the kernel normally keeps running. It closes the process's descriptors and can continue processing data and metadata already accepted into kernel memory.

When an application process crashes, the kernel remains alive and can continue the file-system work already handed to it.

A machine crash, kernel panic, or sudden power loss is different:

When the machine stops abruptly, volatile RAM state is lost, and only completed persistent effects remain.

File-system crash consistency primarily addresses the second failure model. The file system may restart with storage containing some, but not all, updates that were in progress.

A storage-device failure is another category. If a device permanently loses blocks or returns corrupted data, ordering alone cannot reconstruct every byte. Crash consistency assumes the storage remains broadly usable and recovery can reason about the surviving state.

The Basic Failure Model

A useful simplified model assumes:

  1. The system is modifying several blocks of file-system data and metadata.
  2. It stops at an arbitrary point.
  3. Volatile state disappears.
  4. Some issued storage writes reached persistent media and others did not.
  5. Recovery runs before normal operation resumes.

This is a fail-stop model: execution halts rather than continuing with arbitrary malicious behavior.

Even this simplified model contains difficult cases.

Writes can complete in a different order from the order in which software logically requested them. Devices and controllers can contain volatile caches. A file-system block can be larger than the atomic-write unit guaranteed by the storage stack, making a torn write possible: part of the old block and part of the new block survive.

The exact atomicity and ordering guarantees depend on the device, controller, driver, and file-system protocol. A correct design cannot simply assume that source-code order equals persistence order.

Invariant-Based Consistency

A file system is structurally valid when important relationships agree.

Typical invariants include:

  • Every directory entry refers to a valid inode of the expected file system.
  • A block marked free is not simultaneously referenced as allocated file data.
  • One allocated block does not belong to unrelated files unless explicit sharing is supported.
  • Inode link counts agree with directory-entry references closely enough for correct lifecycle management.
  • File size and block mappings form a valid logical layout.
  • Free-inode and free-block tracking agrees with reachable allocated objects.
  • Directory structure does not contain invalid parent relationships or unsupported cycles.

These are file-system invariants, not application-format rules.

A JSON file containing:

can be application-level corruption while the file system remains structurally consistent. Its inode, blocks, size, directory entry, and free-space accounting can all be valid.

This creates three distinct goals:

A file system can provide the first without automatically providing the other two.

Multi-Block Logical Operations

Consider creating an empty file named report.txt.

Conceptually, the file system must establish at least:

These pieces may occupy separate storage blocks.

The arrows express a dependency, not merely CPU execution order. The directory entry should not become persistent while it can point to a free or uninitialized inode.

If a crash occurs after allocation and initialization but before the directory entry, the inode can be allocated but unreachable. That is a leak, but it is generally safer than a directory name pointing at an inode that another file can also allocate.

This illustrates an important design preference:

When perfect atomicity is unavailable, an unreachable allocation is often easier to repair than a live reference to unowned or invalid storage.

Recovery still needs a way to identify and repair the leak.

The Create Operation's Possible Partial States

Let:

Possible crash states include:

Persisted piecesResult
NoneOld state: no file exists
A onlyAllocated but uninitialized inode; leaked metadata
A and IValid inode with no name; unreachable allocation
A, I, and DComplete new file
D without A or IDirectory points to invalid or reusable inode
A and D without IDirectory points to uninitialized metadata

The first four states can arise from a dependency-respecting prefix. The final two violate the intended ordering and can create dangerous aliasing or garbage metadata.

Simply issuing A, then I, then D does not prove only prefix states can survive. The persistence layer can reorder work unless the file system uses ordering controls or a stronger update protocol.

Loading simulation...

Dependencies of Data Appends

Appending to a file can require:

Suppose the file previously ended at 8 KiB and receives another 4 KiB.

Partial persistence creates several risks.

If the inode points to a block still marked free, the allocator can give that block to another file. Two files can then refer to one allocation unintentionally.

If the block is marked allocated but never connected to the inode, storage leaks.

If the new size persists without a valid mapping or initialized data, reads can expose an unintended region or produce a layout the file system must interpret as a hole.

If new data reaches storage but the old size remains, those bytes are outside the logical file and may become unreachable.

No single ordering prevents every undesirable outcome while also making the whole operation atomic. The file system needs a protocol that can distinguish complete updates from interrupted ones or reconstruct an acceptable state.

Reversed Dependencies During Deletion

Removing a file's final name can involve:

Freeing storage before removing every live reference is dangerous:

Removing the name first is safer:

A crash in that window can leak storage, but no surviving pathname points to an object already made available for reuse.

Again, a leak is undesirable but structurally easier to find than double allocation.

Open file descriptions add runtime state. Before a crash, an unlinked file can remain alive through an open descriptor. After a machine restart, those process references no longer exist, so recovery must finish reclaiming any object that had no names but was kept alive only by pre-crash runtime references.

Rename Atomicity vs. Persistence

Within one file system, rename() is designed to be atomic with respect to namespace observers:

A concurrent lookup should not observe half a pathname operation.

Crash persistence asks a different question:

Renaming can update multiple directory blocks, link counts, and timestamps. Replacing an existing destination also changes the lifecycle of the old object.

The file system's crash protocol must make recovery choose an acceptable namespace state. Depending on the guarantees and which persistence steps completed, recovery can expose the old name arrangement or the new one.

Application code should not infer crash durability merely from concurrency atomicity:

The distinction is central to safe file replacement.

Ordering, Atomicity, and Durability

These terms answer different questions.

Ordering

If update B depends on update A, ordering ensures A becomes persistent before B can become persistent.

Ordering can prevent dangerous combinations but can still expose an incomplete prefix.

Atomicity

Atomicity means observers or recovery see all of a logical update or none of it.

One file operation can require more persistent writes than the hardware can atomically update, so the file system builds a higher-level protocol.

Durability

Durability means an update that has reached a defined completion point will survive the specified failures.

A state can be consistent but not durable:

A state can also contain durable bytes but be inconsistent if related metadata did not persist.

Good APIs and file-system designs state which completion point provides which property. A successful ordinary write() alone usually reports kernel acceptance, not a complete power-loss guarantee.

Persistence Order vs. Program Order

Imagine code logically performs:

The lower storage path can queue, combine, and reorder operations. If the inode update reaches persistence first, a crash can leave the inode pointing to data that did not.

File systems use mechanisms such as dependencies, barriers, flushes, transaction records, or copy-on-write publication to control what recovery can observe.

The correct mechanism depends on the storage contract. If a device reports completion before data is truly persistent, or ignores required ordering controls, the higher-level guarantee can fail despite correct file-system logic.

Hybrid Blocks from Torn Writes

Ordering complete blocks is not enough if one metadata block can tear.

Suppose a directory block contains many entries and the file system modifies one:

A torn write could leave part of the old block and part of the new block. If internal lengths or checksums no longer agree, the entire directory block may be unreadable.

File systems defend against this in different ways:

  • Restrict critical updates to a known atomic storage unit where possible
  • Record checksums to detect incomplete writes
  • Write a recoverable copy before replacing the main location
  • Store new metadata elsewhere and publish it through a smaller atomic pointer update

A checksum detects that bytes are wrong. It does not by itself reconstruct the intended bytes. Detection and recovery are separate capabilities.

Old-State Preference in Crash Consistency

It is tempting to define success as “the newest state always survives.” That is durability, not the minimum requirement for consistency.

Suppose an application creates report.txt, receives a successful return, and the machine loses power before the update is guaranteed persistent.

After recovery, either state can be structurally valid:

An inconsistent mixture would be:

Returning to the old valid state can be acceptable for file-system consistency even though the application loses recent work.

This distinction prevents a misleading conclusion:

“The file system recovered cleanly” means its structures are usable; it does not mean every recent application update survived.

Recovery by Full Consistency Checking

One recovery approach scans file-system metadata and rebuilds relationships. Unix systems traditionally use tools in the fsck family for this purpose.

A checker can:

  • Walk directory trees and validate referenced inodes
  • Recompute link counts from directory entries
  • Compare allocated-block maps with inode mappings
  • Detect blocks claimed by more than one incompatible owner
  • Find allocated but unreachable inodes or blocks
  • Repair free-space accounting
  • Place recoverable orphaned objects in a special recovery directory

This method can restore structural consistency without knowing which high-level operation was in progress.

Its limitation is cost. Scanning a very large file system can take substantial time, especially when the checker must examine metadata proportional to the whole file-system size.

A checker also cannot infer every application intention. It can identify a structurally valid file, but it does not know whether an incomplete database transaction should commit or roll back.

Recovery tools should normally operate on an unmounted or otherwise safely quiesced file system. Repairing metadata while normal writers mutate it can make observations stale and cause further damage.

Recovery-Ambiguity Reduction Through Protocols

Modern file systems use protocols that leave enough evidence to distinguish complete updates from interrupted ones.

Three broad approaches are common.

Ordered updates with repair

The file system persists dependent updates in a safe order so crashes favor leaks over dangling references. A checker repairs remaining leaks and accounting differences.

Ordering alone can require many waits and still leaves recovery work.

Write-ahead journaling

The file system records a transaction description in a dedicated log before treating updates to their normal locations as complete. Recovery can identify committed transactions and replay or ignore work according to the log protocol.

Journaling can bound recovery work and provide multi-block metadata atomicity. The exact protection depends on whether the journal covers metadata only or includes file data.

Copy-on-write publication

The file system writes changed blocks to new locations, leaving the old structure intact. After the new tree is complete, it switches a root or parent reference to publish the new state.

Recovery selects a valid old or new tree. The allocator must still avoid prematurely reusing blocks and must manage tree and reference metadata consistently.

These are families of techniques, not guarantees by name alone. Each implementation defines which operations and data are covered.

Metadata Consistency vs. File-Content Guarantees

Many file-system consistency mechanisms focus on metadata:

After recovery, a file can have a valid name, inode, size, and block mapping while its newest data was not persisted.

Possible content outcomes include:

  • Old data
  • New data
  • Zero-filled regions
  • A partial application record
  • A valid file-system byte sequence that violates application format

A metadata-consistent file system should not expose blocks belonging to unrelated files, but it cannot promise that every unsynchronized application write survived.

Applications needing transactional contents use their own techniques such as write-ahead logs, checksummed records, versioned files, or append formats that detect and discard incomplete tails.

The file system protects the container of bytes. The application protects the meaning of those bytes.

Safe File Replacement as a Protocol

Overwriting a configuration file in place can expose a partial file:

A common replacement protocol is:

  1. Create a temporary file in the destination directory.
  2. Write the complete new contents.
  3. Validate the temporary file.
  4. Ensure the temporary file's required contents and metadata have reached the intended persistence point.
  5. Rename the temporary file over the destination.
  6. Ensure the containing directory update reaches its required persistence point.

The same-directory rename prevents concurrent readers from seeing a destination that is gradually rewritten. The persistence steps address the separate crash question.

If a crash happens before rename, the old destination remains the intended visible version. If it happens after a fully persistent replacement protocol, the new version survives.

Exact synchronization calls and guarantees vary by operating system and file system. In particular, persisting a file's contents and persisting the directory entry that names it are separate responsibilities.

This pattern gives one file atomic replacement. Updating several files as one application transaction needs a broader application-level protocol.

Incomplete-Tail Recovery in Append-Only Formats

An append-only log can make application recovery simpler if each record contains enough framing and validation information:

On restart, the application scans records:

This does not make every append automatically durable or atomic. It makes an interrupted tail detectable so the application can return to the last complete record boundary.

Database write-ahead logs apply the same broader idea with transaction records and ordering rules: preserve enough evidence to decide which higher-level changes are complete.

The file system's own consistency protocol and the application's log solve problems at different abstraction layers.

RAID vs. Backup Responsibilities

RAID can keep data available after some device failures. It does not automatically make a multi-block file-system update atomic.

If an inconsistent sequence is written successfully to every mirror, all mirrors contain the same inconsistency.

Backups and replication also serve different purposes. They can recover older versions or survive device loss, but they do not prevent the active file system from requiring crash recovery.

Reliable systems often need several of these layers together.

Inspecting a File-System Image Safely

It is safer to inspect consistency tooling on a disposable image than on a live mounted file system.

On a Linux system with ext4 tools installed:

image is a newly created regular file in a temporary directory. Verify that variable before any mkfs command; never substitute a real device containing needed data.

Inspect high-level metadata:

Run a forced, read-only consistency check:

-f requests a check even when the image is marked clean. -n answers no to repairs, leaving the image unchanged.

A newly created image should report a clean structure. This exercise does not simulate a crash; it shows that a checker reasons about superblock state, allocation metadata, inode relationships, and directory structure rather than parsing application file formats.

Never run a repairing checker against a mounted production file system unless the file-system-specific documentation explicitly supports that operation and the maintenance plan accounts for active writers.

Designing a Real Crash Test

Killing a writer process is not a reliable machine-crash simulation because the kernel remains alive and can continue write-back.

A meaningful test needs a controlled environment that can stop persistence at chosen points, such as:

  • A disposable virtual machine terminated without an orderly shutdown
  • A fault-injection block device that drops or reorders selected writes
  • A file-system test harness with explicit crash points
  • A model checker that enumerates allowed persistence subsets

The test must also define what counts as correct:

Without a stated failure model and acceptable outcomes, repeatedly pulling power produces anecdotes rather than a useful correctness test.

Summary

Crash consistency is the requirement that file-system structures remain valid after an abrupt machine stop interrupts a multi-write operation. Process termination is different because the kernel remains available to finish accepted work.

File-system invariants connect directory entries, inodes, link counts, block mappings, and free-space records. Create, append, unlink, and rename can each update several persistent blocks, so a crash can expose dangerous mixtures unless ordering or transactional protocols constrain recovery.

Ordering, atomicity, and durability are distinct. A recovered file system can be consistent while losing recent updates, and it can preserve valid metadata while an application file contains an incomplete logical record.

Full consistency checking can reconstruct metadata relationships but can be slow and cannot infer application intent. Ordered updates, write-ahead journaling, and copy-on-write publication reduce ambiguity in different ways. Applications still need their own safe-replacement or logging protocols for application-level consistency.

The central mental model is:

After any crash point, recovery must choose or reconstruct an allowed state whose persistent references and allocation metadata agree.

Quiz

Crash Consistency Quiz

5 quizzes