Premium Video

This video is available to premium subscribers only

Unlock Full Access

When you send data over a network or save it to disk, you expect to get back exactly the same bytes. Most of the time, that's what happens. But data can sometimes get corrupted during transmission or storage, and the system might not even notice.

That's where checksums come in. They help systems detect accidental changes to data.

In this chapter, we'll explore how data corruption happens, how checksums help detect it, and how different techniques compare, from simple sums to CRCs and cryptographic hashes.

Data can change in several ways between the moment it is written and the moment it is read:

What's especially dangerous is that corruption can be silent. The program doesn't necessarily crash or report an error. It may simply read the wrong bytes and continue as though nothing happened.

One changed bit could turn a financial value into the wrong number or make a compressed file impossible to open.

So we need an efficient way to check whether data has changed between the moment it was created and the moment it's received or read back. That's where checksums come in.

A checksum is a small, fixed-size value calculated from a piece of data.

Before sending a file or storing a block, we run its contents through a checksum algorithm. Suppose the result is four bytes long. We store or transmit those four bytes alongside the original data.

Later, the receiver runs the same algorithm on the data it received and compares the result with the original checksum.

"Likely" is the right word. A checksum is much smaller than the data it covers, so there are far more possible inputs than possible checksum values. Some different inputs must produce the same value. This is called a collision, and it means no fixed-size checksum can guarantee a match is correct. A good algorithm makes it very unlikely that the kinds of errors it was designed for will slip through.

Detection, Not Repair

There's an important distinction here. A checksum detects corruption. It doesn't tell us exactly which bits changed, and it doesn't repair them by itself.

Error-correcting codes take a different approach by adding enough redundancy to recover certain errors. ECC memory and the error correction built into SSDs are examples.

In many distributed systems, detecting a bad copy and retrieving a good one is sufficient. Recovery then comes from somewhere else: a retransmission, another replica, or a backup.

But how reliable that detection is depends on the algorithm we choose.

Let's start with the simplest possible checksum: adding up the bytes.

Suppose we have three bytes with values 10, 20, and 30. Their sum is 60, so we send 60 as the checksum.

If the middle byte changes from 20 to 21, the receiver calculates 61. The values don't match, so we know something went wrong.

But now imagine the bytes are reordered to 30, 20, and 10. The sum is still 60, even though the data has changed.

Or suppose one byte increases by one and another decreases by one. Those changes cancel each other out, leaving the same checksum.

So adding bytes is fast and can catch some errors, but different data can easily produce the same result. More useful checksum algorithms are designed to detect a wider range of accidental changes.

Parity is an even simpler check. A parity bit records whether a group of bits has an even or odd number of 1s. It catches any single flipped bit but misses two flips in the same group. It is still used in some memory and serial links, but it is too weak for files or messages.

One example is the Internet checksum, used by IPv4, TCP, and UDP.

It divides the input into 16-bit words, adds them using a method called one's-complement addition, and flips the bits of the final sum. The result is a 16-bit checksum carried in the relevant protocol header.

In IPv4, the header checksum covers only the IP header, not the data inside the packet. Since routers decrease the TTL field as packets move through the network, they also need to update that checksum at every hop.

TCP and UDP have separate checksums that cover their headers and payloads, along with information from the IP layer, such as the source and destination addresses. This helps detect corruption beyond a single network link.

The Internet checksum was designed to be inexpensive to compute, which was especially important on older hardware. But it's not particularly strong:

IPv6 removed the IP header checksum, avoiding that extra work at each router. Other layers still provide error detection, including the link layer, which often uses a stronger technique called a CRC.

CRC stands for Cyclic Redundancy Check.

Rather than simply adding bytes, a CRC treats the data as a sequence of bits and performs a special form of binary division using a fixed polynomial. The remainder becomes the checksum.

You don't need to work through the math to understand why it's useful. The result depends on both the values of the bits and their positions. So changing the order of bytes is much more likely to change the CRC than it would a simple sum.

With a suitable polynomial, a 32-bit CRC detects:

A burst error happens when a short stretch of neighboring bits is damaged, perhaps by electrical noise on a connection.

For random errors, the chance of an undetected mismatch with CRC-32 is roughly one in four billion (1 / 2^32). That's much better than a simple byte sum.

CRCs are also extremely fast. Network hardware can calculate them directly, and modern processors have instructions for variants such as CRC-32C.

A CRC is not a cryptographic algorithm. It is built to catch accidents, not attacks. Anyone who changes the data on purpose can simply compute a new CRC.

If Ethernet already checks each frame, and TCP checks the data again, does that mean corruption can no longer get through? Not quite.

An Ethernet CRC protects a frame as it travels across a single link. A device that forwards the frame validates that check and sends the data onward with a new frame check value for the next link.

Now imagine the packet is corrupted inside a router's memory, after the incoming frame has been checked but before the outgoing one is generated.

The router can calculate a perfectly valid new CRC over the already-corrupted data. The next device sees a valid frame, with no indication of what happened earlier.

TCP provides an additional end-to-end check, which can catch errors introduced between the sender and receiver. But as we've seen, its 16-bit checksum can miss certain changes. Studies of real network traffic have found cases where corrupted packets passed that check. Even a rare failure matters when you're moving enormous amounts of data.

This illustrates the end-to-end principle: checks at lower layers are useful, but they can't always guarantee that the final application received the data it expected. If correctness is critical, the application may need its own integrity check.

HTTPS adds another layer of protection. Modern TLS uses an authenticated encryption tag to detect changes to the protected data. If verification fails, the record is rejected and the connection is typically terminated.

That raises a question: if cryptographic integrity checks are stronger, why use CRCs at all?

A cryptographic hash function, such as SHA-256, also produces a fixed-size value from data. SHA-256 produces 256 bits, compared with 32 bits for a common CRC.

But the main difference isn't just the length. It's what the algorithm is designed to protect against.

That makes it useful as a fingerprint for content:

Older algorithms such as MD5 and SHA-1 have known collision attacks, so they shouldn't be used when security depends on collision resistance. They can still detect many accidental errors, but they're not appropriate for adversarial integrity checks.

Hashes Don't Prove Who Made the Data

There's another important limitation. A hash only proves that the data matches the hash you're comparing it with. If an attacker can replace both, the comparison still succeeds.

For authenticity, you need a trusted hash value or a mechanism such as an HMAC or digital signature that can verify who produced it:

The Cost

The trade-off is computation. SHA-256 generally costs more to calculate than CRC-32C, which can matter when you're processing gigabytes of data every second.

So which approach should you choose?

How Much Data Each Checksum Covers

Also think about how much data each checksum covers.

If you have one checksum for an entire 10 GB file, a mismatch tells you something is wrong, but not which part. You might have to download the entire file again.

If you checksum smaller chunks independently, you can identify the damaged chunk and replace only that part. This is a common pattern in storage systems.

Smaller chunks make repair more precise, but they also mean more checksums to store and more work to verify.

Storage devices have their own error-detection mechanisms, but some failures can still return incorrect data without raising an error. Bit rot, misdirected writes, and firmware bugs are examples.

That's why storage systems often add checksums at a higher layer.

HDFS

HDFS, the Hadoop Distributed File System, traditionally stores a checksum for each 512-byte chunk of a block.

When a client reads the data, it verifies those checksums. If one fails, it can read another replica and report the corrupt copy so the system can repair its redundancy.

ZFS

ZFS takes a different approach. It stores a block's checksum in the metadata pointer that references the block, rather than alongside the data itself.

This makes it easier to detect cases where the wrong block is returned or data was written to the wrong location. A checksum stored next to the data would still match if the disk returned a valid but wrong block. A checksum stored in the parent pointer would not.

If ZFS detects corruption and has a healthy redundant copy, it can repair the damaged data automatically.

Databases

Databases rely on similar checks. PostgreSQL, for example, includes checksums in its write-ahead log records. After a crash, recovery can use those checks to identify invalid or incomplete records instead of replaying corrupted log data.

Many databases can also checksum their data pages. When a page fails its check, the database reports an error rather than returning the bad data. If the damaged data is needed for recovery, some databases will refuse to start at all. That is deliberate: running on data known to be corrupt can spread the damage to replicas and backups.

Object Storage

Cloud object stores also support checksums. With Amazon S3, a client can provide a checksum during an upload, allowing the service to verify that the received data matches what was sent. If the values differ, the upload is rejected.

Scrubbing

There's another useful practice called scrubbing.

Some data might go unread for months or years. If corruption develops, you don't want to discover it only when the data is urgently needed, or after other replicas have also become damaged.

So storage systems can periodically read stored blocks, verify their checksums, and repair corrupted copies while healthy redundancy is still available. ZFS and many distributed storage systems run scrubs on a schedule. The same idea applies to backups: a backup that is never test-restored hasn't really been verified.

How does all of this translate into system design?

Verify End to End

First, verify data end to end when its integrity matters. Calculate a checksum or hash close to where the data originates, preserve the expected value, and check it where the data is ultimately used.

That gives you coverage across the network, intermediate services, and storage, rather than trusting just one layer.

Decide What Happens on Failure

Second, decide what happens when verification fails. Don't silently accept corrupted data. Depending on the operation, the system might:

It should also record metrics or raise alerts so recurring failures can be investigated. A steady trickle of mismatches from one disk or one network path usually points to failing hardware.

Compare Large Data Sets with Merkle Trees

Third, use hashes to compare large data sets efficiently.

Suppose two replicas each contain millions of records. Sending every record across the network just to check whether they match would be expensive.

Systems such as Cassandra use Merkle trees for this. Each leaf contains a hash representing part of the data, and each parent summarizes the hashes below it.

The replicas compare their root hashes. If they differ, they can work down the tree to find which ranges need to be synchronized, without comparing everything.

Detect Changes with Content Hashes

Finally, use content hashes to detect changes.

Backup and file-sync systems can hash chunks and upload only the chunks that changed.

HTTP servers may use a content-derived ETag so clients can check whether a resource has changed without downloading the entire response again.

Data can be corrupted silently by noise, faulty memory, bit rot, or misdirected writes. A checksum is a small value computed from the data that lets a later reader check whether anything changed. It detects corruption. Recovery comes from retries, replicas, or backups.

The algorithm matters. A simple sum misses reordered bytes and changes that cancel out. The 16-bit Internet checksum used by IPv4, TCP, and UDP is cheap but weak. CRCs depend on bit positions, catch burst errors, and are fast enough for hardware, which makes CRC-32C a good default for accidental corruption.

Lower-layer checks don't cover everything. Data can be corrupted between links, so critical data needs an end-to-end check.

Cryptographic hashes like SHA-256 are for content identity and tamper detection. A hash is only as trustworthy as the expected value, so use HMACs or digital signatures to prove who produced the data.

Storage systems like HDFS, ZFS, PostgreSQL, and S3 add their own checksums, and scrubbing finds corruption in data that is rarely read.

The larger lesson is that checksums aren't just a networking detail. Whenever your system moves, stores, or replicates data, ask three questions: where could corruption happen, how will we detect it, and what will we do when a check fails?