Premium Video

This video is available to premium subscribers only

Unlock Full Access

The CAP theorem describes what happens to a distributed system when its servers cannot communicate with each other. It is one of the most quoted ideas in distributed systems, and also one of the easiest to explain badly.

This chapter uses an online store running in two regions to explain consistency, availability, and partition tolerance, and the choices we make during a network failure.

Suppose our store keeps a copy of its product data in Region A and another in Region B. Both currently show a product price of 50 dollars.

An administrator changes the price to 60 dollars in Region A, and the system confirms the update. Before that change reaches Region B, the network connection between the regions fails.

Both regions are still running, and customers can reach them. But when a customer asks Region B for the price, it only has the old value of 50 dollars. It cannot contact Region A to find out what changed.

Region B has two options.

It can return 50 dollars from its local copy. The request succeeds, but the customer sees an older price after the update was confirmed.

Or Region B can refuse to answer until it can establish the current value. That avoids a stale answer, but the customer gets an error or has to wait.

In the first case, we keep the request available but give up consistency. In the second, we keep consistency but give up availability for that request.

Region B has no way to discover an update it never received while communication remains broken. No protocol can change that.

This is the situation CAP describes. During a network partition, we cannot guarantee both a consistent view of the data and successful responses to every request at healthy nodes.

The C in CAP stands for consistency. In this context, consistency means the system behaves as if there were only one up-to-date copy of the data.

Suppose a product price is updated from 50 dollars to 60 dollars. Once that update succeeds, any read that starts afterward must return 60 dollars or a newer value. It should not return the old 50 dollar price.

The sequence above shows one way to get there. The write is copied to the second node before the client hears "OK", so a later read from either node returns the new value.

This does not mean every replica must update at exactly the same instant. A replica that is behind can coordinate with other replicas, or simply avoid serving a read until it can return a consistent result.

In our store, a read from Region A after the update returns 60 dollars and is consistent. A read from Region B is consistent only once Region B has caught up. If Region B answers before that, it returns 50 dollars and consistency is broken. If Region B refuses because it cannot confirm it has the latest value, nothing stale was returned, so consistency holds, even though the customer did not get an answer.

CAP Consistency vs ACID Consistency

This is different from the C in ACID. ACID consistency is about keeping database rules and constraints valid. CAP consistency is about making replicated data behave like a single up-to-date copy.

The A in CAP stands for availability. It says every request sent to a healthy node should eventually get a valid response, even if replicas cannot coordinate.

For example, returning an older local price keeps the system available, but gives up strong consistency. Returning a "service unavailable" error does not satisfy CAP availability for that request.

CAP availability does not mean the response has to be fast, only that the system eventually returns a response. It is a narrower idea than the everyday meaning of the word.

A stale value from a healthy node counts as available. A value that arrives after a long delay counts too. An error from a healthy node that could not coordinate does not count. And a monthly uptime figure like 99.99% is a different measurement altogether, because CAP availability is about individual requests during a partition, not about time spent online.

The P in CAP stands for partition tolerance. A partition happens when groups of nodes can no longer communicate with each other, even though the nodes themselves are still running.

This could happen because of a broken network link, a routing issue, or a firewall misconfiguration between regions.

A broken link between data centers leaves both regions running while replication traffic stops. A routing issue can send packets down a path that never arrives. A firewall change can block the port replication uses. An overloaded network device can drop or delay messages until timeouts fire. In every case the nodes are fine, and the problem is the path between them.

Partition tolerance means the system is designed to handle these communication failures. It does not mean every request will still succeed.

Once replicas are separated, the system must decide whether to keep serving requests using local data or reject operations that require coordination.

CAP is often explained as "pick any two out of three," but that is too simplistic.

A distributed system cannot prevent network partitions. When the network is healthy, we can often provide both consistency and availability. The real trade-off appears when communication breaks.

At that point, operations that depend on separated replicas must choose: keep serving requests and risk stale data, or stop and preserve consistency.

And this choice does not have to be the same for every operation. Reading a product description may tolerate stale data, while reserving the last item may require strict coordination.

The labels CP and AP are most useful when we apply them to an operation, not to a whole database.

In CP systems, we preserve consistency during a partition.

Suppose there is only one unit of a product left. A reservation must check the inventory and reduce it as one coordinated operation. If both regions independently see one item available and each confirm a sale, we have sold the same item twice.

A CP system avoids this by requiring reservations to go through a leader or a group of replicas that agrees on the update. If a region cannot reach that authority, it must wait or reject the reservation.

Some customers may not be able to complete checkout during the partition, but the system avoids confirming the same item twice.

Reads follow the same rule. A replica that cannot prove its copy is fresh refuses the read rather than answering with a guess.

Now consider shopping-cart updates. A customer adds a book from their phone through Region A and a notebook from their laptop through Region B.

The regions are disconnected, but we may still want both additions to succeed. An AP design allows each reachable replica to accept its local update and replicate it later.

During the partition, the phone may show only the book and the laptop only the notebook. Once communication returns, the system can merge the additions so both views contain both items.

The merge rule is part of the design. Combining two cart additions is easy. Other conflicts, such as two different edits to the same product description, need a rule for which version wins, and the application has to own that rule.

So an AP system trades a period of drift for uninterrupted service. Each side accepts writes locally, replication waits until the link returns, and the two sides are reconciled afterward.

The last label is CA: consistency and availability.

A single-node database can provide both while it is healthy, because there are no replicas to coordinate across a network. A distributed system can also provide both as long as communication is working.

But once replicas can become separated by a network partition, the system still has to decide what to do. For replicated data during a partition, we come back to the same trade-off: preserve consistency or keep affected operations available.

That is why CA is not a useful label for a distributed system. A design that never says what happens during a partition has not escaped the trade-off. It has left the answer undefined.

Even when the network is working normally, coordination takes time.

Reading from a nearby replica may be faster than contacting a leader in another region, but that local replica could be slightly behind. Similarly, waiting for multiple replicas to agree on a write adds extra latency.

The PACELC theorem extends CAP to cover this normal case. If there is a partition (P), the trade-off is between availability (A) and consistency (C). Else (E), the trade-off is often between latency (L) and consistency (C).

Stronger consistency usually requires more coordination, which can make requests slower. So the right choice depends on what each operation actually needs.

CAP describes what happens to replicated data during a network partition. When replicas cannot communicate, we cannot guarantee both strong consistency and successful responses for every affected request.

The trade-off shows up during the partition, not before it, and it applies per operation. In our store, browsing the catalog and updating the cart can keep going with local data, while reserving the last unit must stop until the regions can coordinate.

Before moving on, think about two operations in an application you know. Which one could safely continue during a partition, and which one should stop to avoid inconsistent data?