Reading from a database or calling another service takes time. Under heavy traffic, that cost adds up quickly.

A cache is a fast storage layer that keeps copies of frequently used data. When the application needs that data again, it can read from the cache instead of going all the way back to a slower system.

The idea is simple, but the payoff is large. Used well, caching can make requests faster, reduce database load, lower cost, and help the system survive traffic spikes.

The tricky part is that a cache stores a copy. If the real data changes, the cached copy can become old.

This chapter covers what a cache is, where caches live in a system, what makes data a good fit for caching, and the trade-offs that come with every cached copy.

Consider a social media application. When a user opens their feed, the application must:

Without caching, a single feed load might trigger 50+ database queries. Multiply that by millions of users, and the database becomes the bottleneck.

Example Performance Impact:

The Shape of a Cache

At its core, a cache is usually a key-value store built for fast lookups. The application gives the cache a key, and the cache returns the value stored under that key.

Key Parts

Keys are unique names for cached data. A good key is predictable: the same request should always produce the same key. It should also describe what it stores, and different data should not accidentally share the same key.

Values are the cached data itself. A value might be JSON, a string, binary data, an HTML fragment, or a precomputed result like a count or summary.

Metadata is extra information about the cached entry, such as when it was created, when it expires, how often it has been used, and how large it is.

Caching happens at multiple levels in a system. Each layer has a different job:

Browser Cache

The browser cache is the closest cache to the user. It stores static assets and sometimes API responses on the user's device. If the browser already has a valid copy, it may not need to make a network request at all.

Behavior is controlled through HTTP headers like Cache-Control and ETag, which makes browser caching a natural fit for images, scripts, stylesheets, and data that does not change often.

CDN Cache

Content Delivery Networks, or CDNs, cache content at edge locations around the world. Users can download content from a nearby edge instead of from your main servers.

CDNs work best for static content, media files, public pages, and other content that many users can share.

Application Cache

The application cache lives in memory inside the application process itself. It is extremely fast because the data is already inside the running application.

The trade-off is that each application instance has its own local cache. That can duplicate memory and create different cached values on different instances.

Distributed Cache

A distributed cache is a separate caching service shared by all application instances. Redis and Memcached are common choices.

Lookups are slower than local memory because they cross the network, but the cache is shared and can hold much more data. Distributed caches are commonly used for shared state, session storage, and database query results.

Database Buffer Pool

Databases also keep their own internal cache. They store frequently accessed database pages in memory on the database server, so repeated reads do not always need to hit disk.

This layer is usually invisible to application code. The database manages it automatically.

When the application asks the cache for data, two things can happen:

Cache Hit: The data exists in the cache. Return it immediately.

Cache Miss: The data is not in the cache. Fetch it from the database or another source, optionally store it in the cache, then return it.

Cache Hit Ratio

The hit ratio is the percentage of cache lookups served from the cache:

Once you have that number, this table gives a rough sense of where it stands:

A 90% hit ratio means 90% of requests avoid the database for that data. If your database can handle 10,000 queries per second for those reads, a 90% hit ratio can let the system handle roughly 100,000 read requests per second.

Not all data benefits equally from caching.

Caching pays off the most when the same data is read many times, when the data is expensive to compute, when the source is slow, or when the data changes rarely.

It is a poor fit when data changes constantly, when every request needs different data, when objects are too large, or when even a little stale data is unacceptable.

The 80/20 Rule

In most applications, 20% of the data serves 80% of the requests. Focus caching efforts on that hot 20%:

Caching that hot slice captures most of the benefit while keeping memory use small. Rarely requested keys usually do not repeat enough to be worth caching aggressively.

Cache consistency means keeping cached data reasonably aligned with the real data. When the database changes, the cache can become stale.

Consistency Approaches

There are a few common ways to keep cached data fresh enough:

The right approach depends on the feature. A product description being stale for 5 minutes might be fine. A bank account balance being stale for 5 seconds is not.

Caching has costs as well as benefits. The patterns below often cause more problems than they solve.

Cache Everything

Blindly caching everything fills memory with data that may never be read again. The cache becomes larger, more expensive, and often less useful.

Infinite TTL

Data that never expires can stay wrong forever. If one cache delete is missed, users may keep seeing old data until someone notices and fixes it.

Cache as Primary Storage

Treating the cache as the real data store is dangerous. Caches can evict data, restart, or fail. If the database was never updated, that data may be gone.

The Thundering Herd

When a popular cache entry expires, many requests can hit the database at the same time:

Common fixes include per-key locking, early refresh before expiration, and background refresh.

In distributed systems, caching has a few extra problems because there are many servers and cache nodes.

Consistency Across Nodes

When multiple application servers or cache nodes store copies of the same data:

Cache entries may need to be deleted or refreshed in several places. Some nodes may see the new value before others.

Data Partitioning

Large caches split data across multiple nodes. A technique called consistent hashing helps avoid moving too many keys when nodes are added or removed:

Failure Handling

What happens when cache is unavailable?

Picking the wrong mode can turn a cache outage into a full outage. Decide in advance whether the database can safely handle traffic without the cache.

The key metrics to track are hit ratio, cache latency, memory usage, eviction rate, and miss latency.

Hit ratio tells you whether the cache is useful. Cache latency tells you whether cache lookups are fast. Memory usage and evictions tell you whether the cache is too small. Miss latency tells you how expensive it is when requests fall through to the database.

In latency metrics, p50 means the typical request, and p99 means one of the slowest requests users still regularly see.

A sudden drop in hit ratio or spike in evictions is a warning sign. Traffic may have changed, the cache may be too small, or keys may be getting deleted too aggressively.

Caching stores frequently accessed data in a faster layer to reduce latency and database load. Caches exist across the stack: browser, CDN, application, distributed cache, and database buffer pool.

Hit ratio is one of the most important metrics to watch. A 90% hit ratio can give roughly a 10x improvement for the reads being cached.

What gets cached matters more than how much gets cached. Read-heavy, expensive-to-compute, and stable data benefit the most. Write-heavy data and one-off requests usually benefit less.

Common anti-patterns include caching everything, setting infinite TTLs, treating the cache as the real data store, and ignoring stampedes when popular entries expire.

Understanding what caching is and why it matters sets the stage for the next question: how exactly should an application interact with the cache?

That brings us to caching patterns, starting with the most common one, the cache-aside pattern, where the application explicitly manages what goes in and out of the cache.