Core Building Blocks for System Design Interviews

In this chapter, we'll cover the core concepts and building blocks you will use most often in system design interviews. For each concept, we'll look at the problem it solves, when to use it, and the trade-offs to consider.

In a system design interview, these concepts usually come up when you're trying to meet a specific requirement or constraint.

The interviewer might not directly ask you to explain replication or sharding. Instead, they might say the system needs to handle 100,000 reads per second. That could lead you to consider caching or read replicas.

The important skill is knowing when to use each concept and understanding the trade-offs involved:

So, as we go through these concepts, try to understand not just how they work, but what problems they solve and when you would use them in a real system.

Let's start with scaling. As traffic grows, a single machine may eventually run out of CPU, memory, or storage. There are two main ways to handle this.

Vertical Scaling

Vertical scaling means upgrading to a more powerful machine with more resources.

It's often the simplest option because it requires few changes to your application. However, there's a limit to how much you can upgrade a single machine, and it can still become a single point of failure.

Horizontal Scaling

Horizontal scaling means adding more machines to distribute the workload.

This works best when any application server can handle any incoming request. That's why application servers are typically designed to be stateless. They don't store user sessions or other important state locally. Instead, they keep that data in a shared database or cache.

This makes it easier to add servers when traffic increases or redirect new requests if a server fails.

When you have multiple application servers running, you need a way to distribute incoming requests among them. This is the job of a load balancer.

Layer 4 vs Layer 7

Load balancers typically operate at layer 4 or layer 7 of the network stack.

A layer 4 load balancer routes traffic using information like IP addresses and ports, without looking at the application request itself. This makes routing relatively simple and efficient.

A layer 7 load balancer understands HTTP and can make routing decisions based on URL paths, headers, or cookies. For example, it can send requests to the payments API to one group of servers while directing other requests elsewhere.

Load Balancing Algorithms

The load balancer also needs an algorithm to decide which server receives each request. Common algorithms include round robin and least connections.

Health Checks

To handle failures, load balancers use health checks to identify unhealthy servers. If a server fails its checks, the load balancer stops sending it new traffic until it recovers.

Finally, the load balancer itself needs to be highly available. Otherwise, it becomes another single point of failure. You can address this by running multiple load balancers or using a managed service that handles redundancy for you.

Scaling is only one part of system design. We also need to make sure the system remains available when something fails.

Availability is usually measured as the percentage of time a service is operational.

Achieving higher availability typically requires more infrastructure and operational effort, so it's important to understand what the system actually needs. A payment service, for example, might have much stricter availability requirements than an internal analytics dashboard.

Redundancy

One common way to improve availability is through redundancy. The idea is to identify components whose failure could bring down the service and make sure there's another instance that can take over.

Availability Zones

We can improve availability further by distributing these components across multiple availability zones. These are physically separate locations within a cloud region, designed to reduce the risk of a single failure affecting the entire system.

If one zone goes down, the system can continue operating from the remaining zones, provided they have enough capacity to handle the traffic.

As you design a system, it's useful to go through each component and ask what would happen if it failed. Would the system continue working, or would the entire service become unavailable? This simple exercise helps you identify single points of failure and dependencies you might have overlooked.

Once the system can handle failures, another common concern is improving read performance. One of the most effective ways to do this is caching, which means storing frequently accessed data somewhere faster, usually in memory.

A cache can exist at multiple layers: in the browser, at a CDN, inside an application server, or in a shared caching service like Redis.

Cache-Aside

One of the most commonly used caching patterns is called cache-aside.

When a request comes in, the application first checks the cache. If the data is available, it returns it directly. Otherwise, it fetches the data from the database, stores a copy in the cache, and then returns it.

This works particularly well for data that's read frequently but doesn't change very often, like user profiles, product details, and popular posts.

Keeping the Cache Up to Date

The main challenge with caching is keeping the data up to date. There are two common approaches:

One important factor in database performance is how efficiently the database can find the data your application needs. This is where indexing comes in.

Without a suitable index, the database may have to scan every row in a table to find matching records, which can become very slow as the table grows.

An index helps solve this problem by maintaining a separate data structure that allows the database to locate records more efficiently.

One of the most common index structures used in relational databases is a B-tree, which keeps values sorted and allows the database to find records in just a few steps, even when a table contains millions of rows.

In a system design interview, focus on the queries your application will perform most frequently and explain which indexes would help make those queries efficient.

Indexes help individual queries run faster. But as read traffic grows, you may also need to increase the database's read capacity. One common approach is replication, which means maintaining copies of the same data across multiple machines.

Leader-Follower Replication

A common replication pattern is leader-follower replication. All writes go to the leader, which then sends those changes to its followers.

The followers can serve read requests, reducing the load on the leader. If the leader fails, one of the followers can also be promoted to take its place.

Synchronous vs Asynchronous Replication

Replication can be synchronous or asynchronous.

With synchronous replication, the leader waits for the required followers to confirm they've stored a change before reporting that the write succeeded. This improves durability because the write exists on multiple machines when it's acknowledged. However, waiting adds write latency. If too few followers are available, writes may be delayed or fail.

With asynchronous replication, the leader reports success without waiting for its followers. This reduces write latency, but followers may temporarily fall behind and return older data. That delay is called replication lag. If the leader fails before its latest changes reach the followers, some recently acknowledged writes could be lost during failover.

In a replication setup, every replica still stores the entire dataset, and all writes go through a single leader. If the data grows too large or write traffic becomes too high for one machine, we may need to split the data across multiple machines.

This is called sharding. Each machine stores and manages a portion of the dataset, so both storage and workload are spread across servers.

Choosing a Shard Key

The key decision is choosing a good shard key, which determines where each record is stored. Ideally, the shard key should distribute data and traffic evenly while keeping related data together.

For example, in a chat application, conversation ID could be a good shard key. All messages belonging to the same conversation would be stored on the same shard, making it easier to fetch the conversation history without querying multiple machines.

Choosing the wrong shard key can lead to uneven traffic distribution. For example, sharding users by country might put most of your users on just one shard.

The Cost of Sharding

Sharding also introduces additional complexity:

In a system design interview, don't introduce sharding just because the system has a lot of data. First, consider whether better queries, indexes, caching, or read replicas can handle the workload. Sharding becomes useful when you need to scale storage or write capacity beyond what a single database machine can reasonably support.

Once you've chosen a shard key, the next step is deciding how to map each key to a shard.

One simple approach is to hash the key and take the result modulo the number of shards:

This works, but there's a problem when you add or remove shards. For example, if you go from 4 shards to 5, most keys will map to different shards, which means you may have to move most of your data around.

The Hash Ring

Consistent hashing helps reduce this problem. The idea is to arrange hash values in a circle, often called a hash ring. Both shards and keys are assigned positions on this ring. Each key is stored on the first shard you encounter when moving clockwise from the key's position.

Moving clockwise around this ring, key 1 is stored on Shard B, keys 2 and 3 on Shard C, and key 4 on Shard A.

Now, suppose we add a new shard. Instead of redistributing most of the data, the new shard only takes over the keys between its position and the previous shard on the ring. Everything else stays where it is. With an even distribution, adding a fifth shard would move roughly one-fifth of the data.

Virtual Nodes

Assigning just one position to each shard can lead to uneven distribution. To solve this, we use virtual nodes, where each physical shard gets multiple positions on the ring. This helps distribute data more evenly, and makes it easier to spread the workload when a machine is added or removed.

Consistent hashing is used in systems like Cassandra, Amazon's original Dynamo, and distributed caches. It's particularly useful when machines frequently join or leave the cluster, because it minimizes data movement.

Most databases are meant to store application records, such as user profiles and orders. But for larger files like images and videos, we need a different approach to storing and delivering content.

Object Storage

Object storage is built for this. Services like Amazon S3 store files as objects, each identified by a unique key. Object storage is designed to handle large amounts of data reliably, and is generally more cost-effective than storing large files directly in a database.

Instead of storing the files themselves, the database can store metadata such as the object's key, owner, and size.

Pre-Signed URLs

For uploads, we can also allow clients to send files directly to object storage using pre-signed URLs. The application generates a temporary URL that gives the client permission to upload a file to a specific object key.

This way, the file doesn't need to pass through our application servers, which reduces their bandwidth and processing load.

Content Delivery Networks

For downloads, we can use a content delivery network, or CDN, to serve content faster. A CDN caches files at locations around the world, closer to users.

For example, if a user in Tokyo requests an image that's already cached at a nearby CDN location, it can be served directly from there, instead of being fetched from an origin server in another region. This reduces latency and traffic to the origin.

CDNs are especially useful when many users access the same content, such as popular images, videos, or other static files.

So far, we've focused on operations that happen while a request is being processed. But not every task needs to finish before we respond to the user. Some work can be handled asynchronously in the background.

Consider a video upload on a platform like YouTube. The video needs to be converted into multiple resolutions to support different devices. This is called transcoding, and it can take several minutes.

Instead of making the user wait, the service can store the uploaded video in object storage, add a processing job to a queue, and respond that the video is being processed. Background workers then pick up the job from the queue and handle the transcoding.

This approach has a few advantages:

Let's bring these concepts together with an example. Suppose we're designing a URL shortener. The main operation is simple: given a short code, find the original URL and redirect the user.

Let's say the system needs to handle 10,000 new links and 1 million redirects per second at peak. That's roughly 100 reads for every write, so optimizing read performance is especially important.

Notice how each decision addresses a specific requirement:

This is how you should approach a system design interview. Rather than adding components simply because they're commonly used, think about the requirements, identify the problems you need to solve, and explain why each component belongs in your design.

These concepts give you a solid foundation for approaching most system design problems.

In a system design interview, the goal isn't to use as many of these concepts as possible:

In the next chapter, we'll map these concepts to the technologies engineers use, and look at how to justify those choices in an interview.