Premium Video

This video is available to premium subscribers only

Unlock Full Access

In this chapter, we'll cover the networking concepts you need for system design interviews. We'll follow a request from a user's device to backend servers and see how these concepts guide your design decisions along the way.

When designing a system, you need to consider how much time requests spend traveling over the network.

For example, a request that crosses an ocean will take longer than one that stays within a data center, no matter how fast your application code runs. This affects where you place your servers and store your data.

To understand it better, let's follow a request from a user's device to the server and see what happens along the way.

Imagine a user opens a photo-sharing app and waits for their feed to load.

Each of these steps takes time, and each introduces potential points of failure. Understanding how a request moves through the system helps us identify where delays can occur and how techniques like caching and connection reuse can improve performance.

Let's look at each step in more detail, starting with DNS.

DNS translates a domain name, such as api.example.com, into an IP address.

If the device doesn't already have the address cached, it typically contacts a recursive DNS resolver, which might be operated by an internet provider or a public DNS service. If the resolver doesn't have the answer cached either, it follows the DNS hierarchy to find the domain's authoritative name server and retrieve the DNS record.

Each DNS record has a time to live, or TTL, which specifies how long the result can be cached. This avoids repeating the entire lookup process for every request and helps reduce latency.

Routing with DNS

DNS can also help direct traffic to different servers or regions. This is often called DNS load balancing.

However, one important limitation is that DNS changes don't take effect immediately. If a region goes down, some clients may continue using its cached IP address until the TTL expires.

Once the client has an IP address, it connects to a specific port on that address. The port determines which service should receive the traffic. For example, HTTPS uses port 443 by default.

In a typical cloud architecture, not every server is directly accessible from the internet. Instead, a public load balancer accepts incoming requests and forwards them to application servers running on private IP addresses within a network, such as an AWS VPC. Databases are also usually kept private so they can't be accessed directly from the internet.

However, private servers may still need to communicate with external services, such as a payment provider. For outbound IPv4 connections, a common solution is a NAT gateway. It allows servers in private subnets to initiate connections to the internet without making them directly accessible to unsolicited incoming traffic.

Once the client has an IP address and a port, it can start communicating with the server. Most network communication relies on one of two transport protocols: TCP or UDP.

TCP

TCP provides reliable, ordered delivery of data.

However, this reliability comes with a trade-off. If a packet is lost, later data may have to wait until the missing data arrives. This is known as head-of-line blocking.

UDP

UDP takes a different approach. It sends individual packets, called datagrams, without first establishing a connection. It doesn't automatically retransmit lost packets or guarantee that data will arrive in the correct order, or even arrive at all.

This makes UDP useful for applications where receiving the latest data is more important than recovering older data. For example, during a video call, it's often better to skip a lost frame than wait for it to be retransmitted, which could cause the call to fall behind.

Real-time video, online games, and traditional DNS queries commonly use UDP. Applications can also implement their own reliability and congestion control mechanisms on top of it when needed.

For most APIs, database connections, and messaging systems, TCP is a good default because reliable delivery is important. But for real-time applications where waiting for lost data can introduce noticeable delays, UDP-based protocols may be a better fit.

Establishing a new connection adds latency before the client can even send its first request. How long this takes depends on the round-trip time between the client and server.

This means establishing a new HTTPS connection using TCP and TLS 1.3 takes about 2 round trips before the first HTTP request can be sent.

For example, if the round-trip time between the client and server is 100 milliseconds, that's roughly 200 milliseconds spent just establishing the connection, even before accounting for DNS lookups or application processing.

Reducing Connection Overhead

So, when optimizing network performance, it's worth considering not just how quickly the server processes requests, but also how much time is spent establishing connections in the first place.

TLS is what makes HTTPS connections secure. It encrypts data as it travels over the network, protects it from tampering, and allows the client to verify the server's identity using a trusted certificate.

TLS Termination

In many systems, TLS connections terminate at a load balancer or reverse proxy. This means the load balancer handles the TLS handshake, manages certificates, decrypts incoming requests, and forwards them to the appropriate application servers. It also allows the load balancer to inspect HTTP paths and headers to make routing decisions.

Handling TLS at this layer simplifies certificate management and reduces the encryption work for application servers, although modern hardware can handle encryption quite efficiently.

However, terminating TLS at the load balancer doesn't mean traffic to the backend has to be unencrypted. The load balancer can establish a separate TLS connection to the application servers to protect that traffic as well.

Internal services can also use mutual TLS, or mTLS, where both sides verify each other's identities before communicating.

HTTP defines how clients and servers exchange requests and responses, but different versions of HTTP handle connections differently.

HTTP/1.1

With HTTP/1.1, clients typically wait for a response before sending another request over the same connection. Although HTTP/1.1 supports pipelining, responses must still arrive in order. Because of this, browsers often open multiple connections to fetch resources in parallel.

HTTP/2

HTTP/2 improves on this by introducing multiplexing, which allows multiple requests and responses to share a single connection at the same time. It also compresses headers to reduce the amount of repeated data being transferred.

However, HTTP/2 still runs over TCP, which introduces a limitation. If a packet is lost, all streams sharing that connection may be delayed while TCP recovers the missing data. This is another example of head-of-line blocking.

HTTP/3

HTTP/3 addresses this problem by using QUIC, a transport protocol built on top of UDP.

In a system design interview, you rarely need to choose a specific HTTP version. However, understanding these differences helps you explain why technologies like gRPC commonly use HTTP/2, and why HTTP/3 can offer better performance on networks with packet loss.

Two important factors that affect network performance are latency and bandwidth.

High bandwidth doesn't necessarily make small requests faster, because the data still needs time to travel between the client and server.

Distance

One of the biggest factors affecting latency is physical distance. Light travels through optical fiber at roughly 200 kilometers per millisecond.

For example, New York and London are about 5,600 kilometers apart, so even with a direct fiber connection, a round trip would take at least 56 milliseconds. In practice, network routes are longer, and additional processing introduces further delays.

For rough estimates:

Sequential Calls

These delays become especially important when services make multiple network calls in sequence. For example, if a service makes 10 sequential calls, each over a network path with a 100-millisecond round-trip time, that's roughly 1 second spent waiting on the network alone.

To reduce this overhead, we can:

Large Transfers

For large data transfers, such as video streaming, file uploads, and cross-region data replication, bandwidth becomes more important because it determines how much data the network can carry at once. The actual transfer speed, or throughput, depends on factors like network congestion and protocol overhead.

Moving large amounts of data can also increase cloud networking costs, especially when transferring data between regions. So, when designing systems that handle large amounts of data, we need to consider both transfer speed and cost.

Now that we've covered connections and latency, let's look at how incoming traffic is handled.

A load balancer distributes traffic across multiple servers, allowing you to scale your application and avoid sending new requests to unhealthy instances.

There are two common types of load balancers: layer 4 and layer 7.

Routing Algorithms

Load balancers use different algorithms to decide which server should handle traffic, including round robin, least connections, and hashing.

The information available for routing depends on the layer. For example, layer 4 can use client IP addresses, while routing based on user IDs in HTTP headers requires layer 7 inspection.

Health Checks

Load balancers also perform health checks to detect unavailable servers. They might periodically establish a connection or call a health endpoint, and stop routing traffic to a server after repeated failures.

In most system design interviews involving HTTP APIs, a layer 7 load balancer in front of stateless application servers is a good starting point.

Load balancers are often confused with two related components: reverse proxies and API gateways.

Reverse Proxies

A reverse proxy sits in front of backend servers and handles incoming requests on their behalf.

Depending on its configuration, it can terminate TLS connections, compress or cache responses, serve static files, and forward requests to the appropriate servers. Many layer 7 load balancers also provide these capabilities, so there's considerable overlap between the two.

API Gateways

An API gateway serves a similar purpose but focuses specifically on managing API requests. In a microservices architecture, it provides a common entry point and routes incoming requests to the appropriate services.

It can also handle shared responsibilities such as validating authentication tokens, enforcing rate limits, and logging requests. This avoids having to implement the same functionality separately in every service.

However, because an API gateway handles traffic for multiple services, it needs enough capacity to handle the expected load. It should also be deployed with redundancy so that a single failure doesn't make the entire application unavailable. Like any additional component in the request path, it also introduces some processing overhead and latency.

A content delivery network, or CDN, is a network of servers distributed across different locations around the world, commonly called edge locations or points of presence. Its main purpose is to serve content closer to users, reducing latency and the load on your application servers.

When a user requests an image or video segment, a nearby CDN server first checks whether the content is available in its cache. If it has a fresh copy, it serves the content directly. Otherwise, it fetches the content from the origin server, such as object storage or an application server, and caches it if allowed.

This reduces the distance data needs to travel and avoids repeatedly fetching the same content from the origin. It's especially useful when thousands or millions of users are accessing the same files.

Controlling the Cache

Caching behavior is controlled through HTTP headers like Cache-Control and the CDN's configuration.

For static assets, a common practice is to include a version number or content hash in the filename. This allows files to be cached for a long time because any update generates a new URL rather than replacing the existing file.

CDNs are commonly used to deliver images, videos, JavaScript files, and stylesheets.

Dynamic Requests

CDNs can also help with dynamic requests that cannot be cached. In this case, the CDN forwards the request to the application server, but a nearby edge can handle the initial connection and reuse existing connections to the origin. This reduces connection setup overhead and can improve performance, even when the response itself isn't cached.

When your application runs across multiple regions, you need a way to direct users to the right location. Two common approaches are geographic DNS and anycast.

Geographic DNS

Geographic DNS directs users to different servers based on their estimated location. For example, a user in India might be directed to a server in Mumbai, while a user in the US might be directed to a server in Virginia.

DNS providers usually estimate the user's location based on the IP address of the DNS server handling the lookup, rather than the user's actual IP address. Some DNS servers also share information about the user's network, which helps the provider make a more accurate routing decision.

This approach is useful, but it has some limitations. Location estimates aren't always accurate, and DNS responses are cached. If a region goes down, some users may continue connecting to it until their cached DNS records expire.

Anycast

Another approach is anycast, where servers in multiple locations advertise the same IP address. When a user connects to that address, internet routing determines which location receives the traffic, often choosing one nearby. However, the choice depends on network routing policies, so it isn't always the closest or fastest location.

One advantage of anycast is that if a location becomes unavailable and stops advertising its route, traffic can be redirected to another location without waiting for DNS caches to expire. However, routing changes still take time, and existing connections may be interrupted.

Anycast is commonly used by CDNs and DNS providers.

In practice, many global systems use geographic DNS or anycast to direct users to a nearby edge location. That edge then forwards requests to the appropriate application region.

Within a system, services also need a way to find and communicate with each other. This is where service discovery comes in.

For example, an order service might need to call a payment service, but the payment service instances can change as they scale up, scale down, or restart. Hard-coding their IP addresses would make the system difficult to maintain.

There are two common solutions:

Every network call can fail or take longer than expected, which is why setting timeouts is important.

Timeouts

Without proper timeouts, requests waiting on a slow dependency can continue holding resources like threads, memory, and connections. Over time, this can exhaust the calling service's resources, causing it to become slow or unavailable as well.

Timeouts should be based on expected response times and the overall request deadline. For example, if a client expects a response within 2 seconds, it doesn't make sense for a downstream service call to wait for 30 seconds. Passing a deadline through the call chain helps each service work within the remaining time.

Retries

Retries can help recover from temporary failures, but they also add more load to the system.

For example, if 3 layers each make up to 3 attempts, a single request could result in 27 calls to the final dependency. Retrying at one carefully chosen layer is usually easier to control.

Understanding networking helps you explain how your system performs, where delays can occur, and what happens when something fails.

In a system design interview, the key is to connect your networking decisions to the system's requirements. Think about:

These considerations will help you make practical design decisions and explain the trade-offs involved.