Latency vs Throughput vs Bandwidth
Practice this topic in a realistic system design interview
When discussing system performance, three terms come up repeatedly: latency, throughput, and bandwidth. These concepts are often confused or used interchangeably, but they measure fundamentally different things.
Understanding these metrics is crucial for:
- Diagnosing performance bottlenecks
- Making informed architectural decisions
- Setting realistic expectations with stakeholders
- Answering system design interview questions
This chapter walks through each concept, the relationships between them, and which metric matters most in different situations.
The Highway Analogy
A simple analogy makes the distinctions concrete. Think of a highway connecting two exits:
Loading simulation...
- Bandwidth is the number of lanes on the highway. More lanes mean more cars can travel simultaneously.
- Throughput is how many cars actually pass through per hour. This depends on traffic conditions, not just the number of lanes.
- Latency is the time it takes for a single car to travel from one Exit 1 to Exit 10.
A highway might have 4 lanes (high bandwidth), but if there is an accident, only 100 cars per hour pass through (low throughput). Meanwhile, each car might take 2 hours to complete the journey (high latency).
This analogy helps explain why these metrics do not always move together.
Latency
Latency is the time it takes for a single request to travel from source to destination and back. It measures delay.
In networking, latency is often called round-trip time (RTT), the time from sending a request to receiving a response.
Components of Latency
Latency is not a single value. It is the sum of multiple delays:
- Propagation delay: Time for signals to travel through the medium. Light in fiber travels at ~200,000 km/s. A cross-Atlantic request (6,000 km) takes ~30ms just for propagation.
- Transmission delay: Time to push bits onto the wire. Depends on packet size and link bandwidth.
- Processing delay: Time for routers, load balancers, and servers to process packets.
- Queuing delay: Time spent waiting in buffers when components are busy.
Measuring Latency
Latency is typically measured using percentiles:
Why percentiles matter: Average latency hides outliers. A system with 10ms average might have p99 of 500ms, meaning 1% of users experience terrible performance.
What Affects Latency?
Reducing Latency
- Use CDNs: Serve content from edge locations closer to users
- Caching: Eliminate round trips by caching at multiple layers
- Connection pooling: Avoid repeated connection setup
- Database optimization: Add indexes, optimize queries
- Geographic distribution: Deploy servers closer to users
- Protocol optimization: Use HTTP/2, HTTP/3 (QUIC)
Throughput
Throughput is the amount of work completed per unit of time. It measures volume.
For web systems, throughput is often expressed as requests per second (RPS) or transactions per second (TPS).
Throughput vs Bandwidth
A common confusion: bandwidth is theoretical maximum capacity, while throughput is actual achieved rate.
You can never have throughput higher than bandwidth, but throughput is almost always lower due to:
- Protocol overhead (headers, acknowledgments)
- Congestion and packet loss
- Processing limitations
- Inefficient resource utilization
Calculating Throughput
For a single-threaded system:
For a multi-threaded system:
What Limits Throughput?
The bottleneck determines maximum throughput. A system is only as fast as its slowest component.
Improving Throughput
- Horizontal scaling: Add more servers
- Vertical scaling: Add more CPU, memory
- Async processing: Do not block on slow operations
- Batching: Process multiple items together
- Caching: Reduce work by reusing results
- Connection pooling: Reuse expensive connections
- Load balancing: Distribute work evenly
Bandwidth
Bandwidth is the maximum rate at which data can be transferred. It measures capacity.
Bandwidth is typically expressed in bits per second (bps): Kbps, Mbps, Gbps.
Types of Bandwidth
Bandwidth-Delay Product
An important concept that connects bandwidth and latency:
BDP represents how much data can be "in flight" at any moment.
Example:
- Bandwidth: 1 Gbps = 125 MB/s
- Latency: 100ms (coast-to-coast US)
- BDP: 125 MB/s × 0.1s = 12.5 MB
This means 12.5 MB of data can be traveling through the pipe at any instant. If your TCP window size is smaller than BDP, you will not fully utilize available bandwidth.
Summary
Latency, throughput, and bandwidth measure different aspects of system performance. Confusing them leads to optimizing the wrong thing.
Key takeaways:
- Latency is the delay for a single request. Measure it with percentiles (p50, p95, p99), not averages, because averages hide the slow tail.
- Throughput is the volume of work completed per unit time. It is capped by the slowest component in the path, not the headline capacity number.
- Bandwidth is the theoretical maximum data transfer rate. Achieved throughput is almost always lower because of protocol overhead, congestion, and processing limits.
- Latency and throughput often trade off. Batching, queuing, and pipelining raise throughput but add latency to individual requests.
- Little's Law connects them: concurrency = throughput x latency. To raise throughput, either raise concurrency or lower per-request latency.
- The bandwidth-delay product sets the in-flight data ceiling. A TCP window smaller than BDP leaves bandwidth on the table even when the link is fast.
- Lower latency with caching, CDNs, geographic placement, and protocol upgrades like HTTP/2 and HTTP/3. Round trips dominate at distance.
- Raise throughput with horizontal scaling, async processing, batching, and connection pooling. Find the bottleneck first; widening anything else has no effect.
A system can have high bandwidth and low throughput, or low latency and low throughput. The metrics describe different bottlenecks. The first step in performance work is naming which one applies.
Quiz
Latency vs Throughput vs Bandwidth Quiz
10 quizzes