{"title":"Design Load Balancer","description":"","content":"\n> **What is a Load Balancer?**\n>\n> A Load Balancer is a critical infrastructure component that distributes incoming network traffic across multiple servers to ensure no single server becomes overwhelmed.\n>\n> The core idea is straightforward: instead of routing all traffic to one server (which would eventually crash under heavy load), a load balancer acts as a traffic cop, intelligently spreading requests across a pool of healthy servers. This improves application availability, responsiveness, and overall system reliability.\n>\n> **Popular Examples:** AWS Elastic Load Balancer (ELB), NGINX, HAProxy\n\n\nLoad balancers appear in almost every distributed system architecture. Understanding how to design one from scratch demonstrates deep knowledge of networking, high availability, and system scalability.\n\nIn this chapter, we will explore the **high-level design of a Load Balancer**.\n\nLets start by clarifying the requirements.\n\n---\n\n# 1. Clarifying Requirements\n\nBefore starting the design, it's important to ask thoughtful questions to uncover hidden assumptions, clarify ambiguities, and define the system's scope more precisely.\n\nHere is an example of how a discussion between the candidate and the interviewer might unfold:\n\n\n> **DISCUSSION**\n>\n> **Candidate:** \"What type of traffic should the load balancer handle? Are we focusing on HTTP/HTTPS traffic, or should it support any TCP/UDP traffic?\"\n>\n> **Interviewer:** \"Let's design a general-purpose load balancer that supports both Layer 4 (TCP/UDP) and Layer 7 (HTTP/HTTPS) load balancing.\"\n>\n> **Candidate:** \"What is the expected scale? How many requests per second should the system handle?\"\n>\n> **Interviewer:** \"The load balancer should handle up to 1 million requests per second at peak traffic.\"\n>\n> **Candidate:** \"How should we handle server failures? Should the load balancer automatically detect and route around unhealthy servers?\"\n>\n> **Interviewer:** \"Yes, health checking is critical. The system should detect failures within seconds and stop routing traffic to unhealthy servers.\"\n>\n> **Candidate:** \"Do we need to support session persistence, where requests from the same client go to the same backend server?\"\n>\n> **Interviewer:** \"Yes, sticky sessions should be supported for stateful applications, but it should be configurable.\"\n>\n> **Candidate:** \"What about SSL/TLS termination? Should the load balancer handle encryption?\"\n>\n> **Interviewer:** \"Yes, SSL termination at the load balancer is required to offload encryption work from backend servers.\"\n>\n> **Candidate:** \"What availability target should we aim for?\"\n>\n> **Interviewer:** \"The load balancer itself must be highly available with 99.99% uptime, since it's on the critical path for all traffic.\"\n\n\nAfter gathering the details, we can summarize the key system requirements.\n\n\n### Functional Requirements\n\n1. **Traffic Distribution:** Distribute incoming requests across multiple backend servers using configurable algorithms.\n2. **Health Checking:** Continuously monitor backend servers and automatically remove unhealthy ones from the pool.\n3. **Session Persistence:** Support sticky sessions to route requests from the same client to the same server.\n4. **SSL Termination:** Handle SSL/TLS encryption and decryption to offload work from backend servers.\n5. **Layer 4 and Layer 7 Support:** Support both transport-level (TCP/UDP) and application-level (HTTP/HTTPS) load balancing.\n\n### Non-Functional Requirements\n\n1. **High Availability:** The load balancer must be highly available (99.99% uptime) with no single point of failure.\n2. **Low Latency:** Should add minimal latency to requests (< 1ms overhead).\n3. **High Throughput:** Handle up to 1 million requests per second at peak.\n4. **Scalability:** Should scale horizontally to handle increasing traffic.\n5. **Fault Tolerance:** Continue operating even when individual components fail.\n\n\n---\n\n# 2. Back-of-the-Envelope Estimation\n\nBefore diving into the design, let's understand the scale we are working with. These numbers will guide our architectural decisions, particularly around memory management and network capacity.\n\n### 2.1 Traffic Estimates\n\nStarting with the numbers from our requirements:\n\n\n```mermaid\nflowchart LR\n subgraph \"Traffic Profile\"\n direction TB\n Peak[\"Peak Traffic
1M RPS\"]:::red\n Avg[\"Average Traffic
~300K RPS\"]:::orange\n Conn[\"Concurrent Connections
~500K\"]:::primary\n end\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n```\n\n\n#### Request Volume\n\nOur target is 1 million requests per second at peak. Traffic is rarely uniform, so let's assume the average is about one-third of peak:\n\n\n```python\nPeak RPS: 1,000,000\nAverage RPS: ~300,000 (typical 3x peak-to-average ratio)\n```\n\n\n#### Concurrent Connections\n\nHow many connections are open at any given time? This depends on connection duration. If the average request takes 500ms to complete (including network round trip and server processing):\n\n\n```python\nConcurrent connections = RPS × average duration\n = 1,000,000 × 0.5 seconds\n = 500,000 connections\n```\n\n\nHalf a million concurrent connections is substantial but manageable with modern hardware.\n\n### 2.2 Bandwidth Estimates\n\nNetwork bandwidth is often the first bottleneck at high throughput. Let's calculate what we need:\n\n\n```python\nAverage request size: ~2 KB (HTTP headers + small payload)\nAverage response size: ~10 KB (typical API response)\n\nIngress bandwidth = 1M RPS × 2 KB = 2 GB/s\nEgress bandwidth = 1M RPS × 10 KB = 10 GB/s\nTotal bandwidth = 12 GB/s = ~96 Gbps\n```\n\n\nThis is significant. A single 10 Gbps network card would not be enough. We need either multiple NICs, faster networking (25/40/100 Gbps), or multiple load balancer nodes to distribute the bandwidth requirement.\n\n### 2.3 Health Check Overhead\n\nHealth checks generate their own traffic. With 1,000 backend servers and checks every 5 seconds:\n\n\n```python\nHealth check traffic = 1,000 servers / 5 seconds = 200 checks/second\n```\n\n\nThis is negligible compared to user traffic. Health checking will not be a bottleneck.\n\n### 2.4 Memory Requirements\n\nEach active connection needs state tracking in memory:\n\n\n| Data | Size |\n|------|------|\n| Source IP + port | 6 bytes |\n| Destination IP + port | 6 bytes |\n| Connection state | 4 bytes |\n| Timestamps | 16 bytes |\n| Protocol-specific data | ~200 bytes |\n| Buffer space | ~250 bytes |\n| **Total per connection** | **~500 bytes** |\n\n\nFor 500,000 concurrent connections:\n\n\n```python\nConnection state memory = 500,000 × 500 bytes = 250 MB\n```\n\n\n250 MB is quite modest. Memory will not be our bottleneck. The bigger memory consideration is if we implement SSL termination, where session caches and certificate storage can consume several gigabytes.\n\n### 2.5 Key Insights\n\nThese estimates tell us several important things about our design:\n\n1. **Network bandwidth is the primary constraint.** At 12 GB/s, we need high-speed networking and likely multiple load balancer nodes.\n2. **Memory is not a concern.** Connection tracking is cheap. Even 10x our estimate would only be 2.5 GB.\n3. **Health checks are lightweight.** 200 checks per second is nothing compared to 1M requests per second.\n4. **Horizontal scaling is necessary.** A single machine cannot handle 100 Gbps of traffic. We need to design for multiple load balancer nodes from the start.\n\n---\n\n# 3. Core APIs\n\nA load balancer has two distinct interfaces. The **data plane** handles actual traffic (this is where millions of requests flow through), and the **control plane** handles configuration and management (registering backends, changing algorithms, viewing health status).\n\nThe data plane is implicit since it just forwards network traffic. The control plane needs explicit APIs for operators to manage the system. Let's define those.\n\n\n```mermaid\nflowchart LR\n subgraph \"Control Plane APIs\"\n direction TB\n A[Register Backend]:::green\n B[Remove Backend]:::red\n C[Get Health Status]:::primary\n D[Configure Algorithm]:::orange\n end\n\n subgraph \"Data Plane\"\n E[Traffic Forwarding
No explicit API]:::secondary\n end\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n### 3.1 Register Backend Server\n\n#### Endpoint: `POST /backends`\n\nWhen you spin up a new application server, you need to tell the load balancer about it. This endpoint adds a backend to the pool and starts health checking it.\n\n#### Request Body\n\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `address` | string | Yes | IP address or hostname of the backend server |\n| `port` | integer | Yes | Port the backend is listening on |\n| `weight` | integer | No | Weight for weighted load balancing (default: 1) |\n| `health_check_path` | string | No | HTTP path for health checks (default: `/health`) |\n\n\n#### Example Request\n\n\n```json\n{\n \"address\": \"10.0.1.15\",\n \"port\": 8080,\n \"weight\": 2,\n \"health_check_path\": \"/healthz\"\n}\n```\n\n\n#### Success Response (201 Created)\n\n\n```json\n{\n \"backend_id\": \"backend-a1b2c3\",\n \"status\": \"unknown\",\n \"message\": \"Backend registered. Health check pending.\"\n}\n```\n\n\nThe initial status is \"unknown\" because we have not run a health check yet. Within a few seconds, it will transition to \"healthy\" or \"unhealthy.\"\n\n### 3.2 Remove Backend Server\n\n#### Endpoint: `DELETE /backends/{backend_id}`\n\nWhen you want to decommission a server (for maintenance, scaling down, or replacement), this endpoint removes it from the pool. The load balancer will stop sending new traffic immediately, but existing connections are allowed to complete gracefully.\n\n**Path Parameter:** `backend_id` - the ID returned when the backend was registered\n\n#### Success Response (200 OK)\n\n\n```json\n{\n \"message\": \"Backend removed successfully\",\n \"backend_id\": \"backend-a1b2c3\",\n \"drained_connections\": 42\n}\n```\n\n\nThe `drained_connections` field tells you how many connections were in progress when the backend was removed.\n\n### 3.3 Get Backend Health Status\n\n#### Endpoint: `GET /backends/{backend_id}/health`\n\nReturns detailed health information about a specific backend, useful for debugging and monitoring dashboards.\n\n#### Success Response (200 OK)\n\n\n```json\n{\n \"backend_id\": \"backend-a1b2c3\",\n \"status\": \"healthy\",\n \"last_check\": \"2024-01-15T10:30:00Z\",\n \"response_time_ms\": 12,\n \"active_connections\": 847,\n \"total_requests\": 1250000,\n \"failed_requests\": 23\n}\n```\n\n\nThe response includes both current state (status, active connections) and historical metrics (total requests, failures) to help operators understand backend performance over time.\n\n### 3.4 Configure Load Balancing Algorithm\n\n#### Endpoint: `PUT /config/algorithm`\n\nChanges how traffic is distributed across backends. This takes effect immediately for new connections (existing connections are not affected).\n\n#### Request Body\n\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `algorithm` | string | Yes | One of: `round_robin`, `weighted_round_robin`, `least_connections`, `ip_hash`, `random` |\n| `sticky_sessions` | boolean | No | Enable session persistence (default: false) |\n| `sticky_ttl_seconds` | integer | No | How long sticky sessions last (default: 3600) |\n\n\n#### Example Request\n\n\n```json\n{\n \"algorithm\": \"least_connections\",\n \"sticky_sessions\": true,\n \"sticky_ttl_seconds\": 1800\n}\n```\n\n\n#### Success Response (200 OK)\n\n\n```json\n{\n \"message\": \"Configuration updated\",\n \"algorithm\": \"least_connections\",\n \"sticky_sessions\": true\n}\n```\n\n\n---\n\n# 4. High-Level Design\n\nNow we get to the interesting part: designing the system architecture. Rather than presenting a complex diagram upfront, we will build the design incrementally, starting with the simplest solution and adding components as we encounter challenges. This approach mirrors how you would tackle this problem in an interview.\n\nOur load balancer needs to do three fundamental things:\n\n1. **Distribute Traffic:** Accept incoming requests and forward them to healthy backend servers.\n2. **Monitor Health:** Continuously check backends and route around failures.\n3. **Stay Available:** The load balancer itself cannot be a single point of failure.\n\nThe architecture naturally splits into two parts: the **data plane** that handles actual traffic at high speed, and the **control plane** that manages configuration and health checking at a slower pace.\n\n\n```mermaid\nflowchart TB\n subgraph \"Control Plane\"\n direction LR\n API[Config API]:::orange\n CM[Config Manager]:::orange\n HC[Health Checker]:::orange\n STORE[(Config Store)]:::teal\n API --> CM\n CM --> STORE\n HC --> CM\n end\n\n subgraph \"Data Plane\"\n direction LR\n C[Clients]:::rose --> FL[Frontend Listener]:::primary\n FL --> RE[Routing Engine]:::primary\n RE --> B1[Backend 1]:::green\n RE --> B2[Backend 2]:::green\n RE --> B3[Backend 3]:::green\n end\n\n CM -.->|\"Push config\"| RE\n HC -.->|\"Health status\"| RE\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\nThe data plane must be fast since every request flows through it. The control plane can be slower since configuration changes and health checks happen infrequently compared to request traffic.\n\nLet's build this architecture step by step, starting with the most basic requirement: getting traffic from clients to backends.\n\n---\n\n## 4.1 Requirement 1: Traffic Distribution\n\nThe most basic job of a load balancer is accepting connections from clients and forwarding them to backend servers. This sounds simple, but there are several components involved, and understanding how they work together is key to designing a good system.\n\n### Components for Traffic Distribution\n\nFour components work together to move a request from a client to the right backend server. The diagram maps how they connect:\n\n\n```mermaid\nflowchart LR\n C[Client]:::rose --> FL[Frontend Listener]:::primary\n FL --> RE[Routing Engine]:::secondary\n RE --> BP[Backend Pool]:::teal\n BP --> B1[Backend 1]:::green\n BP --> B2[Backend 2]:::green\n BP --> B3[Backend 3]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\nLet's introduce the components we need.\n\n#### Frontend Listener\n\nThis is the entry point for all client traffic. The frontend listener binds to one or more ports (typically 80 for HTTP, 443 for HTTPS) and accepts incoming TCP connections.\n\nWhen a connection arrives, the listener accepts the TCP connection from the client. For Layer 7 load balancing, it parses enough of the request to make routing decisions, reading details like HTTP headers and the URL path. It then hands off the connection to the routing engine for backend selection.\n\nIn high-performance load balancers like NGINX or HAProxy, this component is optimized to handle hundreds of thousands of connections efficiently using techniques like epoll (on Linux) or kqueue (on BSD).\n\n#### Routing Engine\n\nThis is the brain of the load balancer. Given a connection, the routing engine decides which backend server should handle it.\n\nThe routing engine maintains:\n\n- A list of available backend servers and their metadata (address, port, weight)\n- The current state of each backend (healthy, unhealthy, draining)\n- Counters for connection tracking (needed for least-connections algorithm)\n- The configured load balancing algorithm\n\nWhen the frontend listener hands off a connection, the routing engine applies the configured algorithm (round robin, least connections, etc.) and returns the selected backend.\n\n#### Backend Pool\n\nA backend pool is a logical group of servers that can handle the same type of traffic. In simple setups, you might have just one pool. In more complex architectures, you might have separate pools for different services (one for API servers, another for static content, etc.).\n\nEach pool contains:\n\n- A list of backend servers with their addresses and ports\n- Pool-specific configuration (algorithm, health check settings)\n- Health status for each backend in the pool\n\n### How a Request Flows Through the System\n\nEach step below corresponds to a handoff between components, from the initial TCP connection to the response reaching the client:\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant FL as Frontend Listener\n participant RE as Routing Engine\n participant BP as Backend Pool\n participant B as Backend Server\n\n C->>FL: TCP connection + HTTP request\n FL->>RE: Route this request\n RE->>BP: Get healthy backend\n BP-->>RE: Return Backend 2\n RE-->>FL: Use Backend 2\n FL->>B: Forward request\n B-->>FL: Response\n FL-->>C: Forward response\n```\n\n\nLet's trace a request through the system:\n\n1. **Connection arrives:** A client sends an HTTP request to the load balancer's public IP address (e.g., `https://api.example.com`).\n2. **Frontend listener accepts:** The listener accepts the TCP connection and, for HTTP traffic, reads enough of the request to understand what is being asked (the URL, headers, etc.).\n3. **Routing decision:** The listener asks the routing engine to select a backend. The engine checks the backend pool, filters out unhealthy servers, and applies the configured algorithm.\n4. **Forward to backend:** The request is forwarded to the selected backend server. This might be a simple proxy (read request, forward, read response) or a more sophisticated connection multiplexing setup.\n5. **Return response:** The backend processes the request and sends a response. The load balancer forwards this back to the client.\n6. **Connection handling:** Depending on configuration, the client connection might be kept alive for additional requests (HTTP keep-alive) or closed.\n\nAt this point, we have basic load balancing working. But what happens when Backend 2 crashes? Without health checking, the load balancer would keep sending traffic to it, and users would see errors. Let's address that next.\n\n---\n\n## 4.2 Requirement 2: Health Monitoring\n\nServers fail. It is not a matter of if, but when. A process might crash, a disk might fill up, or a network partition might isolate a server. Without health checking, the load balancer would blindly keep sending traffic to dead servers, and users would see errors.\n\nHealth monitoring solves this by continuously checking each backend and automatically routing around failures.\n\n### The Health Checker Component\n\nWe need a new component: the Health Checker. This is a background process that periodically probes each backend server to verify it is working correctly.\n\nThe health checker is responsible for:\n\n- Sending periodic probes to each backend (typically every 5-10 seconds)\n- Tracking the history of successes and failures\n- Deciding when a backend should be marked unhealthy (usually after 2-3 consecutive failures)\n- Deciding when an unhealthy backend can be marked healthy again (after 2-3 consecutive successes)\n- Notifying the routing engine whenever a backend's status changes\n\n### Types of Health Checks\n\nDifferent applications need different types of health checks. A basic TCP check might be enough for some services, while others need a full HTTP request to verify the application is actually working.\n\n\n| Type | How It Works | When to Use |\n|------|--------------|-------------|\n| **TCP Check** | Opens a TCP connection and closes it | Basic connectivity verification. Fast but does not verify the application is working. |\n| **HTTP Check** | Sends an HTTP request (GET /health), expects 2xx response | Web applications. Verifies the app can handle requests. |\n| **Custom Script** | Runs a user-defined command or script | Complex checks like database connectivity, queue depth, etc. |\n\n\nMost production deployments use HTTP health checks. The application exposes a `/health` or `/healthz` endpoint that returns 200 OK when everything is working, and returns an error (or times out) when something is wrong.\n\n### How Health Checking Works\n\nThe health checker probes every backend on a fixed interval and feeds the results back to the routing engine, which keeps an up-to-date view of which servers can safely receive traffic:\n\n\n```mermaid\nflowchart TB\n subgraph \"Health Check Loop\"\n HC[Health Checker]:::primary\n end\n\n subgraph \"Backend Servers\"\n B1[Backend 1
Healthy ✓]:::green\n B2[Backend 2
Unhealthy ✗]:::red\n B3[Backend 3
Healthy ✓]:::green\n end\n\n subgraph \"Routing\"\n RE[Routing Engine]:::secondary\n BP[Backend Pool
Active: B1, B3]:::teal\n end\n\n HC -->|\"GET /health\"| B1\n HC -->|\"GET /health\"| B2\n HC -->|\"GET /health\"| B3\n\n B1 -->|\"200 OK\"| HC\n B2 -->|\"Timeout\"| HC\n B3 -->|\"200 OK\"| HC\n\n HC -->|\"Update status\"| RE\n RE --> BP\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nHere is what happens when a backend fails:\n\n1. **Continuous monitoring:** The health checker sends probes to each backend every 5 seconds. Backends 1 and 3 respond with 200 OK. Backend 2 does not respond within the timeout.\n2. **Failure tracking:** One failed check is not enough to mark a backend unhealthy (it could be a network blip). After 3 consecutive failures, Backend 2 is marked unhealthy.\n3. **Status update:** The health checker notifies the routing engine that Backend 2 is now unhealthy.\n4. **Traffic rerouting:** The routing engine immediately stops sending new traffic to Backend 2. Only Backends 1 and 3 receive requests.\n5. **Recovery:** If Backend 2 starts responding again, it needs to pass several consecutive health checks before being marked healthy and returned to the pool. This prevents flapping (rapidly switching between healthy and unhealthy).\n\nThis entire process happens automatically. No human intervention is required to handle a failed server.\n\n---\n\n## 4.3 Requirement 3: High Availability\n\nWe have a problem. We built a load balancer to eliminate single points of failure in our backend, but the load balancer itself is now a single point of failure. If it crashes, all traffic stops. This is obviously unacceptable for a system targeting 99.99% uptime.\n\nThe solution is to run multiple load balancer instances. But this introduces new questions:\n\n- How do clients know which instance to connect to?\n- What happens when one instance fails?\n- How do we handle state (like sticky sessions) across multiple instances?\n\nLet's look at the two main patterns for achieving high availability.\n\n### Pattern 1: Active-Passive (Failover)\n\nIn this pattern, one load balancer handles all traffic while a backup waits idle, ready to take over if the primary fails.\n\n\n```mermaid\nflowchart LR\n\n subgraph \"After Primary Fails\"\n direction LR\n C2[Clients]:::rose --> VIP2[Virtual IP
192.168.1.100]:::primary\n LB2A[\"LB Primary
(Failed)\"]:::red\n VIP2 --> LB2B[\"LB Standby
(Now Active)\"]:::green\n LB2B --> BP2[Backends]:::teal\n end\n\n subgraph \"Normal Operation\"\n direction LR\n C1[Clients]:::rose --> VIP1[Virtual IP
192.168.1.100]:::primary\n VIP1 --> LB1A[\"LB Primary
(Handling traffic)\"]:::green\n LB1A --> BP1[Backends]:::teal\n LB1B[\"LB Standby
(Idle)\"]:::orange\n LB1A <-.->|\"Heartbeat\"| LB1B\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n#### How it works\n\nThe key concept here is the Virtual IP (VIP). Clients connect to the VIP, not to either load balancer directly. Both load balancers know about this VIP, but only the active one actually responds to traffic on it.\n\nThe standby continuously monitors the primary using heartbeat messages (typically sent every second). If the primary stops responding, the standby \"claims\" the VIP by broadcasting an ARP message that says \"I am now the owner of 192.168.1.100.\" Within 1-3 seconds, traffic starts flowing to the new active node.\n\nThis pattern is commonly implemented using protocols like VRRP (Virtual Router Redundancy Protocol) or vendor-specific solutions like AWS's Elastic Load Balancer.\n\n**Pros:** Simple to set up and reason about. No state synchronization needed since only one node handles traffic at a time.\n\n**Cons:** Half your capacity sits idle during normal operation. You are paying for servers that are not doing useful work.\n\n### Pattern 2: Active-Active\n\nIn this pattern, multiple load balancer nodes handle traffic simultaneously. If one fails, the others continue without interruption.\n\n\n```mermaid\nflowchart TB\n\n subgraph \"After Node 1 Fails\"\n direction TB\n DNS2[DNS updated or
health check removes]:::secondary\n DNS2 -.->|\"Removed\"| LB1F[LB Node 1
Failed]:::red\n DNS2 --> LB2F[LB Node 2
Handling 100% traffic]:::green\n LB2F --> BP2[Backend Pool]:::teal\n end\n\n subgraph \"Normal Operation\"\n direction TB\n DNS[DNS resolves to
multiple IPs]:::secondary\n DNS --> LB1[LB Node 1
Handling 50% traffic]:::green\n DNS --> LB2[LB Node 2
Handling 50% traffic]:::green\n LB1 --> BP[Backend Pool]:::teal\n LB2 --> BP\n end\n\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**How it works:**\n\nTraffic is distributed across multiple load balancer nodes using one of several methods:\n\n1. **DNS round-robin:** DNS returns multiple IP addresses, and clients pick one. Not ideal since DNS caching can cause uneven distribution and slow failover.\n2. **Upstream load balancer:** Another layer of load balancing (like a hardware L4 load balancer or cloud provider's network load balancer) distributes traffic to your L7 load balancers.\n3. **Anycast:** All nodes share the same IP address. The network routes each connection to the nearest healthy node. This is how Cloudflare and other CDNs work at global scale.\n\n**Pros:** All resources are utilized during normal operation. Better throughput since traffic is spread across multiple nodes. More resilient since the failure of one node only reduces capacity rather than causing an outage.\n\n**Cons:** More complex to manage. If you use sticky sessions, you need to synchronize session state across nodes (typically using a shared Redis cluster).\n\n### Which Pattern Should You Choose?\n\n\n| Consideration | Active-Passive | Active-Active |\n|---------------|----------------|---------------|\n| **Complexity** | Simpler | More complex |\n| **Resource utilization** | 50% (standby is idle) | 100% |\n| **Failover time** | 1-3 seconds | Near-instant (traffic shifts to remaining nodes) |\n| **Sticky sessions** | Easy (all state on one node) | Requires shared state store |\n| **Scale** | Limited to one node's capacity | Scales horizontally |\n\n\nFor most production systems handling significant traffic, **active-active** is the better choice despite its complexity. The benefits of full resource utilization and instant failover outweigh the added complexity of state synchronization.\n\n---\n\n## 4.4 Putting It All Together\n\nNow that we have designed each piece, let's step back and see how they all fit together. Here is the complete architecture that satisfies our requirements for traffic distribution, health monitoring, and high availability.\n\n\n```mermaid\nflowchart TB\n subgraph Clients[\"Client Layer\"]\n C1[Client 1]:::rose\n C2[Client 2]:::rose\n C3[Client 3]:::rose\n end\n\n subgraph LBCluster[\"Load Balancer Cluster\"]\n VIP[Virtual IP / DNS
Entry Point]:::primary\n LB1[LB Node 1]:::secondary\n LB2[LB Node 2]:::secondary\n SS[(Session Store
Redis)]:::teal\n end\n\n subgraph ControlPlane[\"Control Plane\"]\n HC[Health Checker]:::orange\n CM[Config Manager]:::orange\n CS[(Config Store)]:::teal\n end\n\n subgraph BackendPool[\"Backend Pool\"]\n B1[Backend 1
Healthy]:::green\n B2[Backend 2
Healthy]:::green\n B3[Backend 3
Unhealthy]:::red\n end\n\n C1 & C2 & C3 --> VIP\n VIP --> LB1 & LB2\n LB1 --> B1 & B2\n LB2 --> B1 & B2\n LB1 <--> SS\n LB2 <--> SS\n HC --> B1 & B2 & B3\n HC --> LB1 & LB2\n CM --> LB1 & LB2\n CM <--> CS\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\nLet's trace how everything works together:\n\n1. **Clients connect to the VIP.** They do not know (or care) about the individual load balancer nodes. The VIP is a stable entry point that abstracts away the LB cluster.\n2. **Traffic is distributed across LB nodes.** In active-active mode, both LB Node 1 and LB Node 2 handle traffic simultaneously.\n3. **Each LB node routes to healthy backends.** Backend 3 failed health checks and is marked unhealthy (red), so no traffic goes to it. LB nodes only send traffic to Backends 1 and 2.\n4. **Session state is shared.** If a user's first request hits LB Node 1 and sticky sessions are enabled, their session mapping is stored in Redis. If their next request happens to hit LB Node 2, it can still route them to the correct backend by looking up their session in Redis.\n5. **Health checking runs continuously.** The Health Checker monitors all backends and both LB nodes. When it detects Backend 3 is unhealthy, it notifies both LB nodes to stop routing traffic there.\n6. **Configuration is managed centrally.** The Config Manager stores configuration (backend lists, algorithms, health check settings) in the Config Store and pushes updates to all LB nodes.\n\n### Component Summary\n\n\n| Component | What It Does | Key Characteristics |\n|-----------|--------------|---------------------|\n| **Virtual IP / DNS** | Provides a stable entry point for clients | Abstracts away the LB cluster |\n| **LB Nodes** | Accept connections and route to backends | Stateless, horizontally scalable |\n| **Session Store (Redis)** | Stores sticky session mappings | Shared across all LB nodes |\n| **Health Checker** | Monitors backend and LB health | Runs continuously in the background |\n| **Config Manager** | Manages load balancer configuration | Handles API requests, persists config |\n| **Config Store** | Persists configuration | Could be etcd, Consul, or a database |\n| **Backend Pool** | The application servers being load balanced | Grouped by service type |\n\n\nThis architecture handles millions of requests per second while automatically recovering from failures. The separation of data plane (LB nodes) and control plane (health checker, config manager) keeps the traffic path fast and simple.\n\n---\n\n# 5. Database Design\n\nA load balancer does not need a traditional database for its core operation. The data plane (traffic forwarding) is entirely in-memory since every nanosecond counts when you are handling millions of requests per second.\n\nHowever, the control plane does need persistent storage. Configuration (which backends exist, what algorithm to use, health check settings) needs to survive restarts. And if you are using sticky sessions in an active-active setup, session mappings need to be shared across LB nodes.\n\nLet's think through what data lives where.\n\n## 5.1 Where Does Data Live?\n\nDifferent types of data have different requirements for speed, persistence, and sharing. Here is how we split things up:\n\n\n```mermaid\nflowchart LR\n\n subgraph \"Persistent Store (etcd/Consul)\"\n E[Backend Configuration
addresses, weights]:::teal\n F[Pool Configuration
algorithms, settings]:::teal\n end\n\n subgraph \"Shared Cache (Redis)\"\n D[Session Mappings
for sticky sessions]:::orange\n end\n\n subgraph \"In-Memory (Per LB Node)\"\n A[Active Connections
500K entries]:::primary\n B[Backend Health Status
~1000 entries]:::primary\n C[Connection Counters
for least-conn algo]:::primary\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n\n| Data Type | Storage Location | Why? |\n|-----------|-----------------|------|\n| **Active connections** | In-memory on each LB node | Must be microsecond-fast. Each LB node manages its own connections. |\n| **Backend health status** | In-memory on each LB node | Updated frequently (every health check). Pushed from health checker. |\n| **Connection counters** | In-memory on each LB node | Updated on every request for least-connections algorithm. |\n| **Session mappings** | Redis cluster | Needs to be shared across LB nodes for active-active sticky sessions. |\n| **Backend configuration** | etcd/Consul/PostgreSQL | Persistent, versioned. Survives restarts and deployments. |\n| **Metrics and logs** | Prometheus/InfluxDB | Historical data for dashboards and alerting. |\n\n\nThe key insight is that the hot path (handling requests) should never touch disk or cross a network boundary for core routing decisions. Configuration is loaded into memory at startup and updated via push notifications when it changes.\n\n## 5.2 Configuration Schema\n\nThe control plane needs to store information about backends and pools. Here is a schema that supports our API and features.\n\n#### Backend Servers\n\nEach backend server record contains everything the load balancer needs to route traffic to it:\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `backend_id` | String (PK) | Unique identifier (e.g., \"backend-a1b2c3\") |\n| `pool_id` | String (FK) | Which pool this server belongs to |\n| `address` | String | IP address or hostname |\n| `port` | Integer | Port the server listens on |\n| `weight` | Integer | Weight for weighted algorithms (default: 1) |\n| `max_connections` | Integer | Maximum concurrent connections (0 = unlimited) |\n| `enabled` | Boolean | Administrative toggle to disable without removing |\n| `created_at` | Timestamp | When this backend was registered |\n| `updated_at` | Timestamp | Last configuration change |\n\n\n#### Backend Pools\n\nPools group related backends and define shared behavior:\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `pool_id` | String (PK) | Unique identifier (e.g., \"pool-api-servers\") |\n| `name` | String | Human-readable name |\n| `algorithm` | Enum | round_robin, weighted_round_robin, least_connections, ip_hash |\n| `health_check_type` | Enum | tcp, http, custom |\n| `health_check_path` | String | HTTP path for health checks (e.g., \"/health\") |\n| `health_check_interval` | Integer | Seconds between health checks |\n| `healthy_threshold` | Integer | Consecutive successes to mark healthy |\n| `unhealthy_threshold` | Integer | Consecutive failures to mark unhealthy |\n| `sticky_sessions` | Boolean | Enable session persistence |\n| `sticky_ttl` | Integer | How long sticky sessions last (seconds) |\n\n\n## 5.3 Session Store Schema (Redis)\n\nWhen sticky sessions are enabled, we need to remember which backend each user should go to. Redis is perfect for this since it is fast, supports TTLs, and can be shared across LB nodes.\n\n\n```shell\nKey: sticky:{client_identifier}\nValue: backend-a1b2c3\nTTL: 1800 seconds (30 minutes, configurable)\n```\n\n\nThe `client_identifier` depends on the stickiness method:\n\n- For cookie-based stickiness: a session ID from the cookie\n- For IP-based stickiness: the client's IP address\n- For header-based stickiness: a hash of the relevant header value\n\nWhen a request arrives:\n\n1. Look up `sticky:{client_identifier}` in Redis\n2. If found and the backend is healthy, route to that backend\n3. If not found (or backend is unhealthy), apply normal algorithm and store the mapping\n\n---\n\n# 6. Design Deep Dive\n\nThe high-level architecture gives us a solid foundation, but system design interviews often go deeper. A strong answer shows that you understand the trade-offs involved in key decisions. \n\nIn this section, we will explore the most important design choices in detail: load balancing algorithms, health checking strategies, session persistence, Layer 4 vs Layer 7, SSL termination, and handling load balancer failures.\n\n---\n\n## 6.1 Load Balancing Algorithms\n\nThe load balancing algorithm determines how traffic gets distributed across backends. This is one of the most important decisions in your design since it directly affects how evenly load is spread, how well the system handles failures, and how it behaves under various traffic patterns.\n\nA good algorithm should:\n\n- Distribute load reasonably evenly across healthy backends\n- Avoid overloading any single server\n- Work well with your specific traffic patterns (stateless vs stateful, uniform vs variable request costs)\n- Be simple enough to execute quickly (remember, this runs on every request)\n\nLet's explore the main options, starting with the simplest and building to more sophisticated approaches.\n\n### Approach 1: Round Robin\n\nRound robin is the simplest load balancing algorithm, and often the default choice. Requests are distributed sequentially across backends: first request goes to Backend 1, second to Backend 2, third to Backend 3, then back to Backend 1.\n\n#### How It Works\n\nThe algorithm maintains a counter that increments with each request. The selected backend is `counter % number_of_backends`.\n\n\n```mermaid\nflowchart TB\n subgraph \"Request Distribution\"\n direction LR\n R1[\"Req 1\"]:::rose\n R2[\"Req 2\"]:::rose\n R3[\"Req 3\"]:::rose\n R4[\"Req 4\"]:::rose\n R5[\"Req 5\"]:::rose\n R6[\"Req 6\"]:::rose\n end\n\n LB[Load Balancer
counter++]:::primary\n\n subgraph \"Backends\"\n B1[\"Backend 1
Gets: 1, 4\"]:::orange\n B2[\"Backend 2
Gets: 2, 5\"]:::green\n B3[\"Backend 3
Gets: 3, 6\"]:::teal\n end\n\n R1 & R2 & R3 & R4 & R5 & R6 --> LB\n LB --> B1 & B2 & B3\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nThe implementation is trivial:\n\n\n```shell\ncounter = counter + 1\nselected_backend = backends[counter % len(backends)]\n```\n\n\n#### Why use it?\n\nRound robin's simplicity is its strength. There is no state to track beyond a single counter. Each LB node can run independently with its own counter since exact coordination is not necessary for good distribution over time. The algorithm adds essentially zero latency.\n\n#### When it falls short\n\nThe simplicity comes with blind spots. Round robin treats all backends and all requests as equal, which is often not true:\n\n- If Backend 1 is a powerful machine and Backend 2 is weaker, both get the same traffic, potentially overloading Backend 2.\n- If some requests are expensive (complex queries, large file uploads), round robin does not account for this. A backend might get three expensive requests in a row while another gets three cheap ones.\n- If a backend is slow (due to garbage collection, disk I/O, etc.), round robin keeps sending traffic to it.\n\n#### Best for\n\nHomogeneous backends (same hardware, same configuration) handling uniform requests (similar processing cost per request). This is common for stateless API servers.\n\n---\n\n### Approach 2: Weighted Round Robin\n\nWhat if your backends are not identical? Maybe Backend 1 is a beefy 32-core machine while Backend 2 is a smaller 8-core instance. Sending equal traffic to both would waste capacity on Backend 1 and overload Backend 2.\n\nWeighted round robin solves this by assigning a weight to each backend. Backends with higher weights receive proportionally more traffic.\n\n#### How It Works\n\nIf Backend 1 has weight 3, Backend 2 has weight 1, and Backend 3 has weight 2, the total weight is 6. Over every 6 requests:\n\n- Backend 1 gets 3 requests (50%)\n- Backend 2 gets 1 request (17%)\n- Backend 3 gets 2 requests (33%)\n\n\n```mermaid\nflowchart TB\n subgraph Req[\"6 Requests\"]\n R[\"Requests 1–6\"]:::rose\n end\n\n LB[\"Load Balancer\"]:::primary\n\n subgraph WR[\"Weighted Distribution\"]\n B1[\"Backend 1
weight: 3 → 50%\"]:::orange\n B2[\"Backend 2
weight: 1 → 17%\"]:::green\n B3[\"Backend 3
weight: 2 → 33%\"]:::teal\n end\n\n R --> LB\n LB -->|\"Reqs 1, 2, 3\"| B1\n LB -->|\"Req 4\"| B2\n LB -->|\"Reqs 5, 6\"| B3\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### Implementation considerations\n\nA naive implementation might send the first 3 requests to Backend 1, then 1 to Backend 2, then 2 to Backend 3. This creates \"bursts\" to high-weight servers, which can cause momentary overload.\n\nBetter implementations like NGINX's \"smooth weighted round robin\" interleave requests more evenly. Instead of [1,1,1,2,3,3], you get something like [1,3,1,2,1,3], spreading the load more smoothly over time.\n\n#### When to use it\n\nWeighted round robin is great when you know your backends have different capacities and want to utilize them proportionally. Common scenarios:\n\n- Mixed instance types (some large, some small)\n- Gradual rollouts (new version gets weight 1, old version gets weight 9)\n- Geographic distribution (nearby datacenter gets higher weight)\n\n#### Limitations\n\nThe weights are static. If Backend 1 suddenly slows down due to a noisy neighbor or garbage collection, it keeps getting the same traffic. The algorithm does not adapt to runtime conditions.\n\n---\n\n### Approach 3: Least Connections\n\nBoth round robin and weighted round robin are \"blind\" to what is happening on your backends. They distribute requests based on a predetermined pattern, regardless of whether a backend is struggling with a heavy workload.\n\nLeast connections takes a different approach: it sends each new request to whichever backend currently has the fewest active connections.\n\n#### How It Works\n\nThe load balancer tracks how many connections are currently active on each backend. When a new request arrives, it scans the list and picks the backend with the lowest count.\n\n\n```mermaid\nflowchart TD\n NR[New Request]:::rose --> LB[Load Balancer]:::primary\n\n LB -.-> |\"10 connections\"| B1[Backend 1]:::orange\n LB --> |\"5 connections ✓\"| B2[Backend 2]:::green\n LB -.-> |\"8 connections\"| B3[Backend 3]:::orange\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nIn this example, Backend 2 has only 5 active connections compared to 10 and 8 on the others. The new request goes to Backend 2.\n\n#### Why this works well\n\nLeast connections naturally adapts to runtime conditions. If one backend is processing slow requests (maybe a complex database query or a large file upload), it accumulates connections and gets fewer new ones. Meanwhile, a backend that is processing requests quickly keeps its connection count low and picks up more traffic.\n\nThis self-balancing behavior is powerful. It handles:\n\n- Backends with different capacities (faster servers complete requests faster, so they have fewer connections and get more traffic)\n- Variable request costs (expensive requests tie up connections longer)\n- Transient slowdowns (a backend doing garbage collection temporarily gets fewer requests)\n\n#### The catch\n\nLeast connections requires state. Each LB node needs to track connection counts for every backend. In a distributed setup with multiple LB nodes, this gets tricky since each node only knows about its own connections. You have a few options:\n\n- Each node tracks only its own connections (good enough in practice for most workloads)\n- Nodes share connection counts via a shared cache (more accurate but adds latency)\n- Use a hybrid approach where nodes periodically sync counts\n\n#### Best for\n\nWorkloads with variable request processing times, like applications that mix quick API calls with slow database queries or file operations.\n\n---\n\n### Approach 4: Weighted Least Connections\n\nThis combines the best of weighted round robin and least connections. It accounts for both configured capacity (weights) and real-time load (connection counts).\n\n#### How It Works\n\nInstead of picking the backend with the fewest connections, we pick the one with the lowest ratio of connections to weight. A powerful server (high weight) can handle more connections before its ratio becomes high.\n\n\n```shell\nBackend 1: 10 connections, weight 5 → ratio = 10/5 = 2.0\nBackend 2: 6 connections, weight 2 → ratio = 6/2 = 3.0\nBackend 3: 4 connections, weight 2 → ratio = 4/2 = 2.0\n\n→ Backend 1 or 3 selected (tie at 2.0, pick first)\n```\n\n\nEven though Backend 1 has more connections (10) than Backend 3 (4), it has a higher weight, so its ratio is the same. Both are equally loaded relative to their capacity.\n\n#### Why this is often the best choice\n\nWeighted least connections gives you:\n\n- Capacity-aware routing (like weighted round robin)\n- Adaptive behavior (like least connections)\n- Proportional load distribution (each backend stays at roughly the same utilization percentage)\n\nThis is why many production load balancers default to weighted least connections for HTTP traffic.\n\n#### Complexity trade-off\n\nThe algorithm is more complex than round robin. You need to track connections per backend and calculate ratios on each request. But the computation is trivial (a few integer operations), and the better load distribution is usually worth it.\n\n---\n\n### Approach 5: IP Hash (Source Hashing)\n\nSometimes you need session persistence, meaning requests from the same client should always go to the same backend. The simplest way to achieve this without storing any state is to use the client's IP address as a hash key.\n\n#### How It Works\n\nThe routing decision reduces to a single formula, which every LB node can compute independently without coordination:\n\n\n```shell\nbackend_index = hash(client_ip) % number_of_backends\n```\n\n\nThe same IP address always produces the same hash, which always maps to the same backend (as long as the backend count does not change).\n\n\n```mermaid\nflowchart LR\n subgraph \"Clients\"\n C1[\"Client A
192.168.1.10\"]:::rose\n C2[\"Client B
192.168.1.20\"]:::rose\n C3[\"Client C
192.168.1.30\"]:::rose\n end\n\n LB[\"Load Balancer
hash(IP) % 3\"]:::primary\n\n subgraph \"Backends\"\n B0[\"Backend 0\"]:::orange\n B1[\"Backend 1\"]:::green\n B2[\"Backend 2\"]:::teal\n end\n\n C1 -->|\"hash → 0\"| LB\n C2 -->|\"hash → 1\"| LB\n C3 -->|\"hash → 2\"| LB\n\n LB -->|\"Always\"| B0\n LB -->|\"Always\"| B1\n LB -->|\"Always\"| B2\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nClient A (192.168.1.10) always routes to Backend 0. Client B always routes to Backend 1. No state needs to be stored anywhere.\n\n#### Why use it\n\nIP hash gives you session persistence at Layer 4, without needing to parse HTTP or store session mappings. Every LB node can compute the same hash independently since the algorithm is deterministic.\n\nThis is useful when:\n\n- You cannot modify the application to use cookies\n- You need stickiness for non-HTTP protocols\n- You want to avoid the complexity of a shared session store\n\n#### Problems with IP hash\n\nThe approach has significant limitations:\n\n1. **NAT breaks it.** If thousands of users are behind a corporate NAT, they all share the same public IP. All of them will hit the same backend, creating a hot spot.\n2. **Distribution can be uneven.** Hash functions try to spread values evenly, but with a small number of backends, you can get unlucky distributions where some backends get more traffic than others.\n3. **Adding/removing backends causes massive reshuffling.** If you go from 3 backends to 4, the formula changes from `hash % 3` to `hash % 4`. Most clients will map to different backends, losing their sessions.\n\nThe last problem is serious enough that consistent hashing was invented specifically to solve it.\n\n---\n\n### Approach 6: Consistent Hashing\n\nConsistent hashing solves the \"massive reshuffling\" problem of regular hashing. When you add or remove a backend, only a small fraction of requests get remapped instead of most of them.\n\n#### How It Works\n\nConsider a ring numbered from 0 to 2^32. Both backends and requests get hashed onto this ring. Each request goes to the first backend found when walking clockwise from its hash position.\n\n\n```mermaid\nflowchart LR\n subgraph \"Conceptual Ring\"\n direction TB\n RING[\"Hash Ring
0 → 2^32\"]:::secondary\n end\n\n subgraph \"Backends on Ring\"\n BA[\"Backend A
position: 1000\"]:::orange\n BB[\"Backend B
position: 3000\"]:::green\n BC[\"Backend C
position: 6000\"]:::teal\n end\n\n subgraph \"Request Mapping\"\n R1[\"Request (hash=500)
→ Backend A\"]:::rose\n R2[\"Request (hash=2500)
→ Backend B\"]:::rose\n R3[\"Request (hash=5000)
→ Backend C\"]:::rose\n end\n\n RING --> BA & BB & BC\n R1 --> BA\n R2 --> BB\n R3 --> BC\n\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n#### Why this is better than simple hashing\n\nConsider what happens when Backend B is removed:\n\n- Requests that were going to Backend A still go to Backend A (no change)\n- Requests that were going to Backend C still go to Backend C (no change)\n- Only requests that were going to Backend B get remapped (they now go to Backend C)\n\nWith simple `hash % n` hashing, changing from 3 backends to 2 would remap roughly 2/3 of all requests. With consistent hashing, only 1/3 get remapped (the requests that were going to the removed backend).\n\n#### Virtual nodes for better distribution\n\nOne problem with basic consistent hashing is that backends might be unevenly distributed on the ring. Backend A might end up responsible for a huge arc while Backend C gets a tiny one.\n\nThe solution is virtual nodes: instead of placing each backend once on the ring, place it multiple times (e.g., 100-200 positions per backend). This evens out the distribution significantly.\n\n#### When to use consistent hashing\n\nConsistent hashing shines in scenarios with dynamic backend pools:\n\n- Auto-scaling groups that add/remove servers based on load\n- Cache servers where remapping causes cache misses\n- Distributed databases where remapping requires data migration\n\nFor most load balancing use cases where backends are relatively stable, the added complexity is not worth it. But if you are building a distributed cache or a system with frequent scaling events, consistent hashing is valuable.\n\n---\n\n### Choosing an Algorithm\n\nHere is how the algorithms compare across key dimensions:\n\n\n| Algorithm | Complexity | Statefulness | Adaptiveness | Best For |\n|-----------|------------|--------------|--------------|----------|\n| **Round Robin** | Very simple | Stateless | None | Homogeneous backends, uniform requests |\n| **Weighted Round Robin** | Simple | Stateless | None | Known capacity differences |\n| **Least Connections** | Moderate | Per-node state | High | Variable request processing times |\n| **Weighted Least Conn** | Moderate | Per-node state | High | Production with mixed servers |\n| **IP Hash** | Simple | Stateless | None | Basic session persistence |\n| **Consistent Hash** | Complex | Stateless | None | Dynamic scaling, caching |\n\n\n#### Recommendations\n\nFor most HTTP applications, start with **Weighted Least Connections**. It handles heterogeneous backends and variable request costs gracefully, adapting to runtime conditions automatically.\n\nIf your backends are identical and your requests are uniform (a rare but possible scenario), **Round Robin** is fine and slightly faster.\n\nFor session persistence, prefer **cookie-based stickiness** (covered in section 6.3) over IP hash when possible. IP hash is fragile due to NAT and uneven distribution.\n\nUse **Consistent Hashing** when backends frequently scale up/down, or when you are load balancing cache servers where cache locality matters.\n\n---\n\n## 6.2 Health Checking Strategies\n\nWe touched on health checking in the high-level design, but there is more depth to explore. \n\nThe health checking strategy you choose affects how quickly you detect failures, how aggressively you mark servers unhealthy, and how smoothly you handle recovery.\n\n### Configuring Health Checks\n\nEvery health check has several configurable parameters. Getting these right is important since too aggressive and you will get false positives (marking healthy servers as unhealthy), too passive and you will keep routing to failed servers longer than necessary.\n\n\n| Parameter | What It Controls | Typical Value | Trade-off |\n|-----------|-----------------|---------------|-----------|\n| **Interval** | Time between checks | 5-10 seconds | Lower = faster detection, higher load |\n| **Timeout** | Max wait for response | 2-3 seconds | Lower = faster detection, more false positives |\n| **Healthy Threshold** | Passes needed to mark healthy | 2-3 | Higher = more stable, slower recovery |\n| **Unhealthy Threshold** | Failures needed to mark unhealthy | 2-3 | Higher = more stable, slower failure detection |\n\n\nA common configuration is: check every 5 seconds, timeout after 3 seconds, require 3 consecutive failures to mark unhealthy, and require 2 consecutive successes to mark healthy again.\n\n### Types of Health Checks\n\n#### TCP Health Check\n\nThe simplest form. The load balancer opens a TCP connection to the backend and immediately closes it. If the connection succeeds, the backend is considered healthy.\n\n1. The load balancer opens a TCP connection to `backend:8080`.\n2. If the connection succeeds within the timeout, the backend is healthy.\n3. If the connection is refused or times out, the backend is unhealthy.\n\nThis verifies network connectivity and that something is listening on the port, but it does not verify the application is actually working. A process could be hung, accepting connections but never responding to requests.\n\n#### HTTP Health Check\n\nThe load balancer sends an HTTP request and checks the response. This is the standard for web applications.\n\n\n```shell\n1. Send: GET /health HTTP/1.1\n2. Expect: 200 OK (within timeout)\n3. Any other response or timeout → UNHEALTHY\n```\n\n\nThe `/health` endpoint should do meaningful checks. A good health endpoint might:\n\n- Verify database connectivity\n- Check that required services are reachable\n- Return 200 only if the service can actually handle requests\n\nA bad health endpoint just returns 200 unconditionally, which tells you nothing useful.\n\n#### Custom Health Check\n\nFor complex scenarios, you can run a custom script that returns success or failure. This is useful when:\n\n- You need to check something the load balancer cannot access directly (internal databases)\n- Health determination requires complex logic\n- You are integrating with legacy systems\n\n### The Health State Machine\n\nBackends transition through states based on health check results. Understanding these transitions helps you tune the behavior.\n\n\n```mermaid\nstateDiagram-v2\n [*] --> Unknown: Backend added\n Unknown --> Healthy: First check passes\n Unknown --> Unhealthy: First check fails\n\n Healthy --> Unhealthy: Failures >= threshold\n Unhealthy --> Healthy: Passes >= threshold\n\n note right of Healthy: Receiving traffic\n note right of Unhealthy: No traffic\n\n classDef unknown fill:#ffd43b,stroke:#000,color:#000\n classDef healthy fill:#69db7c,stroke:#000,color:#000\n classDef unhealthy fill:#ff8787,stroke:#000,color:#000\n\n class Unknown unknown\n class Healthy healthy\n class Unhealthy unhealthy\n```\n\n\nThe thresholds prevent flapping. If a backend fails one check (maybe a brief network blip), it does not immediately get marked unhealthy. Only after 2-3 consecutive failures does the status change. Similarly, a backend that was down needs to pass multiple checks before being trusted with traffic again.\n\n### Graceful Degradation: Connection Draining\n\nWhen a backend fails health checks, you should not immediately drop all connections to it. That would cause errors for requests that are mid-flight. Instead, use connection draining:\n\n1. **Stop new requests:** The routing engine stops sending new requests to the failing backend immediately.\n2. **Allow existing connections to complete:** Requests that are already being processed are allowed to finish (up to a timeout, e.g., 30 seconds).\n3. **Fully remove:** Once all existing connections complete (or timeout), the backend is fully removed from the pool.\n\n\n```mermaid\nflowchart LR\n subgraph \"Phase 1: Detection\"\n HC[Health Check Fails]:::red\n end\n\n subgraph \"Phase 2: Draining\"\n DR[Stop New Traffic
Allow Existing to Complete]:::orange\n end\n\n subgraph \"Phase 3: Removed\"\n RM[Fully Removed
from Pool]:::secondary\n end\n\n subgraph \"Phase 4: Recovery\"\n REC[Health Checks Pass
Backend Returns]:::green\n end\n\n HC --> DR --> RM --> REC\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThis graceful degradation ensures users experience minimal disruption when backends fail.\n\n---\n\n## 6.3 Session Persistence (Sticky Sessions)\n\nNot all applications are stateless. Many legacy systems (and some modern ones) store user session data in local memory on the backend server. If User A logs in on Backend 1, their session lives in Backend 1's memory. If their next request goes to Backend 2, they appear logged out.\n\nSession persistence (also called \"sticky sessions\") solves this by ensuring that requests from the same user consistently go to the same backend.\n\n### The Problem\n\nWithout stickiness, the load balancer distributes requests without regard to who is making them:\n\n- Request 1 from User A goes to Backend 1, where the user logs in and a session is created.\n- Request 2 from User A goes to Backend 2, which has no session, so the user sees the login page.\n- Request 3 from User A goes to Backend 1, which finds the session, so the user is logged in again.\n\nThis creates a confusing, broken experience. Session persistence fixes it.\n\n### Approach 1: Cookie-Based Stickiness\n\nThe most common and reliable approach for HTTP traffic. The load balancer injects a cookie that identifies which backend should handle the user's requests.\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant LB as Load Balancer\n participant B1 as Backend 1\n participant B2 as Backend 2\n\n Note over C,B2: First request (no cookie)\n C->>LB: GET /login\n LB->>B1: Forward (selected by algorithm)\n B1-->>LB: Response\n LB-->>C: Response + Set-Cookie: SERVERID=B1\n\n Note over C,B2: Subsequent requests\n C->>LB: GET /dashboard + Cookie: SERVERID=B1\n Note over LB: Cookie says B1\n LB->>B1: Forward to Backend 1\n B1-->>LB: Response\n LB-->>C: Response\n```\n\n\n#### How it works\n\n1. User's first request has no sticky cookie. The load balancer picks a backend using the normal algorithm.\n2. The response includes a `Set-Cookie` header with the backend identifier (e.g., `SERVERID=backend-1`).\n3. The browser includes this cookie in all subsequent requests.\n4. The load balancer reads the cookie and routes directly to the specified backend.\n\n#### Why this is the best option for HTTP\n\n- Works reliably regardless of client IP changes (mobile users, corporate proxies)\n- No shared state needed between load balancer nodes\n- Cookie survives load balancer restarts\n- Can include a signature to prevent tampering\n\n#### Limitations\n\n- Requires Layer 7 (HTTP) inspection, does not work for raw TCP traffic\n- If the specified backend is unhealthy, the load balancer must pick a new one (and update the cookie)\n- Adds a small amount of overhead to read and potentially set cookies\n\n### Approach 2: Source IP Persistence\n\nFor non-HTTP traffic or when you cannot use cookies, you can route based on client IP address. This is essentially the IP Hash algorithm applied for persistence.\n\n#### How it works\n\nTwo example mappings show the relationship between a client IP and its assigned backend:\n\n\n```python\nUser with IP 192.168.1.100 → Always routes to Backend 2\nUser with IP 10.0.0.50 → Always routes to Backend 1\n```\n\n\n#### When it works\n\n- Each client has a unique, stable public IP\n- You need persistence for non-HTTP protocols\n- You cannot modify the application or client\n\n#### When it breaks\n\n- Corporate NAT: Thousands of users behind one IP all go to the same backend\n- Mobile users: IP changes as they move between networks\n- IPv4 exhaustion: More clients share fewer IPs\n\nUse IP-based persistence only when cookie-based is not an option.\n\n### Approach 3: Externalized Session Store\n\nThe cleanest solution is to avoid the problem entirely by making your application stateless. Instead of storing session data in backend memory, store it in a shared session store like Redis.\n\n\n```mermaid\nflowchart LR\n C[Client]:::rose --> LB[Load Balancer]:::primary\n LB --> B1[Backend 1]:::green\n LB --> B2[Backend 2]:::green\n LB --> B3[Backend 3]:::green\n B1 & B2 & B3 --> SS[(Redis
Session Store)]:::teal\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### How it works\n\n1. User logs in on Backend 1. Backend 1 creates a session and stores it in Redis.\n2. User's next request goes to Backend 2. Backend 2 looks up the session in Redis and finds it.\n3. Any backend can handle any request since they all share the same session data.\n\n#### Why this is the best long-term solution\n\n- No sticky sessions needed, the load balancer can use pure round robin or least connections\n- Backends can scale horizontally without concern for session affinity\n- Backend failures do not cause session loss (session is safe in Redis)\n- Simpler load balancer configuration\n\n#### The trade-off\n\nThis requires application changes. You need to configure your framework to use Redis (or another external store) instead of local memory for sessions. For new applications, this is easy. For legacy applications, it might require significant refactoring.\n\n### Recommendation\n\nIf you are building a new application, design it to be stateless from the start. Use an external session store (Redis is the go-to choice) and avoid the complexity of sticky sessions entirely.\n\nFor legacy applications that cannot be modified, use cookie-based stickiness for HTTP traffic. It is the most reliable option.\n\nAvoid IP-based persistence unless you have no other choice.\n\n---\n\n## 6.4 Layer 4 vs Layer 7 Load Balancing\n\nWhen discussing load balancers, you will often hear terms like \"L4\" and \"L7.\" These refer to different layers of the network stack where the load balancer makes its decisions. The choice between them affects what the load balancer can do and how fast it can do it.\n\n### Layer 4 Load Balancing\n\nA Layer 4 load balancer operates at the transport layer. It sees TCP/UDP packets but does not understand what is inside them. It makes routing decisions based only on the source and destination IP addresses, the source and destination ports, and the protocol (TCP or UDP).\n\n\n```mermaid\nflowchart TD\n C[Client
192.168.1.100]:::rose --> L4[L4 Load Balancer]:::primary\n L4 --> B1[Backend 1]:::green\n L4 --> B2[Backend 2]:::green\n\n L4 -.- INFO[\"Sees only:
• IP addresses
• Port numbers
• TCP/UDP\"]:::secondary\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### What L4 can do\n\n- Route based on port (send traffic on port 80 to web servers, port 3306 to database servers)\n- Session persistence based on client IP\n- Health checks (TCP connection success/failure)\n\n#### What L4 cannot do\n\n- Route based on URL path (it does not understand HTTP)\n- Set or read cookies\n- Terminate SSL (it cannot decrypt the traffic)\n- Modify headers or content\n\n#### Why use L4?\n\nSpeed. Since L4 does not parse application protocols, it is fast. Some hardware L4 load balancers can handle tens of millions of packets per second. It is also protocol-agnostic, meaning it works with any TCP/UDP traffic (databases, game servers, custom protocols).\n\n### Layer 7 Load Balancing\n\nA Layer 7 load balancer operates at the application layer. It understands HTTP (and sometimes other protocols) and can make intelligent routing decisions based on request content.\n\n\n```mermaid\nflowchart TD\n C[Client]:::rose --> L7[L7 Load Balancer]:::primary\n\n L7 -->|\"/api/*\"| API[API Servers]:::orange\n L7 -->|\"/images/*\"| CDN[CDN/Static]:::green\n L7 -->|\"/admin/*\"| Admin[Admin Servers]:::teal\n\n L7 -.- INFO[\"Sees everything:
• URL, headers, cookies
• HTTP method
• Request body\"]:::secondary\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### What L7 can do\n\n- Route based on URL path (`/api` goes to API servers, `/static` goes to CDN)\n- Route based on hostname (`api.example.com` vs `www.example.com`)\n- Route based on HTTP method (GET vs POST)\n- Terminate SSL and inspect encrypted traffic\n- Set and read cookies for sticky sessions\n- Add, modify, or remove HTTP headers\n- Compress responses\n- Cache content\n\n#### The trade-off\n\nL7 is more resource-intensive. The load balancer must parse HTTP, which takes CPU cycles. SSL termination requires even more processing. A server that could handle 1M packets/second at L4 might handle only 100K requests/second at L7 (rough estimate, varies widely by configuration).\n\n### Comparison\n\n\n| Capability | Layer 4 | Layer 7 |\n|------------|---------|---------|\n| **Speed** | Fastest (packet forwarding) | Slower (protocol parsing) |\n| **Intelligence** | Basic (IP, port) | Rich (URL, headers, cookies) |\n| **SSL termination** | Pass-through only | Full termination and inspection |\n| **Sticky sessions** | IP-based only | Cookie, header, or URL-based |\n| **Content routing** | Not possible | Based on URL, host, method, etc. |\n| **Header modification** | Not possible | Add, remove, or modify headers |\n| **Caching** | Not possible | Can cache responses |\n| **Protocol support** | Any TCP/UDP | HTTP/HTTPS (usually) |\n\n\n### When to Use Each\n\n#### Use Layer 4 when\n\n- You need maximum performance\n- You are load balancing non-HTTP traffic (databases, game servers, custom protocols)\n- Simple round-robin distribution is sufficient\n- You want to pass SSL through to backends\n\n#### Use Layer 7 when\n\n- You need content-based routing (URL paths, hostnames)\n- You want SSL termination at the load balancer\n- You need cookie-based sticky sessions\n- You want to add headers (like `X-Forwarded-For` or `X-Request-ID`)\n- You want to cache or compress responses\n\nMost modern web applications use L7 load balancers because the additional capabilities outweigh the performance cost. The exception is when you have a layer of L7 load balancers behind an L4 load balancer (the L4 distributes across L7 instances).\n\n---\n\n## 6.5 SSL/TLS Termination\n\nManaging SSL certificates on every backend server is tedious and error-prone. You have dozens of servers, and each needs the certificate renewed before it expires.\n\nThis is where SSL termination helps: the load balancer handles all the encryption, and backends receive plain HTTP.\n\n### How SSL Termination Works\n\nEncrypted traffic from the client ends at the load balancer, which holds the certificate. Backends receive and return plain HTTP within the private network:\n\n\n```mermaid\nflowchart TD\n C[Client]:::rose -->|\"HTTPS
(encrypted)\"| LB[Load Balancer
SSL Termination]:::primary\n LB -->|\"HTTP
(plain)\"| B1[Backend 1]:::green\n LB -->|\"HTTP
(plain)\"| B2[Backend 2]:::green\n\n LB -.- CERT[Certificate stored
only on LB]:::orange\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nThe client connects to the load balancer over HTTPS. The load balancer decrypts the request, inspects it (for routing decisions), then forwards it to a backend over plain HTTP. The response follows the reverse path.\n\n### Why Terminate SSL at the Load Balancer?\n\n**Simplified certificate management**\n\nYou install and renew certificates in one place (the load balancer) instead of on every backend server. When a certificate expires, you update it once, not fifty times.\n\n**Offload CPU from backends**\n\nSSL encryption and decryption are CPU-intensive operations. By terminating SSL at the load balancer, your backends can use their CPU for application logic instead of cryptography. This is especially valuable if your backends are handling many small requests.\n\n**Enable content-based routing**\n\nIf you want to route based on URL path or headers, the load balancer needs to read the HTTP request. It cannot do this if the traffic is encrypted end-to-end. SSL termination gives the load balancer visibility into the request content.\n\n**TLS session caching**\n\nThe load balancer can cache TLS sessions, so clients reconnecting do not need to do a full TLS handshake every time. This improves latency for repeat visitors.\n\n### Security Implications\n\nThere is a trade-off: traffic between the load balancer and backends is unencrypted. This is fine if:\n\n- The load balancer and backends are in the same private network (e.g., a VPC)\n- You trust the network infrastructure\n- You have network-level security controls in place\n\nIf you need encryption all the way to the backend (perhaps for compliance reasons), you have options:\n\n#### Re-encryption\n\nThe load balancer decrypts client traffic, inspects it, then re-encrypts it before sending to the backend. This gives you content inspection plus backend encryption, but with double the cryptographic overhead.\n\n#### SSL Passthrough\n\nThe load balancer does not decrypt traffic at all. It routes based on the TLS Server Name Indication (SNI) and forwards encrypted packets directly to backends. You lose the ability to do content-based routing, but traffic is encrypted end-to-end.\n\n### Best Practices\n\nIf you are terminating SSL at the load balancer, follow these security practices:\n\n- **Use TLS 1.2 or higher.** Disable older protocols (SSL 3.0, TLS 1.0, TLS 1.1) which have known vulnerabilities.\n- **Choose strong cipher suites.** Prefer ECDHE for key exchange (provides forward secrecy) and AES-GCM for encryption.\n- **Enable HTTP/2.** Modern browsers support it, and it multiplexes requests over a single connection, improving performance.\n- **Use OCSP stapling.** The load balancer fetches certificate validity proofs and includes them in the TLS handshake, so clients do not need to make separate OCSP queries.\n- **Automate certificate renewal.** Use tools like Let's Encrypt with automatic renewal so you never have an expired certificate.\n\n---\n\n# 7. Follow-ups\n\n## 7.1 Handling Load Balancer Failures\n\nWe covered high availability at a conceptual level in section 4.3. Now let's go deeper into the failure scenarios you need to handle and the mechanisms that make failover work.\n\nThe load balancer sits on the critical path for all traffic. If it fails and you do not have a backup, your entire service goes offline. This is why high availability is non-negotiable for production load balancers.\n\n### How Failures Are Detected\n\nBefore you can recover from a failure, you need to detect it. Load balancer nodes monitor each other using:\n\n- **Heartbeat messages:** LB nodes exchange \"I'm alive\" messages every 1-3 seconds. If a node stops responding, it is presumed dead.\n- **Health checks:** The same health checking mechanism used for backends can monitor LB nodes.\n- **Shared storage heartbeats:** Write timestamps to a shared location (etcd, Redis) to detect liveness.\n\nThe detection threshold is a balance: too sensitive and you get false positives (declaring a node dead when it only had a brief network interruption), too slow and you extend your outage.\n\n### Failover Strategy 1: VRRP (Active-Passive)\n\nVRRP (Virtual Router Redundancy Protocol) is the industry standard for IP failover within a single network segment. Two load balancers share a Virtual IP address, but only one responds to traffic at a time.\n\n\n```mermaid\nflowchart LR\n subgraph \"After Active Fails\"\n direction LR\n C2[Clients]:::rose --> VIP2[VIP: 10.0.0.1]:::primary\n LB2A[\"Former Active
(failed)\"]:::red\n VIP2 --> LB2B[\"Standby
(claimed VIP)\"]:::green\n end\n\n subgraph \"Normal Operation\"\n direction LR\n C1[Clients]:::rose --> VIP1[VIP: 10.0.0.1]:::primary\n VIP1 --> LB1A[\"Active LB
(owns VIP)\"]:::green\n LB1A <-.-> |\"Heartbeat\"| LB1B[\"Standby LB\"]:::orange\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n#### How VRRP failover works\n\n1. Both load balancers run VRRP. The active one (higher priority) \"owns\" the VIP and responds to ARP requests for it.\n2. The standby sends heartbeats to the active every second. As long as heartbeats are acknowledged, the standby stays passive.\n3. If the standby misses 3 consecutive heartbeats, it assumes the active is dead. It broadcasts a gratuitous ARP saying \"I am now 10.0.0.1\" and starts accepting traffic.\n\n**Failover time:** 1-3 seconds\n\n**Trade-off:** Simple and reliable, but half your capacity sits idle during normal operation.\n\n### Failover Strategy 2: DNS-Based\n\nMultiple load balancers have their own IPs. DNS health checks remove failed ones from rotation.\n\n\n```mermaid\nflowchart TB\n\n subgraph \"After Node 1 Fails\"\n DNS2[DNS Server]:::secondary\n DNS2 -.->|\"Removed\"| LB2A[LB Node 1]:::red\n DNS2 -->|\"10.0.0.2\"| LB2B[LB Node 2]:::green\n end\n\n subgraph \"Normal\"\n DNS1[DNS Server]:::secondary\n DNS1 -->|\"10.0.0.1\"| LB1A[LB Node 1]:::green\n DNS1 -->|\"10.0.0.2\"| LB1B[LB Node 2]:::green\n end\n\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Failover time:** Depends on DNS TTL. With a 60-second TTL, failover can take 1-2 minutes as cached DNS entries expire.\n\n**Trade-off:** Works across data centers (unlike VRRP), but failover is slower.\n\n### Failover Strategy 3: Anycast\n\nMultiple load balancers across the world share the same IP address. BGP routing automatically directs traffic to the nearest healthy one.\n\n\n```mermaid\nflowchart TB\n subgraph \"Global Anycast\"\n C_US[US Client]:::rose\n C_EU[EU Client]:::rose\n C_AS[Asia Client]:::rose\n\n IP[Same IP: 1.2.3.4]:::primary\n\n LB_US[LB in US]:::green\n LB_EU[LB in EU]:::green\n LB_AS[LB in Asia
FAILED]:::red\n\n C_US & C_EU & C_AS --> IP\n IP --> LB_US & LB_EU\n IP -.->|\"Route withdrawn\"| LB_AS\n end\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nWhen the Asia LB fails, it stops announcing its BGP route. Traffic from Asia clients automatically routes to the next nearest healthy LB (US or EU).\n\n**Failover time:** Seconds (depends on BGP convergence)\n\n**Trade-off:** Requires BGP expertise and coordination with network providers. Used by Cloudflare, Google, and other global-scale services.\n\n### What Happens to Connections During Failover?\n\nWhen a load balancer fails, connections to it are lost. There is no practical way to transfer an open TCP connection from one machine to another. Clients will see connection resets.\n\nYou can minimize the impact:\n\n\n| Strategy | Benefit |\n|----------|---------|\n| Fast failover (VRRP/Anycast) | Reduces the window where new connections fail |\n| Client retry logic | Applications reconnect automatically |\n| External session store | Users do not lose session data |\n| Short connection timeouts | Clients detect failure faster |\n\n\n**Recommendation:** Design for stateless failover. Assume connections will be dropped on failure, and build applications that handle retries gracefully. The complexity of stateful failover (synchronizing connection state across LB nodes) is rarely worth it.\n\n---\n\n# Quiz","pageType":"system-design-interviews"}

Get Premium