As soon as you run more than one instance of a service, you face a simple question: which instance should handle the next request?

A load balancer answers that question. It sits between clients and backend services. Clients connect to one stable address, and the load balancer forwards each request to a backend that should be able to handle it.

This one piece of infrastructure often handles a lot of production work: scaling, failover, TLS termination, connection draining, traffic rules, and observability.

The goal is not to split requests perfectly evenly. The goal is to keep traffic moving to backends that are healthy and have capacity. If a backend is overloaded or failing health checks, it should stop receiving new work, even if that makes the traffic split uneven.

The diagram below shows the basic shape: clients use one stable address, and the load balancer spreads requests across a pool of backends.

This chapter explains what load balancers do, where they sit in a system, and the production problems they help solve.

Load balancers solve several practical problems:

The main design benefit is separation. Clients talk to the load balancer. Backends can be added, removed, replaced, or marked unhealthy behind it.

Load balancers can appear at several layers of a system.

Common placements:

Do not assume a system has only one load balancer. Large systems often use several, each with a different job.

The most important distinction is whether the load balancer understands the application protocol.

Layer 4 Load Balancers

A Layer 4 load balancer works at the transport layer. It sees IP addresses, ports, TCP, UDP, and connection state. It does not read HTTP paths, headers, cookies, or request bodies.

Layer 4 load balancers are common for:

They are usually faster and simpler than Layer 7 load balancers, but they know less about each request.

Layer 7 Load Balancers

A Layer 7 load balancer understands the application protocol, most commonly HTTP.

It can route based on:

Layer 7 load balancers are common for web apps, APIs, microservices, and edge gateways.

That control has a cost. Reading application traffic gives you smarter routing, but it also adds more configuration, more ways to misconfigure the system, and more CPU work.

Many production stacks use both: Layer 4 near the edge for raw throughput, and Layer 7 behind it when routing depends on request details.

A typical request through a Layer 7 load balancer looks like this:

Important details:

Health checks decide whether a backend should receive new traffic.

Simple checks only prove that a process accepts connections. Better checks prove that the backend is ready to serve real work.

Health checks should be fast, stable, and meaningful. They should also use thresholds:

Without thresholds, a brief network blip can bounce an instance in and out of rotation. With thresholds that are too slow, failed instances keep receiving traffic for too long.

Removing a backend from rotation is not the same as killing it.

During a deployment, the safer flow is:

This matters for HTTP requests, WebSockets, gRPC streams, file uploads, and token-streaming AI responses. If you kill a backend while it is streaming a response, the user sees the failure.

Good systems combine load balancer draining with application shutdown hooks. The application should stop accepting new work before the process exits.

Many load balancers terminate TLS. The client establishes HTTPS with the load balancer, and the load balancer forwards traffic to the backend.

There are three common patterns:

TLS termination keeps certificate management in one place and enables Layer 7 routing. Pass-through keeps traffic encrypted all the way to the backend, but limits application-aware features.

Session affinity, often called sticky sessions, routes the same client to the same backend.

Common methods:

Affinity is useful when backends hold local state, keep warm caches, or handle long-lived sessions.

It can also be a warning sign. If sticky sessions are hiding state that could live in a shared database, cache, or token, the system will usually be harder to scale and recover.

Use affinity deliberately. Document what breaks if a user moves to another backend.

The load balancer needs a rule for picking a backend. Common algorithms include:

No algorithm can fix unhealthy backends, missing timeouts, bad retries, or not enough capacity.

Load balancers improve availability, but many services may depend on them. Design for the ways they can fail.

The Load Balancer as a Bottleneck

A load balancer has limits: connections, packets per second, requests per second, TLS handshakes, memory, and routing-rule cost.

Track those limits. Do not discover them for the first time during an incident.

Bad Health Checks

If health checks are too weak, broken backends stay in rotation. If they are too strict, healthy backends get removed during harmless dependency blips.

Health check design is production logic, not boilerplate.

Retry Storms

Retries at the load balancer can make outages worse. If every failed request is retried three times against an already overloaded pool, the load balancer adds more pressure exactly when the system is weakest.

Retries need budgets, timeouts, backoff, and awareness of whether the request is safe to repeat.

Long-Lived Connections

WebSockets, HTTP/2 streams, gRPC streams, and AI token streams do not behave like short HTTP requests. They hold connections for longer and make deployments harder.

Use draining, max connection age, keepalive settings, and clear reconnect behavior.

Uneven Request Cost

One request may take 2 ms. Another may run a 30-second report or a long model inference. Balancing request counts is not the same as balancing work.

For expensive workloads, route using application-level signals such as queue depth, model type, tenant quota, or estimated cost.

Today, "load balancer" means more than one appliance in front of a web tier. Kubernetes, service meshes, and AI inference systems all add their own balancing layers with different jobs.

The sections below show how the idea appears in modern systems.

Kubernetes

In Kubernetes, "load balancer" can mean several things:

Be precise about which layer you mean. A Kubernetes Service, an ingress controller, and a cloud load balancer solve different problems.

Service Mesh and Sidecars

Service meshes move some load balancing into sidecars or node proxies. The client-side proxy can choose from discovered endpoints, apply retries, enforce mTLS, and collect logs, metrics, and traces.

This is useful, but it increases operational complexity. A bad mesh configuration can break service-to-service traffic just as badly as a code bug.

AI Systems

AI inference traffic stresses load balancers in ways classic HTTP traffic does not. Responses may stream tokens for tens of seconds, sometimes minutes. That makes simple request-count metrics misleading and makes connection draining very important.

A normal Layer 7 load balancer is a good front door for TLS, authentication, and ingress. But model-aware scheduling, such as GPU placement, queue depth, and model availability, belongs in a dedicated inference gateway behind it.

The rules below turn the chapter into practical decisions you can apply when designing a load-balanced service.

A load balancer gives clients one stable endpoint while backend instances scale, fail, deploy, and recover behind it. That separation is the core idea: clients keep using the same address while backends change.

Layer 4 and Layer 7 load balancers solve different problems. Layer 4 works with network and connection details. Layer 7 understands application details such as HTTP paths, hosts, headers, and methods.

Health checks are part of system correctness. A bad health check can send traffic to broken instances or remove healthy ones. Connection draining matters because stopping new traffic before shutdown avoids many user-visible failures.

The algorithm is only one piece. TLS termination, retries, session affinity, and routing rules are all design choices with tradeoffs. Timeouts, capacity, health, and observability matter just as much.

Modern systems usually run several load-balancing layers together: DNS, edge, regional, service mesh, and application routing.

The goal is simple: give clients a stable way in, send work only to backends that can handle it, and make failures predictable instead of chaotic.