A load balancer has to pick one backend for each incoming request. The rule it uses for that choice is called a load balancing algorithm.

That rule matters. It changes how the system behaves when traffic is normal, when one backend slows down, and when traffic spikes.

The right algorithm depends on the workload. A static web server, a WebSocket service, a cache cluster, and an AI inference gateway should not all use the same strategy.

A simple algorithm may spread request counts evenly. But equal request counts do not always mean equal work. One request might return a small image. Another might run a 30-second report or occupy a GPU.

A good algorithm sends work to healthy backends that have enough capacity, while keeping latency, correctness, and failure behavior under control.

This chapter explains the algorithms you are most likely to see, from round robin to least connections to consistent hashing, and how to choose one for a real workload.

A load-balancing algorithm can only use the information available where it runs.

This matters because algorithm names can be misleading.

"Least connections" is useful only if connection count tells you something useful about load. "Least response time" helps only if recent latency is a good hint about what will happen next. "Round robin" is fine only when requests and backends are similar.

No algorithm can make up for missing health checks, bad timeouts, retry storms, or not enough capacity.

Round robin sends requests to backends in order, then loops back to the beginning.

When It Works

Round robin is a good default when:

Where It Fails

Round robin ignores current load. If one backend gets slow because of garbage collection, a noisy neighbor, repeated cache misses, or a long-running request, round robin keeps sending it traffic.

It also performs poorly when requests have very different costs. Ten small requests and ten "generate a large report" requests are not the same amount of work.

Simple Implementation

A round robin picker is just a counter and a remainder operation. The code below shows the same idea across six languages.

In production, the list should contain only healthy backends that are ready for traffic.

Weighted round robin gives some backends more turns than others.

If backends have weights [5, 1, 1], the first backend should receive roughly five times as many picks as either of the others.

When It Works

Use weighted round robin when:

Where It Fails

Weights are fixed unless you update them. They may not match what is happening right now.

A backend with weight 5 may still be overloaded if it receives expensive requests. A backend with weight 1 may be underused if its requests are cheap.

Weighted round robin knows only the capacity you configured ahead of time. It does not automatically understand live load.

Smooth Weighted Round Robin

A simple weighted round robin implementation can create bursts. A smoother version spreads picks across the cycle.

This is still not a replacement for live load feedback.

Least connections sends new work to the backend with the fewest open connections.

In HTTP and gRPC systems, least requests is often more useful. It tracks active in-flight requests or streams instead of just open connections.

When It Works

Use least connections or least requests when:

This is a better fit than round robin for WebSockets, long polling, gRPC streams, and APIs where some requests take much longer than others.

Where It Fails

Connection count is not always the same as load.

One idle WebSocket is not the same as one busy gRPC stream. One image request is not the same as one expensive search query.

If a backend is slow, it may build up more active requests, and least requests will naturally send less traffic to it. That is useful. But if the measurements jump around too much, traffic can swing back and forth between backends.

Basic Implementation

To pick the least-loaded backend, keep a count of active requests per backend and choose the lowest count. The code below shows that bookkeeping, including the increment and release steps.

Production implementations must safely handle backend removal, failed requests, timeouts, and concurrent updates.

The power of two choices, often called P2C, picks two random healthy backends and sends the request to the less loaded one.

It works surprisingly well for such a simple idea. It avoids scanning every backend, but still avoids the worst random choices.

Where the Differences Show Up

For large backend pools, scanning every backend for every request can be expensive or hard to keep in sync. P2C gives most of the benefit while checking only two backends.

Many modern proxies use variants of this idea for least-request balancing.

Simple Implementation

P2C is simple in code: sample two backends at random, then return whichever has fewer active requests.

If backend weights differ, compare an adjusted score instead of raw active counts.

Random selection chooses a healthy backend at random.

That may sound too simple, but it can be useful in distributed systems. It needs very little coordination and avoids every load balancer making the exact same choice at the exact same time.

When It Works

Use random selection when:

Simple Implementation

There is not much to random selection in code: choose an index in the backend list with a random number.

Where It Fails

Pure random can create unlucky streaks. It also ignores capacity and active load. P2C is often a better default when you can track a simple load signal.

Least response time routes traffic to the backend with the best observed latency.

When It Works

Use it when latency varies by backend and recent latency is a useful hint about future latency.

Examples:

Where It Fails

Least response time can be unstable.

If one backend receives fewer requests, its latency data may be old. If one backend gets a run of cheap requests, it may look faster than it really is. If every load balancer chases the fastest backend at the same time, they can all pile onto it and make it slow.

Use smoothing, minimum sample counts, outlier detection, and circuit breakers. Do not route all traffic based on one recent response.

Smoothed Latency

The implementation below uses a moving average for each backend's latency. That way, one unusually slow or fast response does not swing routing decisions by itself.

This example is intentionally incomplete. A production version needs exploration, which means it should occasionally try new or quiet backends so their latency data does not go stale.

Hash-based routing sends the same key to the same backend while the backend set is stable.

The key might be:

Source IP Hash

Source IP hash is simple: hash the client IP and map it to a backend.

It can help with sticky sessions, but it has problems:

Source IP hash is a blunt tool. Cookie-based, header-based, or application-key hashing is often better.

Consistent Hashing

Consistent hashing reduces how many keys move when backends are added or removed.

Instead of computing hash(key) % number_of_backends, the load balancer maps backends and keys onto a hash ring. When one backend changes, only part of the key space moves.

Rendezvous hashing is a simpler alternative with the same basic goal: keep key-to-backend mapping stable. It scores every backend for a key and picks the highest score. When a backend is added or removed, only the keys that prefer that backend move.

Consistent hashing is useful for:

Maglev and Rendezvous Hashing

Modern proxies may use variants such as Maglev hashing or rendezvous hashing. They serve the same broad goal: keep key-to-backend mapping stable while limiting how much traffic moves when the backend set changes.

Use these when key stability matters. Be careful when the workload has hot keys. One popular key can overload one backend unless you have a plan to split or shed that traffic.

Locality-aware routing prefers nearby backends: same zone, same region, same rack, or same cluster.

This reduces latency and cross-zone cost. It can also keep failures smaller when a dependency has regional problems.

Where It Fails

Locality can overload local capacity. If one zone has 40% of clients but only 25% of backend capacity, strict local routing will overload that zone while remote capacity sits idle.

Good locality rules need spillover:

Adaptive routing uses live signals such as queue depth, CPU, error rate, observed latency, or load reports from the application.

This is common in advanced service meshes, RPC frameworks, and inference gateways.

When It Helps

Adaptive routing is useful when simple signals are not enough:

Where It Fails

Adaptive systems can create feedback loops. If every client routes away from a slightly slow backend at the same time, load shifts suddenly and creates a new slow backend.

Use smoothing, caps, outlier detection, slow start, and circuit breakers. Make the algorithm stable before making it clever.

The right algorithm depends less on abstract theory than on the shape of the traffic it serves. This table maps common workloads to a sensible starting point.

In practice, several layers may each apply their own algorithm:

Each layer should use the signals it can actually see.

A good algorithm still fails when the surrounding setup works against it.

Confusing Requests with Work

Equal request count does not mean equal load. Measure CPU, memory, queue time, latency, active requests, and error rates.

Ignoring Slow Start

A newly added backend often has cold caches, empty connection pools, and warmup work such as JIT compilation or model loading. Sending it a full traffic share immediately can hurt it.

Slow start gradually increases traffic to new or recovered backends.

Retrying Without Budgets

Retries can turn a partial outage into a full outage. A load balancer should not blindly retry every failed request.

Retry only when:

Keeping Dead Backends in the Pool

Algorithms assume the backend pool contains only instances that should receive traffic. Health checks and endpoint discovery must remove dead, draining, or overloaded instances.

Using Sticky Routing to Hide Bad State Management

Sticky routing can be valid, but it should not be the only thing keeping the application correct. Backends fail. Deployments happen. Clients move.

If moving a user to another backend breaks the session, write that down as a real reliability risk.

This table gathers every algorithm covered in the chapter into one view. Use it when you need to compare options quickly.

Load-balancing algorithms are rules for choosing a backend. The best rule depends on what the load balancer can see and what the workload needs.

Round robin is simple, not smart. It works best when requests and backends are similar.

Weights express expected capacity, not live load. They help with canaries and mixed-capacity backend pools, but they can drift away from reality.

Least connections and least requests handle uneven request duration better when active-work tracking is accurate. Power of two choices is a strong default for large pools because it balances well without scanning every backend.

Hash-based routing with consistent hashing, Maglev, or rendezvous hashing keeps related traffic near the same backend and moves fewer keys when backends change.

Latency-aware routing needs smoothing so recent measurements do not cause noisy traffic swings. Locality-aware routing needs spillover so nearby capacity is preferred but not overloaded.

AI inference needs application-aware routing. GPU type, model availability, queue depth, tenant quota, and data rules matter more than generic request counts.

In every case, the algorithm is only part of reliability. Health checks, timeouts, draining, slow start, retries, and observability decide how the system behaves under stress.

Choose the simplest algorithm that matches the workload, then prove it with production metrics.