Any system with shared capacity eventually meets a caller that sends too much traffic. It might be an attacker, a buggy retry loop, or one customer having a very busy day. Rate limiting is how we put a sensible speed limit on that traffic.

A rate limiter answers a simple question for each request: should this request start now, wait, or be rejected?

Good rate limits do more than block abuse. They protect shared capacity, keep one tenant from slowing everyone else down, reduce retry storms, and put guardrails around expensive operations. The thing you limit depends on the system: requests per minute for an API key, failed login attempts for an account, or tokens for an AI API.

When a client goes over the limit, the system usually rejects the request quickly with 429 Too Many Requests and tells the client when to try again. The hard part is choosing what to count, who to count it for, how to count it, and what to do when the limiter itself has trouble.

This chapter covers the common rate-limiting algorithms and how to choose limits.

Rate limits exist because backend capacity is finite.

Without limits, one caller can use up thread pools, database connections, cache bandwidth, queue space, third-party API allowance, or model-serving capacity. The caller may be malicious, but it may also be a normal client with a retry bug.

Rate limiting helps with five practical problems.

Protecting Shared Infrastructure

One noisy tenant should not make the API slow for everyone else.

If tenant acme accidentally starts polling GET /orders thousands of times per second, the system should stop most of that traffic near the edge instead of letting it overwhelm the order service and database.

Preserving Fairness

Rate limits help divide capacity across users, tenants, plans, regions, or API keys.

A free plan might get 60 requests per minute. An enterprise plan might get 5,000 requests per minute. Internal services may have separate limits tied to the capacity they are allowed to consume.

Controlling Cost

Some requests are much more expensive than others.

One POST /generate-report request may scan millions of rows. One LLM request with a huge prompt may cost more than hundreds of small metadata reads. Treating every request as equal is often wrong.

Reducing Retry Storms

When a dependency is slow, clients often retry. If every retry is allowed right away, the system can turn a small incident into a much larger one.

Rate limits, retry budgets, and backoff rules keep failures from multiplying.

Making API Behavior Explicit

Clear limits are part of the API contract. Clients can pace themselves, batch work, cache responses, and avoid unnecessary errors.

The first design decision is the unit of work.

Request count is easy to explain, but it is often too blunt. Real systems usually combine a few limits. For an AI chat API, you might limit requests per minute per user, active generations per organization, input tokens per minute per model family, daily spend per workspace, and tool calls that reach outside systems.

The rule of thumb is simple: limit the resource that can actually run out.

Rate limiting can happen at several layers.

Each layer has a different job.

The best systems use layers. The edge blocks obvious abuse. The gateway enforces API policy. Services protect expensive business operations. Workers protect the systems they call.

Every rate limiter needs a few pieces of state.

Key

The key identifies who or what is being limited. Common keys include IP address, user ID, organization or tenant ID, API key, route, model name, or a combination such as tenant_id + endpoint + model.

Choose keys carefully. IP-only limits can hurt many real users who share one IP, and attackers can often move across many IPs. User-only limits can miss traffic before login. Tenant-level limits often protect shared capacity better than only per-user limits.

Limit

The limit defines how much work is allowed. Typical limits look like 100 requests per minute, 10 concurrent exports, 1,000 write operations per hour, or 2 million tokens per day.

Window

The window is the time period used for the decision. Windows can be per second, per minute, per hour, or a rolling 24 hours. Short windows protect immediate capacity. Long windows are better for quota or pricing rules.

Decision

For each request, the limiter returns one of three outcomes: allow, reject, or sometimes delay. Most API gateways reject immediately when the limit is exceeded. Some internal systems queue work or drop lower-priority work instead.

The Decision Flow for One Request

Those four pieces come together the same way on every request, no matter which algorithm sits underneath:

A concrete allow: a request to /api/search for tenant acme derives the key acme:/api/search, the store shows 87/100 for the current minute, 87 is below 100, so the limiter allows it, increments to 88, and returns X-RateLimit-Remaining: 12. The first request after the counter reaches 100 is rejected with 429 until the window resets.

There is no single best algorithm. Each one makes a different tradeoff between accuracy, memory use, burst handling, and operational complexity.

The code examples below are small single-process versions for learning. In production, the same updates usually happen inside one safe datastore operation, a gateway plugin, a local limiter, or a dedicated rate-limit service.

Fixed Window Counter

The fixed window counter is the simplest approach. It splits time into fixed buckets and counts requests in the current bucket.

Example policy:

If the counter reaches 100, new requests are rejected until the next minute starts.

How It Works

Example

Here is what the limiter's state might look like for one user partway through a window, just below the limit.

Code Example

The implementation below tracks a counter per key and window, increments it on each request, and rejects once the count passes the limit.

Use fixed windows when the limit is simple and brief bursts around the edge of a window are acceptable.

Sliding Window Log

The sliding window log stores the timestamp of each request. For every new request, it only counts timestamps that are still inside the rolling window.

Example:

How It Works

Code Example

This limiter keeps a list of timestamps per key, drops old timestamps, and allows the request only when the remaining count is still under the limit.

Use sliding logs when accuracy matters and each key has modest traffic, such as password reset attempts or sensitive account operations.

Sliding Window Counter

The sliding window counter gives you a rolling-window estimate without storing every timestamp.

It keeps two counts: the current fixed window and the previous fixed window. Then it counts only the part of the previous window that still overlaps the last 60 seconds.

Example:

The 75% comes from the overlap. If 25% of the current minute has passed, then 75% of the previous minute is still inside the last 60 seconds.

Code Example

The code below stores only the current and previous window counts per key, then uses the overlap to estimate the rolling total.

Use sliding window counters when you need smoother behavior than fixed windows without storing every request timestamp.

Token Bucket

The token bucket algorithm allows controlled bursts.

Think of the bucket as saved permission to make requests. Tokens refill at a steady rate up to a maximum size. Each request spends one or more tokens. If the bucket has enough tokens, the request is allowed. If not, the request is rejected or delayed.

Example:

This lets a client send up to 100 requests after being idle, then continue at about 10 requests per second.

Code Example

The implementation refills tokens based on how much time has passed since the last request, then allows the call only if there are enough tokens to spend.

Use token buckets for public APIs where short bursts are fine, but long-running traffic must stay within a steady average rate.

Leaky Bucket

The leaky bucket algorithm smooths traffic by letting work leave at a constant rate.

Think of it as a small queue. Requests enter the bucket. Work leaves the bucket at a fixed pace. If the bucket is full, new requests are rejected.

Code Example

The code models the bucket as a queue that drains at a fixed interval. It allows a request only when there is room left in the queue.

The example above uses a non-blocking style: each accepted request reserves a future slot, and the limiter rejects when too many slots are waiting. Another version would make the caller wait until the next slot is available. Both are called "leaky bucket" in practice; choose the one that fits whether your stack can pause callers.

Use leaky buckets when smooth traffic matters more than allowing bursts, especially in front of fragile services or third-party APIs.

Tracing a Burst Through the Algorithms

The differences between these algorithms are easiest to see by sending the same traffic through each one. Take a limit of 100 requests per minute. Now imagine a client sends 80 requests in the last second of one minute, then another 80 in the first second of the next minute. That is 160 requests in about two seconds.

Same traffic, three different answers: fixed window lets all 160 through, while the sliding counter and token bucket hold the client closer to the intended 100. That is why boundary bursts matter when you pick an algorithm.

Each algorithm balances accuracy, memory, and burst handling differently. No single one wins everywhere. Use this table as a practical starting point.

As a default:

Rate limiting becomes harder when traffic is handled by many gateway or service instances.

If each instance keeps only its own counters, a client can go over the real limit by spreading requests across instances.

That means the instances need some way to agree on the count.

Centralized Counter Store

Store counters in Redis, Memcached, DynamoDB, FoundationDB, or another fast shared system.

The limiter must update counters safely so two requests cannot both sneak through at the same time. A common Redis implementation uses INCR plus EXPIRE for fixed windows, or Lua scripts for token buckets and sliding counters.

The tradeoff is latency and availability. Every request may need a network call to the limiter store.

Local Limiters With Periodic Sync

Each node gets a local budget and periodically syncs with a central service.

This reduces latency and avoids depending on a central store for every request, but the global limit becomes an estimate. It works well when small overages are acceptable.

Sharded Limiters

Route each rate-limit key to a specific shard.

This can scale well, but very busy tenants or popular API keys can overload one shard. You may need to split busy keys, change limits dynamically, or give large tenants special handling.

Multi-Region Limits

Global limits across regions are expensive to enforce exactly. Asking regions to coordinate on every request adds latency and can fail when regions cannot talk to each other. Common approaches are to enforce per-region limits and accept small differences, give each region part of a global budget, pin each tenant to a home region, or use strict global limits only for costly operations.

Exact global rate limiting is rarely worth the operational cost for ordinary request counts. It may be worth it for money, quota, abuse, or operations with legal or compliance requirements.

Rate limiters sit on the request path, so their failure behavior matters.

Race Conditions

The check and increment must happen as one safe operation.

This is wrong:

Two requests can read the same count and both pass. Use one safe update operation or a transaction.

Missing Expiration

Counters need expiration times. Without them, every user, route, and window key stays in memory forever.

Hot Keys

Large tenants, popular API keys, or many users behind the same IP can create hot keys. A single counter can become the bottleneck.

Plan separately for many small keys and a few very busy keys.

Clock Skew

Algorithms based on wall-clock time can behave strangely when nodes disagree about time.

Prefer time from the limiter store when possible. For local elapsed-time calculations, use a clock that only moves forward.

Fail Open vs Fail Closed

If the limiter store is unavailable, should requests be allowed?

There is no universal answer.

Make this decision on purpose. Do not let random timeout behavior decide it for you.

Retry Storms

A rejected request can create more traffic if clients retry immediately.

Return clear retry information and design clients to wait longer between retries, with a little randomness so they do not all retry at the same moment.

When an HTTP client exceeds a limit, return 429 Too Many Requests.

Include a response body that explains the limit in a stable format clients can parse.

Example:

The Retry-After header tells the client how long to wait before trying again.

Many APIs also return headers that describe the limit. Newer HTTP guidance defines structured fields named RateLimit and RateLimit-Policy, but exact syntax can vary by platform and version:

Many production APIs still use simpler RateLimit-* fields:

Older APIs may use X-RateLimit-* variants.

Header conventions still vary across platforms. Document exactly what your API sends, what each value means, whether failed requests count, and whether limits are per user, tenant, route, API key, or some combination.

Clients should treat headers as helpful guidance, not a promise that nothing will change. Servers may lower limits during incidents, deploy new policies, or drop load for reasons outside the normal quota window.

AI workloads make rate limiting more complicated because request count does not tell you much about cost.

Two requests can have wildly different costs:

Useful AI rate limits often include input tokens per minute, output tokens per minute, active generations, requests per model pool, tool calls per user, vector search queries per tenant, daily or monthly spend, and model allowlists by plan.

Streaming also changes behavior. A request may be accepted quickly but hold a connection and model slot for a long time. Limit both start rate and active streaming requests.

Do not log full prompts or responses just to debug rate limits. Track token counts, model, tenant, request ID, and policy decision. Keep sensitive content out of logs by default.

Before adding a rate limit, answer these questions:

Rate limits should be visible in dashboards and alerts. Track allowed requests, rejected requests, limiter latency, hot keys, store errors, and the tenants or API keys that hit limits most often.

Rate limiting protects shared capacity by deciding how much work each caller can send. The right unit is not always request count; tokens, cost, bytes, concurrency, or weighted requests often match the real bottleneck better.

Token bucket is a strong default for public APIs because it allows short bursts while controlling the long-term rate. Fixed windows are simple but can allow bursts at window boundaries. Sliding logs are accurate but expensive.

Distributed rate limiting adds a few important decisions: update counters safely, decide what happens when the limiter fails, and plan for hot keys and multi-region differences. For HTTP APIs, return 429 Too Many Requests, include Retry-After when it helps the caller, and document the rate-limit headers clearly.