Design an LLM Inference API Platform
An LLM inference API platform hosts language models and lets developers access them over HTTP. A developer sends a JSON request with a model name and some messages, and the platform returns the model’s response, often streaming tokens as they are generated.
Popular Examples: OpenAI API, Anthropic API, Google Gemini API, Amazon Bedrock
Behind that simple interface is a large serving system. A major provider might run dozens of models across tens of thousands of GPUs, shared by thousands of customers who expect fast responses. Those customers send very different workloads. One might submit a 20-token classification request. Another might send a 150,000-token document and ask for a 4,000-token summary. The second request can require thousands of times more GPU work, even though both arrive as a single HTTP call.
This variation changes how we design the platform. Request limits alone cannot protect GPU capacity, and balancing traffic by connection count can leave some replicas overloaded while others have room to spare. Billing also needs to account for partial responses, including tokens generated before a client disconnects.
In this chapter, we’ll design a public LLM inference API similar to the OpenAI and Anthropic APIs. We’ll start with the requirements and architecture, then examine token-based quotas, priority tiers, scheduling across GPU pools, prompt caching, and accurate metering.