The basic loop is to take the code around the cursor, send it to a model, and show the result. The constraints around that loop are tight. A suggestion that arrives after the developer has already typed past that point can't be used, and one that ignores a helper function defined in another file is wrong. With millions of developers each sending a request every few seconds, even a small cost per request adds up to a large GPU bill.

In this chapter, we'll design a code completion assistant. We'll start with the requirements and high-level architecture, then go deep on the end-to-end latency budget, selecting context within a small window, fill-in-the-middle, and measuring acceptance.

Let's start by clarifying the requirements.

Code completion means different things in different products, from finishing the current line to a chat panel that answers questions about code. So the first step is to pin down the scope.

Here is how the discussion might go in an interview:

The conversation gives us the constraints we need. Let's write them down as functional and non-functional requirements.

Functional Requirements

Non-Functional Requirements

Quality Bar

The main quality signal is whether developers keep what they accept. A suggestion that gets accepted and then deleted ten seconds later wasted the developer's time. We'll track acceptance rate along with how much accepted code is still in the file a few minutes later. Section 7 covers these metrics in detail.

Request Volume

Assume 4 million daily active developers. During the peak hour, about 1 million of them are typing at the same time.

A developer generates far more keystrokes than requests because the client waits for a short pause before sending one. Even so, an active developer triggers about 8 requests per minute.

Many of these requests are cancelled before they finish because the developer kept typing. Cancelled requests still consume compute up to the point they're stopped, so we plan capacity for the full volume.

Tokens per Request

A typical request contains:

Input is more than 100 times larger than output. That's the opposite of a chat product, and it means prefill dominates the compute cost.

Consecutive requests from the same developer are nearly identical. Between two requests, the developer typed a few characters, so most of the prompt is the same. With a prompt layout designed for reuse (covered in Section 7), the model server can cache the attention state for the unchanged part of the prompt. If 70% of input tokens are cache hits, each request needs roughly 900 new tokens of prefill.

GPU Estimate

Assume a code model with about 7 billion parameters running on H100 GPUs. A forward pass costs about 2 × parameters FLOPs per token, so 14 GFLOPs per token. At a realistic utilization of around 40%, one H100 can process about 25,000 to 30,000 prefill tokens per second for this model.

Decode adds more GPUs on top of that, because decode runs at a lower utilization than prefill. A rough total is 5,000 to 6,000 GPUs at peak, spread across several regions. Traffic follows working hours, so the average over a day is closer to half the peak.

At about $2.50 per GPU-hour, with an average of 3,000 GPUs running, that comes to about $5.4 million per month, or roughly $1.35 per active developer per month.

Two things stand out. Without prefix caching, each request needs 3,000 prefill tokens instead of 900, which more than triples the prefill fleet. And every request the client avoids sending, through debouncing or local caching, saves GPU time directly.

Storage

We don't store code for enterprise users, so most of the storage goes to telemetry. With about 3 billion telemetry events per day (shown, accepted, rejected, retained) at around 200 bytes each, that's about 600 GB per day. Raw events can be kept for 30 to 90 days, with aggregated metrics stored longer.

Completion API

The suffix carries the code after the cursor, so the model can produce code that fits what comes next. The snippets are pieces of code from other files, chosen by the client. The session_id lets the router send all requests from one editor session to the same model server so they can reuse cached prefixes.

The client keeps a long-lived HTTP/2 connection open to avoid paying for a TCP and TLS handshake on each request. Cancellation uses the HTTP/2 stream itself. When the client cancels a request, it resets the stream, and the server stops generating for that request. We don't need a separate cancel endpoint.

The response is not streamed to the editor. A suggestion of 20 tokens finishes in about 100 milliseconds, and ghost text that grows character by character flickers and distracts the developer. Multi-line suggestions can use streaming internally so the server can stop early at a block boundary.

Telemetry API

The client batches events and sends them every few seconds. The retained event records how much of the accepted text still exists after a fixed delay. It is measured on the client using edit distance, so no code has to leave the machine for this measurement.

We'll build the design one requirement at a time, starting with the simplest version that works.

Step 1: The Basic Request Path

The simplest version has the editor plugin send the code around the cursor to a completion service, which formats a prompt and calls a model.

The gateway authenticates the developer and applies rate limits. The completion service builds the prompt, calls the model, and returns the suggestion.

This version works for a demo, but it sends a request on every keystroke, only sees the current file, and shows slow responses after the developer has already typed past them.

Step 2: Doing More Work in the Editor Plugin

Much of this system's work happens in the editor plugin, and that's deliberate. The plugin has the developer's files, open tabs, and language server data available instantly, with no network round trip.

The plugin takes on four jobs:

For example, if the model suggests lookup_discount(code) and the developer types look, the remaining up_discount(code) still matches, so the plugin keeps showing it at no cost.

Step 3: A Low-Latency Server

On the server side, the completion service becomes a thin, fast pipeline with a few clear stages:

Step 4: The Telemetry Path

The last piece is the feedback loop. The plugin sends telemetry events to an ingestion service that writes them to an event stream. A stream processor computes live metrics like acceptance rate per model version, and a warehouse keeps events for A/B test analysis.

The Complete Architecture

Putting it all together:

The plugin decides when to send a request and what context to include, the backend builds the prompt and runs the model, and the post-processor cleans up the result. The telemetry path runs separately and never blocks a suggestion.

Compared with a typical chat assistant, this design has no orchestrator making several model calls and no server-side retrieval in the request path. Each would add tens to hundreds of milliseconds that this system can't spare.

Why Not a Frontier Chat Model

A large chat model writes excellent code, but it's a poor fit for inline completion:

A Small, Specialized Code Model

The usual choice is a model in the 1B to 15B parameter range, trained heavily on code and specifically trained for fill-in-the-middle (FIM). Open-weight examples include StarCoder2, Qwen2.5-Coder, DeepSeek-Coder, and Codestral. Some large providers train or distill their own.

A common recipe is to start from a base code model trained with the FIM objective, then fine-tune on examples built the same way the production prompt builder builds them, with cross-file snippets and realistic cursor positions. A larger, slower model can also be distilled into the production model.

Making Inference Fast

Routing Between Single-Line and Multi-Line

The client picks the mode. A cursor in the middle of a line means single-line. A cursor on an empty line inside a block with no body yet means multi-line. Some products send multi-line requests to a larger model, since they're harder to get right and developers may pause longer before them.

What About Fine-Tuning on Customer Code?

Enterprise customers often ask for a model tuned on their own repositories. It helps with internal APIs, but it means training and serving a separate model per customer. Cross-file context in the prompt captures much of the same benefit, since the relevant APIs are usually in nearby files, so per-customer tuning fits best as a premium option for very large codebases.

Telemetry Events

Events are keyed by user_id in the stream so each session's events stay in order. Telemetry never includes code for enterprise users. For individual users who opt in, a sample of prompts and completions is stored separately under strict access control and becomes the main source of new evaluation data.

Local Repository Index

The plugin maintains a lightweight index of the current repository on the developer's machine:

Keeping the index local means lookups take a few milliseconds and only the selected snippets leave the machine. The trade-off is that it only covers what's on disk. Some products add an optional server-side index for very large monorepos, which then needs permission scoping and deletion on request.

Deep Dive 1: The End-to-End Latency Budget

The deadline is the developer's next keystroke, so the total budget is small and each stage gets only a few milliseconds of it.

Here's an illustrative budget for a single-line suggestion at the 50th percentile:

Three decisions follow from this table:

Cancellation

Cancellation saves a large amount of compute and keeps tail latency under control. Here's what happens when the developer keeps typing:

Cancellation has to reach all the way to the model server. If the backend only stops sending the response but the model server keeps decoding, the GPU still does the work and the batch slot stays occupied. Inference servers like vLLM support aborting a request mid-generation, and the completion service should call that abort as soon as the client resets the stream.

Even when cancellation reaches the server in time, the prefill work for request A has already been done. That work isn't wasted, though: request B shares most of its prompt with A, so it can reuse A's KV cache.

Sticky Routing for Prefix Reuse

Prefix caching only helps if consecutive requests from the same session reach the same model server. The router uses consistent hashing on session_id to pick a server. If that server is overloaded, the router sends the request to another server and pays the full prefill cost. This keeps the cache hit rate high without letting one busy developer create a hot spot.

Deep Dive 2: Selecting Context Within a Small Window

The model may support a 16K or 32K token context, but filling it is a bad trade. Prefill time grows with prompt length, and longer prompts don't always help. Completion systems often use a few thousand tokens, for example 2,000 to 8,000, and spend a lot of engineering effort deciding what goes in.

Candidate Sources

The client collects candidates from several sources, roughly in order of value:

Packing the Budget

Each candidate gets a score based on its source and similarity to the code near the cursor. The packer reserves space for the prefix and suffix first, then adds snippets in score order until the budget runs out. Each snippet is labeled with its file path as a comment, so the model sees something like # File: billing/discounts.py followed by the code.

A few practical lessons apply here:

Why Not Embedding Search?

Some products do use embedding retrieval, especially for large repositories. But open tabs, recent edits, and language server definitions already say a lot about what's relevant right now, and they cost almost nothing. Embedding retrieval adds an index to maintain and an embedding call per request, so it's worth adding only where acceptance data shows a gap.

Deep Dive 3: Fill-in-the-Middle

Developers rarely write code strictly at the end of a file. They add a line in the middle of a function, fill in an argument, or write a body between a signature and a return. A model trained only left to right sees the prefix and ignores what comes after the cursor. It might suggest code that duplicates the next line or breaks the closing structure.

How FIM Training Works

FIM training teaches a left-to-right model to use the suffix. During training, a document is split at two random points into a prefix, a middle, and a suffix. The pieces are rearranged so the middle comes last, with special sentinel tokens marking each part:

The model still predicts tokens left to right, but by the time it generates the middle, it has already read both the prefix and the suffix. OpenAI's 2022 paper on FIM training showed that a model can learn this with little to no loss in normal left-to-right ability when a large share of training documents are transformed this way. That's why most modern code models support it.

At inference time, the prompt builder produces the same format, and the model generates the middle until it emits an end-of-middle token. For the discount example, the prompt looks like this, using StarCoder-style sentinel tokens:

The exact sentinel tokens differ between model families, so the prompt builder needs a template for each model it serves.

Prefix-Suffix-Middle vs. Suffix-Prefix-Middle

The order above is called PSM, for prefix, suffix, middle. Some models are also trained on SPM order, where the suffix comes first and the prefix comes right before the middle.

The order matters for caching. When the developer types, the prefix grows at its end. In PSM order, everything after that point, including the whole suffix, changes position, and the cache can't reuse it. In SPM order, the suffix and the start of the prefix stay the same, and only the newly typed characters need prefill. Cross-file snippets go at the very start in both cases, since they change least often. If the model supports SPM well, it's the better choice for a keystroke-driven workload.

Post-Processing the Middle

Even a well-trained FIM model sometimes generates too much. It may repeat the first line of the suffix or continue into the next function. The post-processor handles these cases:

Deep Dive 4: Measuring Acceptance

The Suggestion Funnel

Every suggestion moves through a funnel, and each stage has its own metric:

Limits of Acceptance Rate

Acceptance rate is a useful signal. A 2022 study by GitHub researchers found that it correlated with developers' own sense of productivity more closely than other usage metrics. But it is easy to optimize the wrong way.

Suppose the team adds a filter that only shows suggestions the model is very confident about. Acceptance rate goes up, but the developer now gets far fewer suggestions, and the total amount of useful code written by the assistant goes down. The reverse also happens: very short suggestions like a closing parenthesis get accepted often but save almost nothing.

That's why launch decisions should look at a set of metrics together:

Running Experiments

New models, prompts, and context changes ship through A/B tests, with developers assigned by user ID so each one gets a consistent experience. At this traffic, small changes in acceptance rate, such as half a point, can reach statistical significance within days. Results should be broken down by language and mode, because a change that helps Python single-line suggestions can hurt Go multi-line ones while the aggregate looks fine.

Online metrics only measure changes that are already in front of developers. An offline evaluation pipeline catches regressions before that happens.

Offline Evaluation

Public benchmarks like HumanEval measure whether a model can write a function from a description. That's a different task from completing code in the middle of a real file with cross-file dependencies. A useful offline evaluation set looks like production traffic:

Execution-based scoring is the most trustworthy but only works where tests exist. Edit similarity covers everything but penalizes correct code written differently from the original, so the two complement each other. Only changes that hold or improve these scores move on to an A/B test.

Safety and Policy

Code completion has a narrower set of safety concerns than a chat assistant:

Scaling to 10x

GPU capacity in each region. At 10 times the traffic, the fleet grows to tens of thousands of GPUs, and each region needs enough capacity for its own peak, because sending traffic to a distant region breaks the latency budget. Working hours differ across regions, which helps a little, but capacity planning has to happen per region.

Prefix cache hit rate. More model servers means more chances for a session's request to land on a server without its cache, especially when the router redirects traffic away from busy servers. The cache hit rate should be a top-level metric, because a drop from 70% to 50% increases prefill compute by about two-thirds.

Telemetry volume. Billions of events per day become tens of billions. Sampling the frequent shown and rejected events while keeping every accepted and retained event keeps the pipeline manageable.

Graceful degradation. When capacity is short, the system should shed load in a controlled order: first raise the debounce delay, then disable multi-line suggestions, then reject requests from the lowest-priority tier. The editor still works normally, just with fewer suggestions.

Reducing Cost

The cost comes almost entirely from GPU time spent on prefill. These levers each reduce it:

The first lever can be the largest. If 30% of requests come at moments when developers almost never accept a suggestion, skipping them cuts cost by 30% with little effect on useful output. It still needs to be checked against completed characters per opportunity, because an over-aggressive classifier hurts usefulness the same way an over-aggressive confidence filter does.

A code completion assistant is a latency-bound, high-volume system. Most of the design comes from two constraints: a suggestion has to appear before the developer's next keystroke, and the system has to handle over a hundred thousand requests per second without an unreasonable GPU bill.