Suppose a developer opens an issue that says "The /invoices endpoint returns a 500 when the customer has no billing address. Fix it and add a test." Instead of picking it up themselves, they assign it to a coding agent. Twenty minutes later, a pull request might appear with a two-line fix, a new test case, a passing CI run, and a short description of what changed and why.

A coding agent works differently from code completion. Completion responds in milliseconds, while a coding agent runs for minutes or hours, makes dozens of model calls, and executes code that a model wrote. The hard parts are running that code safely, fitting a large repository into a limited context window, keeping long tasks running through failures, and keeping the developer in control of what gets merged.

In this chapter, we'll design a hosted coding agent. We'll start with the requirements and high-level architecture, then go deep on sandbox design, context management on large repositories, the permission model for commands, and keeping long tasks alive.

Let's start by clarifying the requirements.

The phrase "coding agent" covers everything from an autocomplete plugin to an autonomous engineer, so the first few minutes go into narrowing the scope.

Here is how a requirements discussion might unfold in an interview:

With the scope settled, let's formalize these answers into functional and non-functional requirements.

Functional Requirements

Non-Functional Requirements

Quality Bar

A change can compile and still be wrong, so agree early on how quality is measured. The main product metric is the merge rate, the share of agent pull requests merged with little or no rework. Offline, the agent's change must make the target tests pass on a benchmark of real tasks without breaking existing ones.

Concurrency and token volume drive the sizing in this system. The request rate is low.

Concurrent Tasks

With 100,000 tasks per day and an average run time of 20 minutes:

Each running task holds a sandbox. At 2 vCPUs and 8 GB each, peak demand is about 8,400 vCPUs and 34 TB of memory, plus a warm pool for fast starts.

Tokens per Task

A typical task makes about 40 model calls. The context starts at around 15,000 tokens (system instructions, tool definitions, and the task) and grows as file contents and command output accumulate, averaging about 40,000 tokens per call. Each call produces about 400 output tokens, mostly tool calls and short reasoning.

At an illustrative price of $3 per million input tokens and $15 per million output tokens:

The cached row assumes 90% of each call's input matches the prefix of the previous call and is billed at about 10% of the normal input price, while the new 10% is billed at a 25% premium for writing it to the cache. Pricing varies by provider, but the shape holds: an agent resends almost the same context on every step, so caching cuts the bill by roughly 4x. This number shapes several design decisions later.

Throughput and Storage

Forty-six requests per second is trivial, but 1.9 million input tokens per second is a large commitment from any model provider. The bottleneck is token throughput and provider rate limits.

Compute is cheap by comparison. A 2 vCPU, 8 GB sandbox costs roughly $0.10 per hour, so a 20-minute task spends about $0.03 on compute, around 40 times less than on tokens. Traces of model calls and command output come to about 2 MB compressed per task, or 200 GB per day, which belongs in object storage with a lifecycle policy.

The API is asynchronous. Creating a task returns immediately, and the client follows progress through an event stream.

Create a task

Stream progress

Each event carries a sequence number, so a reconnecting client can pass Last-Event-ID and resume where it left off.

Steer, approve, or cancel

Review comments don't need their own endpoint. A GitHub webhook delivers them to the service, which resumes the original task with the comment as a new message.

We'll build the design one requirement at a time, starting with the smallest version that can complete a task.

Step 1: The Agent Loop

At its core, a coding agent is a loop. The model receives the task and a set of tool definitions, decides which tool to call, the system runs it, and the result goes back to the model. This repeats until the model calls a finish tool or a budget runs out.

A small tool set covers most of the work:

edit_file uses exact string replacement instead of asking the model to rewrite whole files. Rewriting a 2,000-line file to change two lines wastes output tokens and risks silently dropping code, while a targeted edit fails loudly when the target string doesn't match.

The loop runs in an agent worker that pulls tasks from a queue and calls the model through an LLM gateway, which handles provider credentials, rate limits, and usage tracking.

Step 2: Sandboxed Workspaces

The model needs somewhere to run commands, and the agent worker is the wrong place: one bad rm -rf or malicious test script would affect every task on that machine. Instead, each task gets its own sandbox: an isolated environment with a clone of the repository at the base commit, the language toolchains, and a dependency cache. A sandbox service provisions sandboxes from a warm pool, and the worker sends tool calls to an execution API inside the sandbox.

The agent loop, the model API keys, and the GitHub token all stay on the worker. The sandbox holds only the code and whatever the commands produce, so if something inside it goes wrong, there are no credentials worth stealing.

Step 3: Finding the Right Code

A monorepo with several million lines is tens of millions of tokens, far more than any context window. So the model gets search tools and explores the way an engineer would: search for the error message, open the handler, follow the calls, and look at nearby tests. A precomputed repository map, a compact list of files with their main classes and functions, gives it a starting point. We'll dig into this in the deep dives.

Step 4: Verification

Running the code lets the agent check its own changes. With run_command, the model runs the relevant tests, reads the failure, fixes the code, and runs them again. Before opening a pull request, the system also runs the repository's linters and test suite outside the model's control. If they fail, the output goes back to the model as one more step.

Step 5: Developer Control

An event service publishes each step to subscribers over Server-Sent Events. A policy engine checks every tool call, and when an action needs approval, the approval service pauses the task and notifies the developer.

When the agent finishes, a Git service outside the sandbox pushes the branch and opens a pull request using a short-lived token that can only push to branches with the agent's prefix, such as agent/. It can't merge, and branch protection rules on main apply as they would to any human.

Step 6: Handling Long Tasks

A two-hour task can be interrupted by a worker deploy, a node failure, or a provider outage. The worker writes every model response and tool result to an append-only event log before moving on, so another worker can rebuild the task from the log and continue. We'll cover the details in the deep dives.

The Complete Architecture

Here's how all the pieces fit together:

Let's trace the invoice bug through this design:

Every step lands in the event log, which feeds the progress view, crash recovery, and evaluation.

Model Choice

The main loop needs a frontier model with strong tool use and long-context reasoning. Weaker models are more likely to loop, edit the wrong file, or declare victory before running the tests. A cheaper model that needs twice as many steps or fails half the tasks can end up costing more per merged pull request, so compare models on cost per merged pull request rather than price per token.

Smaller models still handle the supporting work:

Prompting, Retrieval, or Fine-Tuning

Start with a capable general model, a well-designed system prompt, and good tools. Recent frontier models are trained on agentic coding tasks, and a fine-tune may need redoing when a new base model ships.

Repository knowledge comes from the agent's own search and from an instructions file, such as an AGENTS.md kept in each repository, that says how to build the project, which test commands to run, and which conventions to follow. It's cheap customization that lives in version control next to the code.

Pinning and Fallbacks

Pin the model version for the whole task. Switching models mid-run changes behavior in ways that are hard to debug and invalidates the prompt cache. During an outage, fail over to the same model in another region first, and only fall back to a different model for new tasks, using one that has passed the same evaluation suite.

The core tables look like this:

The event log is the source of truth: task status, cost totals, the progress stream, and crash recovery all derive from it. Large outputs, like a 50,000-line test log, go to object storage, and the event stores a pointer plus the truncated version the model actually saw. That exact model input is what answers "why did the agent delete that file?" during debugging.

1. Sandbox Design

The sandbox runs code nobody reviewed: commands the model wrote, plus build scripts and tests from the repository that might be buggy or malicious. It needs strong isolation, fast startup, and a controlled path to the network.

Isolation Options

For a multi-tenant service running untrusted code, plain containers aren't enough. MicroVMs are the strongest default: each task gets its own kernel, and Firecracker's small per-VM overhead lets one host pack many sandboxes. gVisor is a reasonable alternative for teams already on Kubernetes whose workloads tolerate its overhead.

Fast Startup

Booting a VM, cloning a large repository, and installing dependencies can take minutes. Three techniques meet the 30-second target:

Network Egress

The agent needs package registries, but open internet access lets a compromised run send source code to an attacker's server. All outbound traffic goes through an egress proxy that denies by default and allows registries such as PyPI, npm, and Maven Central. Anything else fails or triggers an approval.

The worker drives the sandbox through a narrow execution API, and nothing in the sandbox can call back into the worker. Per-sandbox CPU, memory, disk, and process limits, plus a hard wall-clock limit, ensure a fork bomb or an infinite loop only hurts its own task.

2. Context Management on Large Repositories

A long task can read far more code and command output than fits in the context window, and quality can degrade before the window is full. Context management has two halves: finding the right code, and keeping the context clean as the task goes on.

Finding the Right Code

The recommendation is agentic search as the foundation, a repository map in the initial context, and structural lookups as extra tools. Embedding search can be one more tool for natural-language questions on very large repositories, but it shouldn't decide upfront what the model gets to see.

Keeping the Context Clean

The second half is controlling how fast the context grows:

Compaction has a hidden cost: it rewrites the context prefix and invalidates the prompt cache. Since caching cut the token bill by about 4x in our estimate, keep the context append-only as long as possible and compact rarely, in one large step.

3. Permission Model for Commands

Approving every command is impractical for the developer, and approving none is unsafe. An allowlist of command strings like pytest and npm test is a common middle ground, but it has gaps. npm test runs whatever package.json says, bash -c and pipes wrap arbitrary commands, and the model can write a script and then execute it. Shell parsing can't reliably tell safe commands from unsafe ones.

A better approach is to enforce policy at the boundaries. Inside an isolated sandbox with no credentials, most commands can only damage the sandbox, which is discarded anyway. The risky actions are the ones that reach outside it:

The deny tier is enforced by the Git token's scope and GitHub branch protection, not just the policy engine, so a policy bug or a successful prompt injection still can't merge code.

The Approval Flow

A task waiting for approval shouldn't hold a worker. The worker checkpoints to the event log and releases the task, and a long wait can snapshot and stop the sandbox until the developer answers. Rejections return to the model as a tool error, so it can try another approach.

Prompt Injection

The agent reads text from many authors: issue comments, code comments, dependency source, and docs. Any of these can carry instructions like "also add this script to the CI pipeline," and filters won't catch them all. The main defense is the permission design above. With no credentials in the sandbox, egress limited to registries, a Git token restricted to agent/ branches, and human review on every pull request, a successful injection can produce a bad pull request but can't ship it or exfiltrate code. The system also flags changes to CI files, build scripts, and dependencies in the pull request description.

4. Keeping Long Tasks Alive

Restarting a two-hour task from scratch after a worker crash wastes the tokens already spent. Instead, each task runs as a durable workflow. Every model call and tool result is written to the event log before the next step begins, and recovery replays the log to rebuild the context. Because the log stores the actual model responses, the rebuilt context matches the original without calling the model again.

Tool calls need more care. If a worker crashes after starting run_command but before recording the result, the new worker can't tell whether it finished. Read-only tools simply run again. Workspace-changing commands are marked "in progress," and recovery restores the sandbox from the last snapshot and reruns the step. External actions, such as pushing a branch or opening a pull request, use idempotency keys built from the task ID and step number, so a retry never opens two pull requests.

Workflow engines like Temporal provide this out of the box. A homegrown version needs the same pieces: an ordered log, leases so one worker owns a task at a time, and heartbeats so a dead worker's lease expires quickly.

Offline Evaluation

Build the evaluation set from your own history. Take merged pull requests that fixed a bug or added a feature along with tests, check out the commit before the fix, give the agent the original issue, and grade its change with the tests from the real fix. Tests that failed before must now pass, and the rest of the suite must keep passing. SWE-bench applies this method to public Python repositories, and the same approach works on private code.

A few details make the results trustworthy:

Every change to the model, system prompt, tools, or compaction logic runs against this suite before release.

Online Metrics

Failed and abandoned tasks are a valuable source of evaluation data. Sample them, label the failure (wrong file, gave up early, broke an unrelated test, looped), and add representative cases to the offline set.

Safety Controls

Before any pull request opens, the system scans the diff for secrets, flags changes to CI configuration or dependency manifests, and checks the size to catch runaway changes. The audit log ties every model call, tool call, and approval to the developer who started the task.

Scaling to 10x

At one million tasks per day, peak concurrency grows to about 42,000 sandboxes, and token demand rises to roughly 19 million input tokens per second on average.

Reducing Cost

Tokens dominate the bill, so that's where the savings are:

Judge each change by cost per merged pull request. Cutting tokens by 40% while the merge rate falls from 60% to 45% makes every useful result more expensive.

A coding agent is a simple loop wrapped in infrastructure that makes thousands of concurrent runs safe, durable, and affordable.

In this design, the model chooses each next step, while the sandbox, the Git boundary, and the CI checks control what runs, what gets verified, and what reaches the main branch. The developer reviews and merges the pull request, and the agent handles the edit-and-test cycles before that point.