Platforms have traditionally used trained classifiers backed by human reviewers. This works well for known categories, but policy changes are slow to roll out. If the policy team updates its definition of hate speech, engineers may need to collect new labeled examples, retrain the classifier, and validate it before deploying the change, which can take weeks.

LLMs make policy updates much faster. Instead of retraining a model for every change, we can include the updated policy in the prompt and ask the model to evaluate content against it. The problem is cost and speed at this volume. Running a large LLM on hundreds of millions of items per day would cost millions of dollars per month and add noticeable latency.

A practical system combines these methods. Fast, cheap classifiers handle clear cases, LLMs evaluate ambiguous content, and human reviewers handle difficult decisions and appeals.

In this chapter, we'll design a content moderation pipeline that processes 500 million text and image items per day. We'll start with the requirements and high-level architecture, then go deep on cascading cheap and expensive checks, updating policies without retraining, measuring precision and recall, and handling adversarial users.

Let's start by understanding the problem.

A content moderation pipeline takes a piece of content, along with context about who posted it and where, and determines whether it violates a policy. Each decision should identify the action taken, the policy applied, and the stage that made the decision.

For example, a post might produce this result:

The policy_version tells us which rules were applied, making it possible to review the decision if a user appeals a month later. The decided_by field identifies the stage responsible, allowing us to measure the accuracy of each stage independently.

Moderation decisions also go beyond simply allowing or removing content:

These intermediate actions are important because not every case has a clear answer. If a post might be harmful but the system isn't confident enough to remove it, we can temporarily limit its reach while it undergoes further review. This reduces the risk of removing legitimate content or allowing potentially harmful content to spread.

Beyond scale, several factors make moderation difficult:

Content moderation systems vary a lot from one platform to another, so we should clarify the scope before designing the architecture.

Here is how a requirements discussion might unfold in an interview:

The conversation gives us the constraints we need. Let's write them down as functional and non-functional requirements.

Functional Requirements

Non-Functional Requirements

We also need to account for regulatory requirements. For example, the EU's Digital Services Act requires covered platforms to explain why content was removed or restricted and provide an internal process for challenging moderation decisions. This makes it important to store the policy, version, and rationale behind each enforcement action.

The main goal of these estimates is to understand whether we can afford to run an LLM on every item and, if not, how much traffic we should send to it.

Traffic

We'll assume peak traffic is three times the average, so the system needs to handle roughly 17,400 items per second.

Cost of Sending Everything to an LLM

Suppose each LLM request includes about 2,000 tokens of policy instructions, along with the post text and image tokens when applicable. We'll assume an average of 2,600 input tokens per request and 150 output tokens for a structured decision and a short explanation.

Using illustrative prices of $0.50 per million input tokens and $2 per million output tokens:

We can reduce this cost through prompt caching. Since the policy instructions are shared across requests, we can place them in a stable prompt prefix and reuse cached tokens when supported by the provider.

Assuming all 2,000 policy tokens per request qualify for caching at 10% of the normal input price, their daily cost drops from $500,000 to $50,000. The remaining 600 input tokens per request cost $150,000, and output tokens add another $150,000.

That brings the total to approximately $350,000 per day, or $10.5 million per month.

Even with caching, this is expensive. The LLM would also need to handle more than 17,000 requests per second at peak, with each review potentially taking one to three seconds. That's too slow for our 150 ms pre-publish latency target.

Cost with a Cascade

Now suppose lightweight classifiers handle 96% of items, leaving only 4% for LLM review.

With the same token and caching assumptions, this reduces LLM inference costs by 96%.

The first-stage classifiers are much cheaper to run. Depending on the model and content type, a GPU can process hundreds or thousands of items per second. This means the classifier tier may need only tens of GPUs rather than the much larger capacity required to run an LLM on every item.

Human Review

Suppose 0.1% of all items require human review:

We would need at least 834 reviewers, even before accounting for appeals, reviewer availability, language coverage, and more complex cases that take longer to evaluate.

This gives us three important design decisions. First, the LLM should handle only the cases that cheaper classifiers cannot resolve confidently. Second, the thresholds for escalating content to LLMs and human reviewers directly control our operating costs. Third, prompt caching is worth optimizing because shared policy instructions account for most of the LLM's input tokens.

Before designing the system, we need to define how we'll measure its effectiveness. Moderation involves trade-offs between catching harmful content, avoiding incorrect removals, and keeping latency and costs under control.

These metrics need to be evaluated together. Increasing recall by flagging more content may reduce precision and overwhelm human reviewers. Similarly, reducing LLM usage saves money but may allow more violations to go undetected.

Prevalence measures something the other metrics miss. Counting removed posts tells us how much content the system catches, but not how much harmful content users actually encounter. Prevalence measures exposure: a violating post seen by three people has far less impact than one viewed three million times.

The goal is to reduce user exposure to violating content while keeping incorrect enforcement, latency, and cost low.

We'll build the system in stages, starting with the fast checks that every item passes through, then adding LLM and human review for cases that need more careful evaluation.

Step 1: Pre-Publish Fast Checks

When a user submits a post, the content service calls the moderation service synchronously. The fast path performs three types of checks.

Normalization standardizes the input by applying Unicode normalization, handling visually similar characters, and removing zero-width characters that could be used to bypass text filters. For images, OCR extracts embedded text so it can also be checked against the policies. OCR and image processing must fit within the latency budget; slower analysis can run asynchronously.

Hash matching compares uploaded images against databases of known violating content using perceptual hashing techniques such as PhotoDNA or PDQ. Unlike cryptographic hashes, perceptual hashes can identify similar images even after resizing or recompression. Platforms can use trusted hash databases for known CSAM and terrorist content to detect re-uploads without running a full classification model.

Fast classifiers evaluate the content against relevant policy categories. A multilingual text classifier and an image classifier produce violation scores between 0 and 1 for each category. With appropriately sized models and sufficient GPU capacity, these checks can complete in tens of milliseconds.

Step 2: The Decision Engine

The classifier scores are passed to a decision engine that applies policy-specific thresholds.

Each policy has two thresholds. If the score exceeds the upper threshold, the system can take an automatic enforcement action. If it falls below the lower threshold, the item passes that check. Scores between the two thresholds require further review.

The thresholds depend on the severity of the violation. For example, a high-confidence spam classification might trigger immediate removal, while potentially severe content may be held for additional review even when the classifier is less certain.

The decision engine also considers context such as account history, previous violations, and expected reach. These signals help determine review priority and temporary restrictions, but they should not determine whether the content itself violates a policy.

Step 3: LLM Review for Uncertain Cases

Items that require further review are sent to a queue and processed by an LLM review service.

The service retrieves the relevant policy definitions and constructs a prompt containing the content, available context, policy rules, and exceptions. A multimodal LLM evaluates the item and returns a structured decision with the applicable policy, recommended action, and rationale.

Most LLM reviews happen asynchronously. Rather than making users wait several seconds, the decision engine can publish uncertain, lower-risk content with limited reach, excluding it from recommendations and trending feeds until the review finishes.

The LLM then determines whether the content can receive full distribution, should be removed, or needs human review. For potentially severe violations, the system holds the content until an appropriate review is complete.

Step 4: Policy Store

The policy store maintains versioned policy definitions, including rules, examples, exceptions, and enforcement settings.

The LLM review service retrieves the relevant policy text, while the decision engine uses the configured thresholds and actions. Keeping policies separate from model code allows the policy team to update enforcement rules without retraining or redeploying the models.

Every decision records the exact policy version used, making it possible to audit or reconsider the decision later.

Step 5: Human Review and Appeals

Cases that remain uncertain after LLM review, require specialist judgment, or involve appeals are sent to human reviewers.

The review queue prioritizes items by severity and potential exposure, then routes them to reviewers with the appropriate language skills and policy expertise.

Users can also appeal enforcement decisions. Appeals are assigned to reviewers who were not involved in the original decision. Reviewers can examine the content, the original rationale, and the policy version used at the time of enforcement.

Human reviewers can uphold or overturn a decision, with the outcome recorded for auditing and evaluation.

Step 6: The Label Store

Every moderation decision is recorded in a label store, including the original classifier scores, LLM decisions, human corrections, and appeal outcomes.

This data supports three important tasks: building evaluation datasets, training improved classifiers, and monitoring moderation quality across policies.

Over time, reviewed examples can help the fast classifiers handle cases that previously required expensive LLM or human review.

The Complete Architecture

Putting everything together, the initial checks and decision engine run synchronously on the publishing path. LLM review, human review, and appeals run asynchronously.

The LLM and human review services send their decisions back to the decision engine. This keeps enforcement logic centralized, so every stage follows the same policy rules and records decisions consistently.

The dotted line from the label store back to the classifiers represents the feedback loop. LLM and human-reviewed examples become training and evaluation data, allowing the fast classifiers to improve over time and reducing the number of items that need expensive review.

Walking Through a Post

Consider a user who uploads a photo of a protest with the caption, "These people should be dealt with."

If the context suggested a credible threat, or the model couldn't determine whether the statement violated the policy, the item would be escalated to a human reviewer along with the model's assessment.

Moderation States

Each item moves through a set of moderation states shared by the decision engine, review services, and appeals process.

One important transition is from PUBLISHED to REMOVED. Moderation doesn't end when a post goes live. User reports, policy updates, or newly identified harmful content can trigger another review of previously allowed items.

This means the pipeline must support both real-time moderation of new content and asynchronous re-evaluation of existing content, while preserving the history of every decision.

The cascade keeps moderation affordable at this scale. Instead of sending every item to an expensive model, we process content through increasingly sophisticated checks. Each stage costs more but handles less traffic.

These are illustrative processing times, excluding slower operations such as OCR and queueing.

Each stage can make a final decision when the evidence is sufficiently strong. A trusted hash match may trigger immediate removal, while a classifier score below the lower threshold can clear an item for that policy. Only cases that cannot be resolved confidently move to the next stage.

Setting the Thresholds

The two thresholds for each policy determine how much traffic reaches the LLM and how many mistakes the classifiers make. We should tune them using labeled evaluation data rather than choosing arbitrary values.

Start with a dataset containing both violating and legitimate content. Run the classifier on each item and examine how precision and recall change at different score thresholds.

Items scoring between these thresholds go to the LLM. A wider range sends more content for review, improving the opportunity to catch difficult violations but increasing cost and latency. A narrower range reduces LLM usage but puts more responsibility on the classifiers.

The thresholds should also vary by policy. A well-trained spam classifier might need LLM review for only 1% of its traffic, while a more nuanced harassment policy might send 8%.

Severity also matters. For CSAM, credible threats, and indications of imminent self-harm risk, missing a violation can have serious consequences. These categories require more sensitive detection, specialized review, and stronger restrictions before publication. Lower-severity categories, such as mild spam, can rely more heavily on classifier decisions.

The LLM stage needs a similar mechanism for deciding when to escalate to humans. However, an LLM's self-reported confidence isn't necessarily reliable.

We can calibrate it against human-reviewed examples. Group LLM decisions into confidence ranges, measure their agreement with human judgments, and identify the ranges where the error rate becomes unacceptable. Cases below the required reliability threshold go to human review. Calibration should be evaluated separately for each policy, since an LLM may be reliable at detecting spam but less consistent at interpreting harassment.

Protecting the Expensive Tiers

The cascade also needs protection against sudden traffic spikes. A breaking news event or coordinated spam attack could overwhelm the LLM queue, increasing review latency and inference costs.

We can control this with three mechanisms:

These controls limit how much traffic reaches each stage while keeping high-severity cases at the front of the queue. Under normal conditions, the cascade keeps costs low by resolving easy cases early. During traffic spikes, prioritization and temporary restrictions prevent expensive review stages from becoming bottlenecks while ensuring the most important cases are handled first.

Content moderation policies change frequently. A new scam pattern emerges, a regulator issues updated guidance, or an election requires stricter rules around misinformation. With traditional classifiers, adapting to these changes often requires new labeled data and model retraining.

An LLM makes this process more flexible. Instead of learning every policy during training, the model receives the relevant policy definitions at inference time. This allows us to update enforcement behavior without retraining the model.

Policies as Versioned Configuration

Each policy is stored as a structured, versioned document. The LLM uses its definitions, examples, and exceptions to evaluate content, while the decision engine uses its thresholds and enforcement settings.

The LLM review service constructs its prompt using these fields. Policy instructions are placed first, creating a stable prefix that can benefit from prompt caching. The content being reviewed comes last, clearly separated from the policy instructions and treated as untrusted input.

The model returns a structured response following a predefined JSON schema:

The decision engine validates the response before applying any enforcement action. It checks that the policy ID is valid, the output follows the expected schema, and the recommended action is permitted under the current policy.

The rationale is stored for auditing and human review. A simplified explanation can also be shown to users, without exposing internal detection rules that could make the system easier to bypass.

Rolling Out a Policy Change

A policy update can affect millions of users, so it needs a controlled release process, much like a software change.

First, we evaluate the updated policy against a golden dataset containing known violations, legitimate content, and difficult edge cases. This helps identify regressions before the change reaches production.

Next, we run the new policy in shadow mode alongside the current version. Both evaluate the same live content, but only the current policy affects enforcement. We compare their decisions and have policy analysts review a sample of disagreements.

Shadow mode catches cases where small wording changes significantly affect model behavior. For example, removing the word "credible" from a threats policy might cause the model to flag a much broader range of statements.

Once the results are approved, we deploy the policy to a small percentage of traffic, monitor enforcement rates and quality signals, and gradually expand the rollout. If unexpected problems appear, we can roll back to the previous policy version without changing the underlying model.

Retraining Classifiers for New Policies

There's an important limitation to this approach: the LLM can only enforce a new policy on content that reaches it.

Suppose we introduce a policy targeting a new type of scam. Our existing classifiers weren't trained to recognize it, so they may assign low violation scores and automatically allow those posts. The LLM never sees them, even though it understands the new policy.

We can address this with three mechanisms:

This gives us a practical way to separate policy updates from model updates. The LLM can begin applying a validated policy within hours, while the classifiers are improved over the following days or weeks to handle most of the traffic at a much lower cost.

Measuring moderation quality is difficult because we mostly get reliable labels for content the system flags. Allowed content is rarely reviewed, so violations that slip through often go unnoticed unless a user reports them.

Measuring Precision

Precision is relatively straightforward to measure. For each policy and moderation stage, sample items that received an enforcement action and have trained reviewers determine whether those actions were correct.

For example, if reviewers confirm 960 out of 1,000 sampled harassment removals, the precision is 96%.

We should measure precision separately for each stage. An overall precision of 95% might hide a classifier tier achieving 98% and an LLM tier achieving only 85%. However, these numbers aren't directly comparable because the LLM typically receives more difficult cases. Tracking them separately still helps us identify where errors occur and whether each stage is improving.

Measuring Recall

Recall is harder because we need to estimate how many violations exist among the items the system allowed.

One approach is to randomly sample content and have human reviewers label it. The problem is that violations are relatively rare. If only 0.5% of content violates a policy, a random sample of 20,000 items would contain roughly 100 violations. Tracking recall across dozens of policies, languages, and regions would require a much larger sample.

Stratified sampling makes this more efficient. Instead of sampling uniformly, we divide content into groups based on classifier scores and sample more heavily from groups where violations are more likely.

For example:

Because we've sampled different proportions of traffic from each band, we can't simply divide the total violations found by the total items sampled.

Instead, we weight each band's observed violation rate by its share of overall traffic:

Now suppose the classifier automatically allows everything below a score of 0.30. The two lowest bands contain an estimated 0.125% of all traffic as violating content.

That means approximately:

Even if the downstream LLM and human reviewers catch every violation above the lower threshold, the pipeline cannot achieve more than about 79% recall. That's well below our 90% target.

We would need to lower the auto-allow threshold, improve the classifier, or introduce additional checks for content that currently passes through.

The lowest score band has a violation rate of just 0.05%, yet it accounts for a meaningful share of missed violations because it contains 90% of all traffic. This is why we need to sample even the content the classifier considers safe.

These estimates also have uncertainty. Finding only one violation in 2,000 samples gives us a rough estimate of that band's violation rate, not a precise measurement. In production, we'd use larger samples and report confidence intervals, especially for bands representing most of the traffic.

Other Signals and Their Biases

Random sampling provides a more reliable estimate of recall, but other signals help us understand moderation quality.

Low reviewer agreement can point to a problem outside the model. If two trained reviewers agree on only 80% of harassment cases, the policy itself might be ambiguous or difficult to apply consistently.

In these cases, we should review the disagreements, clarify the policy where necessary, and use multiple reviewers with expert adjudication to establish reliable labels. Simply taking a majority vote doesn't always resolve an unclear policy.

Finally, we should report precision and recall separately by language, region, and content category. A classifier might achieve 96% precision overall but only 75% for a language underrepresented in its training data. Users writing in that language would then face a much higher risk of incorrect enforcement.

Multilingual LLMs may help close these gaps, but their performance also varies across languages and cultural contexts. We need representative evaluation data for each important segment rather than relying on aggregate metrics.

The goal is to measure both the violations we catch and the ones we miss. Without reviewing a representative sample of allowed content, a moderation system can appear highly accurate while still letting substantial amounts of harmful content through.

Some users actively try to bypass moderation and adapt their behavior when they discover what gets blocked. We should assume that attackers will eventually find ways around any fixed rule or classifier, so the system needs multiple layers of defense that can evolve as new techniques appear.

Common Evasion Techniques

Protecting the LLM Reviewer

Prompt injection introduces a risk that traditional classifier-based systems largely avoid. Since the LLM reads user-generated content, an attacker can include instructions intended to influence its moderation decision.

For example, a post might contain instructions telling the model to ignore the policy or classify the content as safe. We need to ensure the model evaluates those instructions as content rather than following them.

Several defenses work together:

Prompt separation reduces the risk of injection, but it doesn't guarantee that the model will ignore malicious instructions. Combining it with restricted permissions, independent checks, and human escalation makes the system more resilient.

Account-Level Signals

Content moderation becomes harder when attackers coordinate across multiple accounts. Evaluating each post independently may miss patterns that are obvious when we look at account activity.

For example, several newly created accounts might post nearly identical messages, share device or network signals, or repeatedly submit borderline content. These patterns can indicate coordinated abuse even when individual posts don't clearly violate a policy.

We can feed these signals into the decision engine to adjust review priority and enforcement safeguards. A borderline post from an account associated with coordinated abuse might receive limited distribution and additional review, while similar content from an account without suspicious activity might follow the normal process. Account-level signals should inform risk assessment without automatically turning otherwise legitimate content into a violation.

Enforcement feedback also needs careful design. If users can see the exact terms, classifier scores, or thresholds that triggered a decision, they can repeatedly modify their content until it passes. Instead, we should provide the relevant policy and a clear explanation of the decision without exposing internal detection details.

Finally, the system needs a fast response to new evasion techniques. When reviewers identify an emerging pattern, the policy team should be able to deploy updated rules, terms, or hashes within minutes. Matching content can be routed to LLM review immediately, while new labeled examples are collected to improve the classifiers.

The goal is to detect new patterns quickly, limit their impact, and update the moderation pipeline without waiting for a full model retraining cycle.

With the core architecture in place, let's examine how the system handles traffic spikes, service failures, and operating costs.

Handling Peak Traffic

The pre-publish pipeline needs to handle approximately 17,400 items per second at peak while keeping p99 latency under 150 ms.

The moderation services are stateless and scale horizontally. For GPU-based classifiers, we use dynamic batching to group requests arriving within a few milliseconds. This improves GPU utilization without adding significant latency to the publishing path.

The LLM tier runs asynchronously behind a queue, allowing it to process traffic independently of the content service. During traffic spikes, uncertain items can remain in the limited-reach state until review completes. High-severity cases stay on hold and receive priority.

When Things Fail

Moderation systems need an explicit failure strategy. Failing open means allowing content when a check is unavailable, while failing closed means blocking or holding it until the check recovers.

Neither approach works for every situation. The appropriate response depends on the severity of the potential violation and which service has failed.

We also need to treat model updates as potential sources of failure. A new LLM version may interpret the same policy differently, even when the prompt hasn't changed. Every model upgrade should therefore pass the same golden dataset evaluations, shadow comparisons, and gradual rollout process used for policy updates.

Controlling Cost

Our earlier estimates put LLM inference at approximately $420,000 per month, with classifier infrastructure potentially costing tens of thousands more, depending on model size and GPU utilization.

Human review is another major expense. At 500,000 reviews per day and 600 decisions per reviewer, we need at least 834 full-time reviewer equivalents before accounting for appeals, staffing coverage, and complex cases. Depending on staffing costs, human review could be the largest operating expense.

This means optimizing the LLM purely for inference cost isn't necessarily the best strategy. A more expensive model that resolves ambiguous cases accurately may reduce the number of items sent to human reviewers enough to lower total costs. We should measure cost per correctly resolved case, not just cost per LLM request.

We can also improve human review efficiency by showing reviewers the relevant policy sections, content context, and the LLM's rationale. This reduces the time spent gathering information and helps reviewers apply policies consistently, while keeping the final judgment independent.

Human review also has an important operational consideration: reviewers may be exposed to disturbing content. Production systems should provide protections such as blurred images by default, limits on exposure to severe content, reviewer rotation, and mental health support.

Key Trade-offs

An LLM-based content moderation pipeline combines fast classifiers, LLMs, and human reviewers to balance accuracy, latency, and cost. Lightweight checks handle most content in milliseconds, LLMs evaluate ambiguous cases against written policies, and human reviewers handle difficult decisions and appeals.

Each item goes to the cheapest method that can decide it reliably, which keeps LLM and reviewer costs bounded as traffic grows.