A query understanding system is the layer that turns the raw string a user just submitted into a structured object that downstream retrieval and ranking can act on: a corrected spelling, a predicted intent, a list of recognized entities, a set of expansion terms, and sometimes an outright rewritten query. It runs on every single search request, shares a budget of roughly 20 to 30 milliseconds with everything else that happens before retrieval, and has to work across head, torso, and tail queries in many languages at the same time. It is a strong interview problem because the candidate has to reconcile several noisy supervised tasks into one serving pipeline, balance precision against recall on each task, and design around the hard reality that most of the signal for most of the queries lives in log data rather than in human labels.

The word "query" means different things in different products. A short noun phrase in a general search engine is a different object from a long natural-language question in a shopping app or a GPS destination in a maps app. Pin the surface and the downstream consumer before writing any model.

Clarifying Questions

Candidate: "What surface are we building for? Web search, a specific vertical like shopping or video, or a maps-style destination field?"

Interviewer: "Web search. Assume a general-purpose engine where the downstream retrieval and ranking stack already exists, and we are designing the layer that sits just before retrieval."

Candidate: "What is the scale? Queries per day and peak QPS?"

Interviewer: "Around 10 billion queries per day, peaking at roughly 200,000 QPS. Design for three times headroom on peak."

Candidate: "What latency budget do we have for the query understanding layer alone?"

Interviewer: "Server-side p99 of 30ms. The end-to-end search p99 is around 300ms, and retrieval plus ranking will eat most of it."

Candidate: "What languages and markets do we support, and can I assume a single model or do we need per-language models?"

Interviewer: "About 40 languages, with heavy code-mixing in markets like India and parts of Southeast Asia. You can decide the model topology."

Candidate: "What does the downstream retrieval system actually consume from us? Just a cleaned query string, or a structured object?"

Interviewer: "A structured object. At minimum: corrected query text, a ranked list of intent labels with probabilities, entity spans with linked IDs from our knowledge graph, and up to K expansion terms."

Candidate: "Do we own spell correction in this system, or does autocomplete already do most of it?"

Interviewer: "Autocomplete catches typos while the user is typing, but people still submit malformed queries, especially on mobile and from voice. You own the submit-time correction."

Candidate: "What training signals can I rely on?"

Interviewer: "Full query and click logs, session reformulations, a small editorial synonym dictionary, a knowledge graph with about 50 million entities, and a seed set of about 1 million human-labeled queries spanning intents and entities."

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

Query understanding is a bundle of supervised tasks that share an input. Trying to model each with its own pipeline is appealing on paper and painful in production, because every extra serving path adds latency, cost, and failure modes. A better framing is multi-task learning on a shared query representation, with a head per task.

The objective is not "maximize F1 on each task." It is "maximize downstream search success given a latency budget." Every decision traces back to that.

Metrics

Offline Metrics:

Online Metrics:

The system splits into three loops with different freshness and cost profiles: a slow loop that retrains the shared encoder weekly, a medium loop that rebuilds the rewrite and entity indexes hourly, and a fast loop that ingests streaming signals (trending entities, new product launches, breaking news) into the serving layer within minutes.

The offline loop trains the shared encoder and rebuilds the rewrite and entity indexes. The streaming loop ingests fresh entities and trending signals. The online loop is what every user query touches: language detect, spell, encoder, three heads in parallel, two lightweight lookups, and a merge.

Scale Numbers

Ten billion queries per day works out to an average of about 115,000 QPS, and peak is close to 200,000 QPS. Query distribution is extremely skewed: the top 1% of distinct queries make up roughly 50% of the traffic. That skew is what makes a query cache viable, and it is also what makes the long tail hard. Head queries are easy because you have millions of impressions per query. Tail queries might appear once a week with no click signal at all.

Cache hit rate on exact query matches runs around 50 to 65% with a one-hour TTL. That single number cuts average compute by roughly half and is worth more engineering attention than most people give it.

Data Flow

Training data assembly is where this system earns or loses its quality. The fast loop handles freshness; the slow loop handles model quality. Between them is a label store that merges weak labels from logs, editorial labels, and gold human labels into a single versioned dataset per task.

Weak supervision is the only way to reach the scale needed. A click on a Wikipedia page for "Tokyo" after the query "tokio" is a strong signal that "tokio" is a misspelling of "Tokyo" and a named entity. Session reformulations ("cheap flights to ny" followed 8 seconds later by "cheap flights to new york") give free rewrite pairs. These weak labels are noisy, so the consolidator tracks per-source agreement rates and downweights signals that conflict with the human-labeled set.

Data Sources

Feature Engineering

Features are split across three surfaces: the shared encoder (raw tokens plus a handful of dense inputs), the task heads (no extra features beyond the encoder output), and the entity linker and rewrite scorer (which consume structured features from the knowledge graph and the rewrite index).

Two notes on this table. First, the shared encoder does not consume user-level features. Personalization that deep in a 30ms budget burns latency with little quality gain, so user priors are applied only at the head stage and in the rewrite scorer. Second, entity_trending_score is the feature that earns the streaming loop. Without it, a brand-new product launch or a breaking news entity cannot be linked for hours, which is the single most visible failure mode in entity linking.

Baseline Approach

A rules and dictionary baseline is not a straw man. On head traffic, it handles 60 to 70% of queries acceptably, and it is the right first slice to ship because it gives you a working downstream pipeline and a labeled log stream.

The baseline runs four steps in sequence:

This baseline fails on exactly the queries that matter most in an interview answer: ambiguous short queries ("apple", "jaguar"), long natural-language questions, rare entities outside the gazetteer, and code-mixed languages. That failure is the motivation for a learned model.

Advanced Approach

The production model is a single shared transformer encoder with three task heads. The encoder is a distilled 6-layer model, around 30 million parameters, with a hidden size of 384. That size is deliberate: small enough to serve in single-digit milliseconds on CPU, large enough to hold meaningful cross-task representations.

Each head is cheap: a two-layer MLP for intent, a linear-chain CRF over token outputs for NER, a linear projection for the rewrite embedding that is then matched against a rewrite index using ANN lookup of the same style as the embedding-based retrieval chapter. The heavy lifting is in the encoder, which only runs once per query.

Why multi-task instead of three models? Three reasons that show up in the numbers.

A cascade of specialized small models is a reasonable alternative and some teams choose it, particularly when their tasks live in very different domains. The tradeoff is routing logic and serving complexity: each new language or intent adds deployment work that a single multi-task model absorbs.

Training

Training runs in two stages. A pretraining stage adapts a general-purpose multilingual encoder (mBERT or XLM-R distilled) to the query distribution using masked language modeling on raw query logs. This step is the one that most teams skip and later regret. Pretraining on queries specifically teaches the encoder that "cheap flights ny" and "flights to new york cheap" live in similar neighborhoods, which no general-purpose encoder gets for free.

The fine-tuning stage runs multi-task learning with a weighted sum of losses. Weighting matters: without it, the NER loss (many tokens per query) dominates intent (one label per query) and rewrites (one retrieval objective per query) early in training.

Label noise is the second real problem. Weak labels drawn from click logs are biased by the current ranker, which is a classic feedback loop. Two techniques keep it manageable. First, reserve the 1 million human-labeled queries for evaluation and a high-weight training subset, and never use them to compute weak labels. Second, run confident-learning noise estimation on the weak labels before training: for each (query, label) pair, estimate the probability that the label is wrong using an ensemble of weaker models, and downweight high-noise examples. This reduces tail error rates by a noticeable amount without shrinking the dataset.

A curriculum that starts on head queries and gradually adds torso and tail helps convergence. The intuition is that tail queries have noisier labels and less redundant signal, so warming the encoder on high-volume examples first prevents the model from anchoring on noise.

The registry keeps every candidate model with its full offline report (per-task, per-slice, per-language). No model ships to A/B without clearing the shadow stage first, and every shipped model has a rollback target one version back in the registry so a latency or quality regression can be reverted in minutes rather than hours.

Inference Pipeline

The online path is what a query actually touches. Every component has a latency budget, and the design shape is driven by keeping the encoder forward pass to a single call and running the heads in parallel.

The cache check sits at the top of the pipeline with a one-hour TTL keyed on (normalized_query, language, country). Because the query distribution is so skewed, this cache serves more than half of all requests without touching the encoder. A cache miss runs the full pipeline and writes back asynchronously.

The merge and validate step does two jobs. It reconciles head outputs (for example, downweighting an entity span whose linked ID does not match any intent-consistent type), and it enforces a safety schema so that a malformed output from any single head cannot corrupt the response. If the rewrite scorer fails, the merge step returns the original query with no rewrite and a degraded-service flag.

Batch vs Real-Time

Every piece of the system has a natural freshness. Matching the update mechanism to the freshness is how cost stays bounded.

A common mistake is to try to serve the encoder from a streaming pipeline that retrains continuously. Encoder retraining is a 12- to 24-hour job that produces a new model once a week. What changes hourly or faster are the indexes the model reads from, not the model itself.

Latency Budget

Every millisecond in the query understanding budget is a millisecond retrieval and ranking cannot spend. The budget below assumes a cache miss, which is the worst-case path.

Two design levers keep this budget honest. Running the heads in parallel shaves roughly 6ms over serial execution. Int8 quantization of the encoder halves its forward-pass latency with a tolerable drop in offline metrics, usually around 0.3 F1 on intent and a similar hit on NER.

The decisions that actually get debated at design review are not in the high-level architecture. They are in how you handle ambiguity, freshness, expansion, and language.

6.1 Multi-intent and Ambiguous Queries

A query like "apple" has at least three plausible intents: a shopping query for the company's products, a navigational query for apple.com, and a generic informational query about the fruit. A system that commits to a single intent is wrong often enough to hurt session success by a measurable amount.

Three framings exist for this problem, each with a different contract with downstream retrieval.

Production systems tend to land on a calibrated sigmoid head with a learned threshold per intent, plus a top-K fallback when no intent clears the threshold. The calibration is the part that matters. Raw logits from a classifier are not probabilities, and retrieval decisions that assume they are will overtrigger on the intents with sharper training signal. Platt scaling or isotonic regression on a held-out calibration set brings the reliability diagram flat.

Session context is the strongest disambiguator that is actually cheap to use at serving time. If the previous query in the session was "iphone 15 review", the posterior for the shopping and media intents on a follow-up query "apple" should shift accordingly. A simple implementation concatenates the previous query embedding with the current query CLS vector before the intent head. In practice, this shaves reformulation rate on ambiguous queries by a meaningful amount.

6.2 Entity Recognition and Linking at Scale

General-purpose NER models trained on news or Wikipedia underperform on queries by a wide margin, for three reasons. Queries are short (average 3 to 4 tokens), so contextual cues are sparse. Queries lack grammar, so syntactic features used by general NER are noisy. Query entities are drawn from a long tail (products, niche places, fresh people) that is poorly represented in general corpora.

The practical answer is a three-stage entity pipeline: span detection from the shared encoder, candidate generation from a gazetteer plus an embedding index, and disambiguation using context and popularity.

Gazetteer-only linking covers head entities at near-perfect precision but misses rare and fresh entities. Embedding-only linking covers the tail better but loses precision on popular entity names that have many string matches (for example, "ford" matching both the car company and several people). The union of both, scored by the disambiguator, is the shape most production systems converge on.

Fresh entities are the hardest piece. A new movie release, a breaking news person, or a new product SKU needs to be linkable within minutes. The streaming loop ingests catalog, news, and trending-topic feeds into the entity index with a 5-minute SLO. Each fresh entity gets a bootstrapped popularity score from its source signal (news mention velocity, catalog view count, social mentions) and a temporary alias list derived from the source text. These bootstrap scores decay over a week as real query and click signals replace them.

Linking also has to handle entity ambiguity that is resolved by intent and context rather than popularity. "Michael Jordan" in a query "michael jordan basketball" clearly refers to the athlete; in "michael jordan statistician" it refers to the machine learning researcher. The disambiguator consumes the intent posterior and the context-augmented CLS vector, not just the span text, to catch these cases.

The three-stage split is worth defending against proposals to collapse it into a single neural ranker. Each stage has a different latency shape: span detection is one pass through the shared encoder and costs nothing extra, candidate generation is an index lookup bounded by the number of matches, and disambiguation is a cheap scorer over a small candidate set. A single end-to-end model would have to process the full entity catalog per query, which is not feasible at 50 million entities and 200,000 QPS.

6.3 Query Expansion versus Rewriting

Expansion and rewriting solve related but distinct problems. Expansion adds terms to broaden retrieval. Rewriting replaces the query with a reformulated version that retrieval can handle better. A shopping query like "shoes for flat feet" might be expanded with ["arch support", "stability shoes"] or rewritten as "stability running shoes arch support".

Most production systems blend these sources. Session reformulations provide the bulk of training pairs, editorial synonyms provide the highest-precision coverage for business-critical verticals, and LLM-generated rewrites (for example, GPT-4 class models) fill the long tail in offline pipelines with human review before entering the rewrite index.

The serving split follows the query distribution. Head and torso queries hit a pre-built rewrite index: the rewrite head computes a query embedding, the index returns top-K rewrites by ANN similarity, and the rewrite scorer re-ranks them using click lift, semantic similarity, and the intent posterior. Tail queries, which have no pre-built rewrites, can optionally hit an online LLM rewriter behind a latency-sensitive cache. Most systems skip online generation because it blows the latency budget, and instead rely on the encoder alone to produce a useful rewrite embedding that retrieval can consume directly.

Over-expansion is the failure mode that hurts most. Adding too many expansion terms or accepting a rewrite with low confidence broadens retrieval enough to dilute precision, and the user sees a page of loosely related results. Calibration of the rewrite scorer matters as much as calibration of the intent head. A per-language A/B budget keeps new rewrite rules from shipping until they show downstream NDCG lift on the target slice.

6.4 Multilingual and Code-Mixed Queries

A single multilingual encoder is the only serving shape that scales to 40 languages in a 30ms budget. Running one model per language is tempting because per-language quality is usually slightly higher on high-resource languages, but it multiplies the serving footprint, fragments signal on code-mixed traffic, and makes entity linking across languages painful.

Language ID on short queries is harder than people expect. A three-character query like "ny" has no signal. Script is a cheap and reliable fallback: Devanagari is almost certainly Hindi or a related language, Hangul is Korean, and so on. For Latin-script queries, a character n-gram model reaches about 95% accuracy on queries of five characters or more, and the system falls back to the user's locale or recent session language when the model is uncertain.

Code-mixed queries (Hinglish, Spanglish, Singlish, and many others) are a real and large segment in several markets. "flight ticket delhi se mumbai" mixes English and Hindi in a single string. Two architectural choices matter here.

Production systems in multilingual markets lean on a shared encoder with byte-level BPE or SentencePiece. The tokenizer choice trades off: byte-level BPE is robust to any script and never produces unknown tokens but has longer sequences on non-Latin scripts; SentencePiece produces shorter sequences but requires careful vocabulary sampling to avoid under-representing low-resource languages.

Region-specific entities are the other edge. "mumbai indians" is a cricket team in India, a navigational query. The same tokens interpreted with US context might return travel results. The intent and entity heads consume country as an input feature, and the rewrite index is partitioned by region so that an Indian user does not get expansion terms meant for US traffic. Partitioning is not the same as running a separate model. The encoder is shared; only the indexes the heads consume are region-aware.

Offline metrics drive the fast feedback loop during model development. Online metrics, attributed through A/B testing, are the truth. The gap between them is where most surprises live.

Offline Evaluation

The offline harness evaluates each task on the human-labeled set, plus a set of derived metrics that tie the model to downstream search quality. Per-task metrics are the easy part: intent macro-F1, NER span F1 with entity linking accuracy, rewrite Recall@K against a held-out (query, clicked document) set, and spell correction precision at a fixed recall target.

Per-slice metrics matter more than aggregate numbers. A model that improves aggregate F1 while regressing on local-intent queries is a net loss, because local queries are high-value and users notice when they break. The harness reports per-intent, per-language, and head/torso/tail breakdowns on every candidate model.

Counterfactual evaluation is the bridge from offline to online. For a candidate model, replay a sample of historical queries, compute what the rewrite and intent outputs would have been, and ask whether the existing retrieval and ranking stack would have returned a better result set. This relies on logged document features from the original retrieval run, and is imperfect because retrieval itself is not counterfactually identical, but it catches catastrophic regressions before they reach A/B.

Online Evaluation

A/B testing is run at the query understanding layer rather than at the model level. A new encoder, a new rewrite index, or a retuned calibration all ship as candidate query understanding variants with their own traffic slice. Primary metrics are zero-result rate, query reformulation rate, downstream click NDCG, and session success. Guardrails are p99 latency, crash rate, and a safety-filter false positive rate.

Traffic ramps follow a standard 1% to 5% to 25% to 50% to full path, with a hold at each step long enough for the session success signal to stabilize, usually 48 to 72 hours. The reformulation rate is the fastest-responding metric and often the first to show regressions, which makes it the canary the rollout watches most closely.

Interleaving is a complementary technique for rewrite evaluation specifically. Two rewrite variants are interleaved on the same query for the same user, and a winner is declared based on which variant's results received more clicks. Interleaving gives tighter statistical signal per sample than a parallel A/B, which matters because rewrite-driven improvements often look small in aggregate but consistent per-query.

Monitoring

Production monitoring covers three surfaces: input distribution, task outputs, and downstream impact.

Drift is the monitoring class that catches silent failures. A shift in the intent posterior distribution without a corresponding shift in traffic composition is a signal that the model is becoming miscalibrated, often because a training set artifact is propagating through retraining. PSI on the intent distribution, computed per language, catches most of these within hours.

Shadow traffic runs a candidate model against live queries without serving its output, and compares its predictions to the production model offline. Shadow is cheaper than A/B and catches latency and stability issues before a ramp. It does not catch downstream quality regressions, which only show up in live A/B because retrieval behavior depends on what query understanding returns.

The feedback loop back to training is where the system compounds. Logged queries, with the new model's outputs and the resulting downstream clicks, are written into the label store. The weak labeler runs on them, the consolidator merges them with existing labels, and the next weekly retraining uses them. The one discipline that keeps this from collapsing into a self-confirming loop is to reserve the human-labeled set for evaluation only, never letting weak labels overwrite gold ones.

Two guardrails protect the loop. The first is that a candidate model cannot ship if its outputs drift the intent distribution on a held-out slice by more than a calibrated threshold, even if its offline metrics improve. The second is that periodic human re-labeling refreshes the gold set every quarter, both to keep up with evolving language and to catch taxonomy drift before it bakes into weak labels. Both guardrails cost engineering time, and both are skipped by teams that later discover their query understanding system has quietly converged on whatever the current ranker preferred six months ago.

The next chapter moves from search understanding to personalized ranking, starting with a video recommendation system where the user, not the query, is the primary input.

Q1: Why use a single multi-task encoder with task heads instead of separate models for intent, NER, and rewriting?

One encoder forward pass costs roughly a third of what three specialized models cost, which matters inside a 30ms total budget. Multi-task co-training also transfers signal across tasks: entity examples sharpen intent classification on entity-heavy queries, and intent labels help disambiguate entity spans in very short queries. The quality gap is most visible on the tail, where per-task labels are sparse and the shared representation compensates. The downside is that any change to the encoder requires re-evaluating all three tasks, which makes the release cycle slower than independent models would be.

Q2: A query like "jaguar" has at least three valid intents (animal, car brand, sports team). How do you avoid forcing a single interpretation?

Frame intent as multi-label with a calibrated sigmoid head, returning a distribution rather than a top-1 label. Retrieval then weights verticals by the posterior instead of branching on a single label. Session context, like the previous query in the session, flows into the intent head as an extra input and shifts the posterior toward the right meaning without committing early. Commit too early and you lose the fallback path; leave the posterior too flat and you trigger retrieval across too many verticals and dilute relevance. Calibration (Platt scaling or isotonic regression on a held-out set) is what keeps the distribution meaningful.

Q3: Aggressive query expansion raises recall but hurts precision. How do you tune the tradeoff?

Expansion has to be evaluated on downstream metrics, not on surface similarity. The rewrite scorer uses click lift on historical (query, rewrite) pairs plus intent consistency, not just semantic similarity. A per-language A/B budget keeps new rewrite rules from shipping until they show NDCG lift on the target slice. For torso and tail queries, lower the confidence threshold to expand more aggressively, because the cost of a miss (zero results) outweighs the cost of slightly noisy expansion. For head queries, the threshold is tight because the retrieval stack already works well and any added noise regresses a metric that was fine.

Q4: How do you label intent on 10 billion queries per day without human annotators for each one?

Human labels cover a seed set of around 1 million queries, used as gold for evaluation and as a high-weight training subset. The remaining labels come from weak supervision on logs: clicks on entity pages imply entity mentions and can suggest intent, session reformulations produce rewrite pairs, and downstream click behavior on typed intent verticals produces noisy intent labels. A consolidator merges these sources, tracks per-source agreement with gold labels, and downweights sources that conflict. Confident-learning noise estimation runs on the weak label pool before training to downweight likely-wrong examples. The invariant that keeps this system honest is that gold labels are never used to generate weak labels, and weak labels never overwrite gold ones.

Q5: You trained the model primarily on English query logs, and now the product is launching in India where traffic is heavily Hinglish. What breaks and how do you fix it?

Three things break in order of severity. Language detection starts returning English on Hinglish queries, which causes the wrong rewrite index region to be queried. Entity linking fails on Hindi entity names that never appeared in training. Intent quality regresses because the encoder has a skewed tokenization of Hindi subwords. The fixes, in order: switch the tokenizer to byte-level BPE or a SentencePiece vocabulary sampled to include Hindi coverage, retrain the encoder with a multilingual pretraining phase on Hinglish query logs, partition the rewrite and entity indexes by region so the Indian variant does not pull US-centric expansions, and pass country and detected script as explicit features to the heads. A shadow traffic ramp in India, measuring zero-result rate and reformulation rate per language detected, catches regressions before full rollout.