Before starting the design, it's important to ask thoughtful questions to uncover hidden assumptions, clarify ambiguities, and define the system's scope more precisely.

Here is an example of how a discussion between the candidate and the interviewer might unfold:

Clarifying Questions

After gathering the details, we can summarize the key system requirements.

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

This is a multi-stage ranking problem. No single model can score 10 billion documents per query within 200ms, so we break it into stages:

Each stage trades off expressiveness for speed. The retrieval stage must be fast but can afford lower precision. The ranking stage can be slower but must be accurate. This multi-stage funnel pattern appears in nearly every large-scale search and recommendation system.

The ML objective at the retrieval stage is maximizing recall: don't miss relevant documents. At the ranking stage, the objective shifts to precision: put the best documents at the top. These two objectives require different model architectures and training strategies.

Metrics

Offline Metrics:

NDCG@10 is the primary offline metric because it penalizes relevant results that appear too low in the ranking. MRR is useful for navigational queries where users want one specific page. Recall@1000 measures retrieval quality separately from ranking quality: if 90% of the truly relevant documents for a query make it into the 1,000 candidates handed to the ranker, retrieval is doing its job. A great ranker can't fix results that were never retrieved.

Online Metrics:

CTR alone is misleading because users might click a result and immediately bounce. Session success rate, which combines click behavior with dwell time, is a more reliable signal. A drop in session success rate is often the first indicator that something is wrong.

The system has two major planes: an offline pipeline that processes documents and trains models, and an online pipeline that handles queries in real-time.

Online Path

A query arrives and passes through query understanding (spell correction, intent classification, query expansion). Two parallel retrieval paths run: BM25 (a scoring function that weights term overlap by inverse document frequency and document length) over the inverted index, and ANN search over the embedding index.

Their candidate sets merge through a score-fusion step (covered in the Deep Dives), duplicates are removed, and features are fetched for each candidate. The cross-encoder ranker scores all candidates, and a final re-ranking step applies diversity rules, freshness boosts, and safety filters before returning the top 10.

Offline Path

The offline path handles document ingestion. New and updated documents go through a processing pipeline that extracts text, generates embeddings via the document encoder, and updates both the inverted index and the ANN index. Separately, click logs and human relevance judgments feed into model training for the retrieval and ranking models.

Scale Numbers

A quick back-of-the-envelope grounds the rest of the design.

Query load

50,000 QPS at peak. Assuming a Gaussian-ish traffic curve, average QPS is closer to 15,000-20,000. Daily query volume sits around 1.5-2 billion queries.

Embedding storage

Ten billion documents, each a 768-dimensional float32 vector, is 10,000,000,000 × 768 × 4 bytes = 30.7 TB for raw embeddings alone. No single machine holds that in RAM, and storing it uncompressed would make ANN search wildly expensive.

Product quantization compresses each vector roughly 16-32x (down to 64-128 bytes per document), shrinking the index to 1-2 TB and making sharded in-memory serving feasible. The ANN structure itself (HNSW graph links or IVF centroid lists) adds another 20-40% overhead.

Inverted index storage

Tokenized postings for 10B documents compress to roughly 3-5 TB with standard techniques (variable-byte encoding, front coding, delta-compressed doclists). This lives on SSD with a hot layer in RAM.

Query-time GPU budget

The query encoder is a distilled bi-encoder with 20-30M parameters (MiniLM or DistilBERT scale). One modern GPU encodes 2,000-4,000 queries per second at sub-2ms latency, so handling 50K QPS needs 12-25 query-encoder GPUs plus redundancy.

The cross-encoder ranker is larger (~110M parameters), and each query scores 800 candidates; at ~40ms per query it needs another 30-60 GPUs. Total GPU fleet for online inference is roughly 50-100.

Training data volume

Daily click logs at 2B queries, averaging 1.5 clicks each, yield roughly 3B positive interaction records per day. After filtering for dwell time and de-duping, 50-200M high-confidence positives per day are available for training.

These numbers set hard constraints on every design decision below: no single-node solution survives, quantization is mandatory, and the online path must parallelize aggressively.

Data Flow

The ingestion pipeline runs continuously, processing a few million documents per day. The full ANN index rebuild happens weekly (10 billion documents takes several hours on a GPU cluster), with incremental updates added daily. The training loop runs on a separate schedule, typically weekly, using the latest click logs.

Data Sources

Four sources feed this system:

Document Representation

A web document is not a monolithic blob of text. A typical article has a URL, a title, maybe an h1 header, a body of several thousand words, structured metadata, and often anchor text from inbound links. Naively concatenating all of it and feeding it to the bi-encoder throws away structure the model could use and immediately hits the 512-token limit that BERT-family encoders share.

Two problems need to be solved: length handling and field structure.

Chunking long documents

Most transformer encoders cap at 512 tokens; a long-form article can be 5,000-10,000. Three strategies exist:

Production systems usually mix strategies: the title and first paragraph (often the lede) always get encoded, and the body is split into semantic chunks. Each chunk becomes its own entry in the ANN index with a back-reference to the parent document. At retrieval time, the document score is the maximum over its chunk scores, so matching any relevant chunk surfaces the document. This is called chunk-level retrieval with document aggregation.

Multi-field encoding

Title, URL, and body carry different signals. A query matching the title is almost always more relevant than one matching only the body; the URL often encodes hierarchy and topic. Three options handle this:

The design here uses option 3 for the bi-encoder (one vector per chunk) and relies on the cross-encoder and feature-based ranker to handle field-specific matching. This keeps the ANN index compact while still letting the ranker reason about title vs body matches.

One practical note: when the same document produces N chunks, the ANN index effectively stores N×10B entries. Product quantization makes this manageable, but there is a real index-size multiplier to account for when planning capacity. Most systems cap chunks per document (e.g., 5-10 chunks) and rely on the chunker to extract the highest-signal sections.

Feature Engineering

Features fall into three categories, and the ranking model needs all three to produce good results.

Query features capture properties of the search query itself:

Document features capture static properties of each document:

Cross features capture the interaction between a specific query and a specific document:

Feature Table

Baseline Approach

Start simple: BM25 retrieval followed by a logistic regression ranker trained on hand-crafted features.

BM25 retrieves the top 1,000 documents using lexical matching. It's fast, well-understood, and works well for queries that use the same words as the documents. The logistic regression ranker takes the features from the table above and learns a linear combination. This baseline is surprisingly strong. At web scale, BM25 with a good ranker achieves NDCG@10 around 0.35-0.38.

The baseline breaks on semantic queries. "Best way to handle exceptions in Python" won't match a document titled "Python Error Handling Guide" because the key terms differ. It also struggles with short, ambiguous queries like "apple" (the fruit or the company?) because it has no way to model intent beyond term statistics.

Advanced Approach

The production system uses two ML models in sequence: a bi-encoder for semantic retrieval and a cross-encoder for ranking.

Bi-encoder (retrieval)

Two separate transformer networks encode queries and documents into 768-dimensional vectors. The two towers do not share weights. Queries are short (3-5 tokens) and written in a different style than documents (often imperative fragments vs full prose), so letting each encoder specialize on its own length and vocabulary distribution gives better retrieval quality than a shared encoder.

At query time, only the query encoder runs. Document vectors are precomputed and stored in an ANN index. The top-K nearest vectors (by dot product) become retrieval candidates. This adds a semantic retrieval path alongside BM25.

ANN index design

Bi-encoder retrieval is only useful if the index can return top-K neighbors from ~10B vectors in under 10ms. Brute-force cosine search is ruled out by the math: one query against 10B vectors is 7.6 trillion float-float multiplies. The system uses an approximate nearest neighbor (ANN) index, and the choice of algorithm plus compression scheme determines whether the design works at all.

The production design uses HNSW over product-quantized vectors. Each 768-dim float32 vector is split into 96 sub-vectors of 8 floats each, and each sub-vector is replaced by an 8-bit codebook index trained via k-means. That compresses 3 KB per document to roughly 96 bytes, shrinking the index from 30 TB to just under 1 TB.

Distance computations use precomputed lookup tables, so the compressed search is actually faster than the raw float version, not slower. The recall cost is real (90-95% vs 95-99% for full HNSW) but acceptable because the downstream cross-encoder can recover from minor retrieval noise.

The index is built offline from the bi-encoder's document embeddings and served read-only. Writes go through a separate live tier (covered in the Index Freshness deep dive). Cluster memory is sized at 2-3x the compressed index to hold the HNSW graph, PQ codebooks, and metadata. With the compressed index at ~1 TB and sharded across 20-30 nodes (detailed below), each node holds a manageable 50-100 GB slice.

Cross-encoder (ranking)

A single transformer takes the concatenated query-document pair as input and produces a relevance score from the CLS token representation passed through a small feed-forward network. Because the query and document tokens attend to each other at every transformer layer, the model captures fine-grained interactions between query terms and document terms (like matching "handle exceptions" to "error handling"), something a bi-encoder can't do because its query and document representations never interact during encoding.

The cross-encoder score alone is not the final ranking signal. The features from §3 (domain authority, freshness, spam score, BM25, title match, URL match, and so on) carry information the cross-encoder can't see from raw text: link graph structure, document age in days, prior spam judgments.

In production, the CE score is concatenated with these hand-crafted features and fed to a final ranker, usually a gradient-boosted tree model (LightGBM or XGBoost) or a small MLP trained on the combined input. The CE handles semantic fit; the outer ranker handles everything else. This two-level setup also makes it cheap to add new features without retraining the transformer.

The trade-off between these two architectures is the core of this design:

The bi-encoder sacrifices quality for speed. The cross-encoder sacrifices speed for quality. Late-interaction models like ColBERT sit between them: document tokens are encoded offline, and at query time the system computes lightweight token-to-token similarities (MaxSim) rather than a full transformer pass.

The trade-off is a much larger index (one vector per document token instead of one per document). The production pipeline here uses the bi-encoder + cross-encoder combination. ColBERT-style models are a credible alternative when the cross-encoder becomes a latency bottleneck, discussed more under Deep Dives.

Training

Bi-encoder training

Train with contrastive loss: given a query, the model should produce higher similarity for relevant documents than for irrelevant ones.

Positive pairs come from click logs (queries paired with clicked documents where dwell time exceeded 30 seconds).

Negative pairs come from two sources: random negatives (documents sampled uniformly from the corpus) and hard negatives (documents that look relevant but aren't).

Hard negatives are critical, as they teach the model to distinguish semantically similar but irrelevant documents.

Pure random negatives are too easy because most random documents have nothing to do with the query, so the loss saturates quickly and the model never learns fine distinctions.

Hard-negative mining is typically iterative (often called ANCE-style after the ANCE paper). The first training round uses BM25 top-K non-clicked documents as hard negatives. Once the bi-encoder is trained, its own top-K non-relevant retrievals (documents it scores highly but that weren't clicked) become the next round's hard negatives, and training repeats. Each iteration pushes the model to correct its own blind spots.

A common refinement is to distill hard-negative labels from the cross-encoder: the CE labels each candidate's true relevance, and candidates it rates low but the bi-encoder rates high become hard negatives. Three to five mining rounds usually close most of the gap between random-negative training and full hard-negative training.

Cross-encoder training. Train with a pairwise or listwise loss. For each query, take several documents with different relevance grades (from human judgments or click signals) and train the model to rank them correctly. LambdaMART-style listwise losses work well here because they directly optimize NDCG.

Training data volume: the bi-encoder needs millions of query-document pairs to learn good representations. The cross-encoder needs fewer examples (hundreds of thousands) because it sees the full query-document interaction and learns more from each example.

Inference Pipeline

Here's how a single query flows through the system:

Steps 2a and 2b (BM25 and ANN retrieval) run in parallel, which is key to meeting the latency budget. The cross-encoder ranking step is the most expensive component, taking about 40ms to score ~800 candidates. Batching candidates and running inference on GPUs keeps this feasible.

A query cache sits in front of the pipeline and short-circuits popular head queries. Web search traffic follows a heavy power law: the top few thousand queries account for 20-40% of total volume, and their top-10 results change slowly.

Caching these responses in Redis or Memcached for a short TTL (typically 1-5 minutes for news-sensitive queries, longer for stable ones) cuts backend load substantially and improves p50 latency for the most common queries. Personalized and location-sensitive queries must either skip the cache or cache per-user-segment, so the cache key includes intent class and coarse user context.

Batch vs Real-Time

The biggest operational challenge is keeping the ANN index fresh. A full rebuild of a 10B-document index takes 6-8 hours on a GPU cluster. To handle new documents between rebuilds, use a small "live" index that holds the last 24 hours of documents and merge results from both indexes at query time.

Latency Budget

The budget intentionally targets ~100ms median rather than the full 200ms requirement. The extra headroom absorbs tail variance: GC pauses, shard stragglers, cold feature-store reads, and transient network hiccups are all common in a distributed system, and each can easily add 20-50ms at the 99th percentile. Budgeting to the SLA directly would mean routine tail violations.

The cross-encoder ranking step dominates the budget. If latency becomes an issue, the first lever is reducing the candidate set from 800 to 400, which halves ranking time at the cost of some recall. The second lever is distilling the cross-encoder into a smaller model, trading a bit of relevance quality for speed.

Sharding and Distribution

Neither the inverted index nor the ANN index fits on a single node at this scale. The 3-5 TB inverted index and the 1-2 TB compressed ANN index are both sharded horizontally, and every query fans out to all shards of both indexes.

Sharding by document ID vs by term

Two strategies exist for an inverted index. Document-ID sharding splits the corpus into N partitions (roughly 10B / N documents per shard) and replicates the full inverted index within each partition. Every query hits every shard but each shard searches a smaller document set. Term sharding routes each term to a specific shard that owns its postings list.

Document-ID sharding is the standard choice for web search: query fan-out is wide but per-shard work scales well, load balances naturally, and adding capacity means adding shards rather than re-routing terms. The same partition scheme is used for the ANN index so that document-level features co-locate with both retrieval paths.

Query fan-out

The query router sends the incoming query to every shard in parallel. Each shard returns its local top-K (K=50-100 is typical for a 20-30 shard cluster). A root aggregator merges all shard results, sorts by local score, and keeps the global top-K before handing off to ranking. Network fan-out is the latency floor: with 25 shards and a 10ms per-shard budget, the query completes in roughly max(shard_latency) + merge_time, not sum, assuming enough network bandwidth to run in parallel.

Tail latency and stragglers

Fan-out exposes the system to tail-latency amplification: a single slow shard slows the whole query. Two techniques mitigate this.

First, each shard is replicated 2-3 times and the router uses request hedging: after a small delay (typically the p95 of shard latency), send the same request to a replica and take whichever response arrives first.

Second, shards may return partial results if they exceed their time slice (a best-effort contract), and the aggregator tolerates a few missing shards rather than blocking. Losing 1 of 25 shards drops recall by ~4% in the worst case, which is acceptable when the alternative is a timeout.

Replication and fault tolerance

Every shard runs on at least three replicas across availability zones. The router load-balances reads across replicas and fails over on timeout. Index updates are applied to a leader and asynchronously replicated; staleness between leader and follower is measured and bounded (typically under a few seconds for the live tier).

Capacity planning

At 50K peak QPS, 25 shards, and 2.5-3x replication, roughly 75-100 serving nodes handle retrieval. This sits behind the 50-100 GPU nodes for query encoding and cross-encoder ranking, plus a feature-store tier and a caching tier. The total online-serving footprint lands at roughly 200-300 nodes, cost-dominated by the GPUs.

Query Understanding

Raw user queries are messy. They contain typos, ambiguous terms, and implicit intent. The query understanding module transforms the raw query into a form that the retrieval and ranking stages can work with effectively.

Spell correction uses a combination of edit distance, language models, and query logs. If "pythn tutorail" has been corrected to "python tutorial" in thousands of past sessions, the system learns that mapping directly. For novel misspellings, a character-level model proposes corrections.

Intent classification determines what the user wants. A query like "amazon" is navigational (the user wants amazon.com). "Best noise-canceling headphones 2025" is informational. "Buy AirPods Pro" is transactional. The retrieval strategy differs: navigational queries rely heavily on URL matching and domain authority, while informational queries need strong semantic matching.

Query expansion adds related terms to improve recall. For "ML deployment strategies," the system might add "model serving," "inference optimization," and "MLOps" to the query. This can be done through a learned model (predict expansion terms from query embeddings) or through mining co-occurrence patterns in query logs.

A more recent approach uses a small LLM to rewrite the query into a more expressive form or to generate several paraphrases that are searched in parallel and whose results are merged. LLM rewriting handles ambiguity and long-tail phrasing well, at the cost of added latency (a 1-3B-parameter rewriter adds 10-30ms) and occasional hallucinated terms that drift off topic.

The risk is the same either way: expanding "jaguar speed" with "car" and "animal" retrieves irrelevant results for both intents. Careful expansion, ideally conditioned on the predicted intent, avoids this.

Entity recognition identifies named entities (people, companies, locations) and links them to a knowledge graph. When a user searches "Tim Cook net worth," recognizing "Tim Cook" as Apple's CEO allows the system to boost documents from authoritative sources and fetch structured data (like a knowledge card).

Query understanding must stay under 5ms, so all these components use lightweight models: small distilled transformers or even logistic regression classifiers trained on query logs.

Hybrid Retrieval Fusion

Running BM25 and ANN retrieval in parallel is only half the story. Each path returns a ranked list of candidates, and the system has to combine them into a single list before ranking.

The score distributions are fundamentally incompatible: BM25 produces unbounded positive floats (often 5-50 for reasonable matches), while cosine similarity is bounded in [-1, 1] and most live results cluster in [0.5, 0.9]. Adding them directly, or even normalizing and adding them, is unstable because the scales shift with corpus size, query length, and document length.

Three fusion strategies solve this:

Reciprocal Rank Fusion (RRF)

Ignore the raw scores entirely and use the rank positions. For a document d appearing at rank r_bm25 in the BM25 list and r_ann in the ANN list, the combined score is:

RRF has a few nice properties: it is score-scale invariant, tuning-free (the k=60 constant from the original paper works across most corpora), and robust to one retriever failing (a missing rank contributes zero). It is the default choice when a principled tuning setup is not in place, and it often matches or beats tuned linear combinations.

Linear score combination

Normalize each path's scores (min-max or z-score per query) and take a weighted sum: final = alpha * norm_bm25 + (1 - alpha) * norm_ann. The weight alpha is tuned offline against a held-out set. This can outperform RRF when the system is well-calibrated per query type, but drift in either retriever's score distribution silently degrades quality.

Learned fusion

Treat rank_bm25, rank_ann, score_bm25, score_ann, and a handful of query features as inputs to a small gradient-boosted model that outputs a single fusion score. This handles query-dependent weighting (navigational queries want more BM25, informational queries want more semantic) automatically but adds another model to train and monitor.

The design here uses RRF at launch, with the option to swap in learned fusion once enough labeled data accumulates and the team has confidence that the added complexity pays for itself. One subtle but important detail: RRF needs both retrievers to return their top-K before fusion, which means the candidate fetch is 2K documents, not K. The "Merge & Dedup ~800 docs" step in the latency budget reflects this.

Index Freshness

New web pages are published every second. News articles need to be searchable within minutes of publication; social and trending content within seconds. A full rebuild of the ANN index takes 6-8 hours on a GPU cluster, far too slow to serve the freshness requirement directly. The system bridges the gap with a two-tier index design.

Base tier

Holds the full 10B-document corpus in the PQ-compressed HNSW index described earlier. Rebuilt weekly end-to-end. Optimized for scale and steady-state quality.

Live tier

A much smaller index (100-200M documents) covering everything ingested in the last 24-48 hours. Uses full-precision HNSW without quantization, because the size is small enough to fit in RAM uncompressed and the extra recall is valuable for fresh content. Rebuilt every 1-2 hours. Writes to the live tier can happen in near-real-time (minutes from document publication to searchability).

At query time, both indexes are searched in parallel, their results are merged and deduplicated by URL, and the union is passed to the cross-encoder. When the weekly base rebuild completes, documents that were in the live tier migrate to the base tier and the live tier shrinks back to the new cutoff window.

Deletion and updates

Both tiers support document deletion via tombstones rather than physical removal. A URL that appears in both tiers takes the most recent version based on a crawl timestamp. This pattern avoids having to rebuild either index for routine updates and keeps staleness bounded.

The cost

The system now serves two retrieval paths per retrieval type, so fan-out doubles. The live tier is small enough that its query latency is negligible (~2-3ms), but the complexity budget is real: more moving parts, more versions of the bi-encoder to keep aligned between tiers, and careful attention to embedding versioning (discussed in Monitoring).

Relevance Feedback

The system has access to billions of user interactions, but turning raw clicks into training signal is harder than it sounds.

Implicit feedback comes from user behavior:

The biggest challenge with click data is position bias. Users interact with higher-ranked results more often, regardless of actual relevance. If you train directly on clicks, the model learns to reproduce the existing ranking rather than improve it.

Three approaches handle position bias:

In practice, a combination works best: use inverse propensity weighting for the bulk of training data, and supplement with a small randomized dataset (1-5% of traffic) for calibration.

Explicit feedback from human annotators provides unbiased relevance grades but is expensive ($0.50-2.00 per judgment) and doesn't scale to cover the tail of queries. Use it primarily for evaluation and to calibrate the click-based training pipeline.

The feedback loop closes when improved models produce better rankings, which generate better click data, which trains better models. This virtuous cycle is powerful but can also amplify biases. If the model consistently ranks a category of documents lower, they get fewer clicks, which makes the model rank them even lower. Monitoring for these feedback loops is essential.

Personalization

Not all users searching the same query want the same results. A software engineer searching "tree" likely wants data structures. A botanist wants the plant. Personalization injects user context into the ranking to resolve these ambiguities.

Personalization signals:

The simplest approach is adding user features to the ranking model. Concatenate a user embedding (learned from click history) with the query-document features and let the ranker learn when personalization helps. The user embedding can be a time-decayed average of document embeddings the user has clicked on, giving more weight to recent interactions.

A more sophisticated approach learns a user-specific query embedding. Instead of encoding just the query text, the query encoder takes both the query and a summary of the user's recent search context, producing a personalized query vector. This shifts results toward the user's interests at the retrieval stage, not just the ranking stage.

When personalization hurts. For navigational queries ("gmail login") and fact-seeking queries ("population of France"), personalization adds noise. The right answer doesn't depend on who's asking. The system should detect these query types (via intent classification) and reduce or disable personalization for them.

A good design decision is to make personalization a re-ranking signal rather than a retrieval signal, at least initially. This limits the blast radius: if personalization goes wrong, the base results are still good. Once the system is well-calibrated, you can introduce personalized retrieval for ambiguous queries.

Safety and Content Filtering

Relevance and safety pull in different directions. The most "relevant" document for some queries is also misleading, harmful, or illegal to surface. The system enforces safety at two points: during ingestion, and during re-ranking. The spam score in the feature table is one signal, but production search applies a broader policy layer.

Ingestion filters remove documents that are clearly out of scope: malware-hosting domains, confirmed phishing sites, adult content for non-opted-in queries, and content that violates platform policy. These checks run during document processing before embeddings are computed, keeping the corpus clean.

Re-ranking filters handle content that made it into the index but should be down-ranked or removed for specific queries: misinformation on medical or civic topics, NSFW content without explicit intent, and safety-sensitive topics that require authoritative sources. A set of classifiers (medical, legal, adult, violence) score each candidate. Scores above a threshold either demote the document or replace it with a safer alternative, depending on the query's sensitivity class.

Authoritative-source boosts. For health, finance, elections, and similar high-stakes topics, the system boosts results from sources with strong editorial review (major news outlets, government sites, academic institutions). This is policy, not pure relevance, and it lives entirely in re-ranking so that the retrieval and cross-encoder stages stay focused on topical match.

The trade-off is transparency. Aggressive filtering can suppress legitimate results, and quiet demotion is hard to audit. Production systems keep a full decision log (what was filtered, which classifier fired, what the threshold was) and sample filtered queries for human review.

Offline Evaluation

Build a test set from human relevance judgments, covering a mix of query types: head queries (frequent, well-served), torso queries (moderate frequency), and tail queries (rare, often poorly served). The distribution matters because models often perform well on head queries but poorly on the tail, and the tail is where semantic search adds the most value over BM25.

Constructing the evaluation set

Query sampling deserves real care. A uniform sample from query logs heavily oversamples the head (the top 1,000 queries are a huge fraction of traffic) and under-represents the tail, which is where the model actually breaks.

The standard fix is stratified sampling: draw separate samples from head, torso, and tail bands (by frequency quantile), plus targeted samples for navigational, informational, transactional, and local intents. Aim for 3,000-10,000 labeled queries, each with 20-50 judged documents, which is enough statistical power to detect NDCG differences of 0.01-0.02.

Labeling

Annotators rate each (query, document) pair on a 5-point relevance scale (Perfect, Excellent, Good, Fair, Bad). Every query gets at least three annotators, and disagreements are resolved by a senior reviewer.

Track inter-annotator agreement (Cohen's kappa or similar) as a quality signal: kappa below 0.5 means the guidelines are ambiguous, and the labels are noisier than the model's differences. Refresh the test set quarterly to catch distribution shift, and version it so past experiments remain reproducible.

Compute NDCG@10 on the full test set and on slices:

Output:

If NDCG on tail queries is significantly lower than head queries, you likely need more diverse training data or better query expansion for rare queries.

Online Evaluation

A/B testing in search requires care. Standard randomized experiments work, but two pitfalls are specific to search:

Novelty effects

Users initially click more on new result layouts or orderings simply because they're different, not because they're better. Run experiments for at least 2 weeks to let novelty wear off.

Interleaving

Instead of showing different users different result sets, interleave results from the control and treatment systems into a single result page and measure which system's results get more clicks. Interleaving detects ranking quality differences with 10-100x fewer queries than traditional A/B tests, making it the preferred method for search quality experiments.

Guard against regressions by monitoring secondary metrics during any experiment: zero-result rate, latency p99, and crash rate. An improvement in NDCG that comes with a 20ms latency increase might not be worth shipping.

The offline-online gap

A recurring headache in search is that offline NDCG improves but online metrics (CTR, session success, reformulation rate) refuse to move, or occasionally move the wrong way. There are a few reasons this happens. Human judges rate documents on topical relevance, but users also weigh freshness, readability, layout, and authority signals that judges may under-weight.

The judgment distribution doesn't match the traffic distribution, so an improvement concentrated on the tail can show up strongly offline and invisibly online. And some online metrics are noisy enough that only large changes clear the significance bar.

The practical response is to track both kinds of metrics side by side during every experiment, and to treat offline NDCG as a guardrail and filter rather than the single source of truth. When the two disagree, trust the online signal and investigate why the offline set missed the effect.

Monitoring

Production search systems degrade silently. The queries don't stop coming, but relevance erodes gradually as the world changes and the model stays fixed.

What to monitor:

Set up automated alerts for these metrics and retrain models when offline evaluation shows degradation. For the bi-encoder, retraining every 1-2 weeks on fresh click data is typical. The cross-encoder can be retrained less frequently (monthly) because it sees the full query-document pair and is less sensitive to distribution shift.

Embedding versioning and migration

Retraining the bi-encoder produces a new embedding space that is mathematically incompatible with the old one: the new query vectors cannot be compared against old document vectors and still produce meaningful distances. This means every bi-encoder retrain invalidates the entire 30 TB document index. The migration pattern is a shadow rebuild:

The live tier (from the Index Freshness deep dive) goes through the same cutover. Because the live tier rebuilds every 1-2 hours anyway, its migration is cheap. The expensive part is the base-tier re-embedding, which runs on a dedicated GPU pool separate from the serving fleet so training load doesn't impact serving latency.

A common mistake is to mix old and new embeddings in the index (for example, serving some new documents with the new encoder and older documents with the old one); the resulting score distributions are silently inconsistent and recall degrades in ways that don't show up in aggregate metrics. Keep each index pure to a single embedding version.