{"title":"Design a Semantic Search Engine","description":"","content":"\n> **What is a semantic search engine?**\n>\n> A semantic search engine retrieves results based on the meaning of a query, not just keyword overlap. Traditional keyword search matches exact terms, so it fails when users describe what they want in different words than the documents use.\n>\n> Semantic search closes that gap using embeddings and learned representations.\n\n\n---\n\n# 1. Problem Formulation\n\nBefore starting the design, it's important to ask thoughtful questions to uncover hidden assumptions, clarify ambiguities, and define the system's scope more precisely.\n\nHere is an example of how a discussion between the candidate and the interviewer might unfold:\n\n### Clarifying Questions\n\n\n> **DISCUSSION**\n>\n> **Candidate:** \"What kind of documents are we searching over? Web pages, internal enterprise docs, or something more specific?\"\n>\n> **Interviewer:** \"General-purpose web search. Think web pages, articles, blog posts, documentation.\"\n>\n> **Candidate:** \"What's the scale of the document corpus?\"\n>\n> **Interviewer:** \"About 10 billion documents, with a few million new or updated pages per day.\"\n>\n> **Candidate:** \"Are we handling just text queries, or also images and other modalities?\"\n>\n> **Interviewer:** \"Text queries only for this design. The queries are short, typically 3-5 words.\"\n>\n> **Candidate:** \"What's the latency requirement?\"\n>\n> **Interviewer:** \"End-to-end results in under 200ms at the 99th percentile.\"\n>\n> **Candidate:** \"Do we need to support multiple languages?\"\n>\n> **Interviewer:** \"Let's focus on English for now, but the architecture should be extensible to other languages.\"\n>\n> **Candidate:** \"How important is personalization? Should results vary by user?\"\n>\n> **Interviewer:** \"Some personalization is expected, like factoring in location and recent search history, but relevance should dominate.\"\n>\n> **Candidate:** \"Do we have click logs and human relevance judgments available for training?\"\n>\n> **Interviewer:** \"Yes. We have billions of click logs and a smaller set of about 500K human-rated query-document pairs.\"\n\n\nAfter gathering the details, we can summarize the key system requirements.\n\n### Requirements\n\n#### **Functional Requirements:**\n\n- Return ranked results that match the semantic intent of the query, not just keyword overlap\n- Handle typos, synonyms, and paraphrased queries\n- Support filtering by freshness, domain, content type\n- Return results across multiple document fields (title, body, URL)\n\n#### **Non-Functional Requirements:**\n\n- Sub-200ms end-to-end latency at p99\n- Index 10 billion documents with daily incremental updates\n- Handle 50,000 queries per second at peak\n- High relevance quality: NDCG@10 above 0.45\n\n### ML Problem Framing\n\nThis is a multi-stage ranking problem. No single model can score 10 billion documents per query within 200ms, so we break it into stages:\n\n1. **Retrieval** narrows 10 billion documents to ~1,000 candidates using cheap, fast methods (BM25 + embedding similarity).\n2. **Ranking** scores those 1,000 candidates with a more expressive model and returns the top 100.\n3. **Re-ranking** applies final business logic, diversity, and freshness adjustments to produce the top 10.\n\nEach stage trades off expressiveness for speed. The retrieval stage must be fast but can afford lower precision. The ranking stage can be slower but must be accurate. This multi-stage funnel pattern appears in nearly every large-scale search and recommendation system.\n\n\n```mermaid\nflowchart LR\n A[\"10B
Documents\"]:::primary --> B[\"Retrieval
~1,000\"]:::teal\n B --> C[\"Ranking
~100\"]:::orange\n C --> D[\"Re-ranking
~10\"]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe ML objective at the retrieval stage is maximizing recall: don't miss relevant documents. At the ranking stage, the objective shifts to precision: put the best documents at the top. These two objectives require different model architectures and training strategies.\n\n### Metrics\n\n#### **Offline Metrics:**\n\n\n| Metric | What It Measures | Target |\n|--------|-----------------|--------|\n| NDCG@10 | Ranking quality of top 10 results, accounting for position | > 0.45 |\n| MRR (Mean Reciprocal Rank) | How high the first relevant result appears | > 0.55 |\n| Recall@1000 | Fraction of relevant documents captured in the top 1,000 retrieved | > 0.90 |\n\n\nNDCG@10 is the primary offline metric because it penalizes relevant results that appear too low in the ranking. MRR is useful for navigational queries where users want one specific page. Recall@1000 measures retrieval quality separately from ranking quality: if 90% of the truly relevant documents for a query make it into the 1,000 candidates handed to the ranker, retrieval is doing its job. A great ranker can't fix results that were never retrieved.\n\n#### **Online Metrics:**\n\n\n| Metric | What It Measures | Why It Matters |\n|--------|-----------------|----------------|\n| Click-through rate (CTR) | Fraction of queries where a user clicks a result | Indicates result attractiveness |\n| Zero-result rate | Fraction of queries returning no results | Signals coverage gaps |\n| Reformulation rate | Fraction of queries followed by a rephrased search | Indicates dissatisfaction |\n| Session success rate | Fraction of sessions ending without reformulation | Best proxy for user satisfaction |\n\n\nCTR alone is misleading because users might click a result and immediately bounce. Session success rate, which combines click behavior with dwell time, is a more reliable signal. A drop in session success rate is often the first indicator that something is wrong.\n\n---\n\n# 2. System Architecture\n\nThe system has two major planes: an offline pipeline that processes documents and trains models, and an online pipeline that handles queries in real-time.\n\n\n```mermaid\nflowchart TD\n subgraph Online Pipeline\n Q[Query]:::primary --> CACHE[Query
Cache]:::teal\n CACHE --> |cache miss| QU[Query
Understanding]:::teal\n CACHE --> |cache hit| RES[Results]:::green\n QU --> BM25[BM25
Retrieval]:::orange\n QU --> ANN[ANN Retrieval
Base + Live]:::orange\n BM25 --> MERGE[Candidate
Fusion]:::teal\n ANN --> MERGE\n MERGE --> FF[Feature
Fetch]:::teal\n FF --> RANK[Cross-Encoder
+ GBM Ranker]:::orange\n RANK --> RR[Re-ranking,
Filters, Safety]:::teal\n RR --> RES\n end\n\n subgraph Offline Pipeline\n DOCS[Document
Corpus]:::primary --> PROC[Document
Processing]:::teal\n PROC --> ENC[Document
Encoder]:::orange\n ENC --> IDX[(ANN Index
Base + Live)]:::teal\n PROC --> INV[(Inverted
Index)]:::teal\n CLICKS[Click
Logs]:::primary --> TRAIN[Model
Training]:::orange\n JUDGE[Human
Judgments]:::primary --> TRAIN\n TRAIN --> RANK\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### Online Path\n\nA query arrives and passes through query understanding (spell correction, intent classification, query expansion). Two parallel retrieval paths run: BM25 (a scoring function that weights term overlap by inverse document frequency and document length) over the inverted index, and ANN search over the embedding index. \n\nTheir candidate sets merge through a score-fusion step (covered in the Deep Dives), duplicates are removed, and features are fetched for each candidate. The cross-encoder ranker scores all candidates, and a final re-ranking step applies diversity rules, freshness boosts, and safety filters before returning the top 10.\n\n#### Offline Path\n\nThe offline path handles document ingestion. New and updated documents go through a processing pipeline that extracts text, generates embeddings via the document encoder, and updates both the inverted index and the ANN index. Separately, click logs and human relevance judgments feed into model training for the retrieval and ranking models.\n\n### Scale Numbers\n\nA quick back-of-the-envelope grounds the rest of the design.\n\n#### **Query load**\n\n50,000 QPS at peak. Assuming a Gaussian-ish traffic curve, average QPS is closer to 15,000-20,000. Daily query volume sits around 1.5-2 billion queries.\n\n#### **Embedding storage**\n\nTen billion documents, each a 768-dimensional float32 vector, is `10,000,000,000 × 768 × 4 bytes = 30.7 TB` for raw embeddings alone. No single machine holds that in RAM, and storing it uncompressed would make ANN search wildly expensive. \n\nProduct quantization compresses each vector roughly 16-32x (down to 64-128 bytes per document), shrinking the index to 1-2 TB and making sharded in-memory serving feasible. The ANN structure itself (HNSW graph links or IVF centroid lists) adds another 20-40% overhead.\n\n#### **Inverted index storage**\n\nTokenized postings for 10B documents compress to roughly 3-5 TB with standard techniques (variable-byte encoding, front coding, delta-compressed doclists). This lives on SSD with a hot layer in RAM.\n\n#### **Query-time GPU budget**\n\nThe query encoder is a distilled bi-encoder with 20-30M parameters (MiniLM or DistilBERT scale). One modern GPU encodes 2,000-4,000 queries per second at sub-2ms latency, so handling 50K QPS needs 12-25 query-encoder GPUs plus redundancy. \n\nThe cross-encoder ranker is larger (~110M parameters), and each query scores 800 candidates; at ~40ms per query it needs another 30-60 GPUs. Total GPU fleet for online inference is roughly 50-100.\n\n#### **Training data volume**\n\nDaily click logs at 2B queries, averaging 1.5 clicks each, yield roughly 3B positive interaction records per day. After filtering for dwell time and de-duping, 50-200M high-confidence positives per day are available for training.\n\nThese numbers set hard constraints on every design decision below: no single-node solution survives, quantization is mandatory, and the online path must parallelize aggressively.\n\n### Data Flow\n\n\n```mermaid\nflowchart LR\n\n subgraph Training Loop\n\t\tdirection LR\n CLOGS[Click
Logs]:::primary --> LABEL[Label
Extraction]:::teal\n LABEL --> PAIRS[Training
Pairs]:::teal\n PAIRS --> BIENCT[Bi-Encoder
Training]:::orange\n PAIRS --> CROSST[Cross-Encoder
Training]:::orange\n end\n\t\n subgraph Ingestion\n RAW[Raw
Documents]:::primary --> PARSE[Parse &
Clean]:::teal\n PARSE --> CHUNK[Chunk &
Extract]:::teal\n CHUNK --> EMBED[Encode to
Embeddings]:::orange\n CHUNK --> TOK[Tokenize for
BM25]:::orange\n end\n\n subgraph Index Build\n EMBED --> ANNIDX[(ANN
Index)]:::green\n TOK --> INVIDX[(Inverted
Index)]:::green\n end\t\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe ingestion pipeline runs continuously, processing a few million documents per day. The full ANN index rebuild happens weekly (10 billion documents takes several hours on a GPU cluster), with incremental updates added daily. The training loop runs on a separate schedule, typically weekly, using the latest click logs.\n\n---\n\n# 3. Data\n\n### Data Sources\n\nFour sources feed this system:\n\n1. **Document corpus.** The raw web pages. Each document has a title, URL, body text, metadata (publish date, language, domain authority). The crawler refreshes documents at different cadences: news sites every few hours, Wikipedia daily, long-tail pages monthly.\n2. **Click logs.** Billions of (query, clicked_document, position, dwell_time) records per day. These are the primary training signal. A click on position 3 with 45 seconds of dwell time is a strong positive signal. A click on position 1 with 2 seconds of dwell time (pogo-sticking) is likely negative.\n3. **Query logs.** Raw query strings with timestamps, session IDs, and user context (location, device, language). Used for query understanding features and analyzing search patterns.\n4. **Human relevance judgments.** About 500K (query, document, relevance_grade) tuples rated by trained annotators on a 5-point scale. Expensive to collect but provide unbiased labels. Used primarily for evaluation and as a complement to click data for training.\n\n### Document Representation\n\nA web document is not a monolithic blob of text. A typical article has a URL, a title, maybe an h1 header, a body of several thousand words, structured metadata, and often anchor text from inbound links. Naively concatenating all of it and feeding it to the bi-encoder throws away structure the model could use and immediately hits the 512-token limit that BERT-family encoders share.\n\nTwo problems need to be solved: **length handling** and **field structure.**\n\n#### **Chunking long documents**\n\nMost transformer encoders cap at 512 tokens; a long-form article can be 5,000-10,000. Three strategies exist:\n\n\n| Strategy | How It Works | Trade-off |\n|----------|--------------|-----------|\n| Truncation | Encode the first 512 tokens only | Simple, but loses content below the fold |\n| Fixed-size chunks | Split into overlapping 256-token windows, encode each, store all vectors | Preserves full content; index grows 5-20x |\n| Semantic chunks | Split on section headers or paragraph boundaries, encode each | Cleaner boundaries; needs reliable parsing |\n\n\nProduction systems usually mix strategies: the title and first paragraph (often the lede) always get encoded, and the body is split into semantic chunks. Each chunk becomes its own entry in the ANN index with a back-reference to the parent document. At retrieval time, the document score is the maximum over its chunk scores, so matching any relevant chunk surfaces the document. This is called chunk-level retrieval with document aggregation.\n\n#### **Multi-field encoding**\n\nTitle, URL, and body carry different signals. A query matching the title is almost always more relevant than one matching only the body; the URL often encodes hierarchy and topic. Three options handle this:\n\n1. Encode the concatenation `[TITLE] title [SEP] [URL] url [SEP] body`. Simple, and the transformer learns field weights implicitly, but the body dominates the token budget.\n2. Encode each field separately into its own vector, then combine (`weighted_sum` or learned gating). Expensive at index time but gives fine-grained control.\n3. Single vector for the document, but inject field-level matching signals as hand-crafted features into the ranker (which is what the `title_match` and `url_match` features in the table above do).\n\nThe design here uses option 3 for the bi-encoder (one vector per chunk) and relies on the cross-encoder and feature-based ranker to handle field-specific matching. This keeps the ANN index compact while still letting the ranker reason about title vs body matches.\n\n\n```mermaid\nflowchart LR\n DOC[Raw
Document]:::primary --> PARSE[Parse
Fields]:::teal\n PARSE --> T[Title]:::orange\n PARSE --> U[URL]:::orange\n PARSE --> B[Body]:::orange\n T --> CHUNK[Lede + Body
Chunks]:::teal\n B --> CHUNK\n CHUNK --> E[Encode Each
Chunk]:::orange\n E --> V[\"Chunk Vectors
(768-d each)\"]:::green\n U --> FEAT[url_match,
title_match features]:::teal\n T --> FEAT\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nOne practical note: when the same document produces N chunks, the ANN index effectively stores N×10B entries. Product quantization makes this manageable, but there is a real index-size multiplier to account for when planning capacity. Most systems cap chunks per document (e.g., 5-10 chunks) and rely on the chunker to extract the highest-signal sections.\n\n### Feature Engineering\n\nFeatures fall into three categories, and the ranking model needs all three to produce good results.\n\n**Query features** capture properties of the search query itself:\n\n- Query length (number of tokens)\n- Predicted intent (navigational, informational, transactional)\n- Whether the query contains entities (person, company, place)\n- Query frequency (common vs rare)\n\n**Document features** capture static properties of each document:\n\n- Domain authority score (based on inbound link graph)\n- Document freshness (days since last update)\n- Content length (word count)\n- Document type (article, product page, forum post)\n- Spam score from a separate classifier\n\n**Cross features** capture the interaction between a specific query and a specific document:\n\n- BM25 score (lexical match strength)\n- Bi-encoder embedding similarity (semantic match)\n- Query term coverage (fraction of query terms present in the document)\n- Title match score (query overlap with document title)\n- URL match score (query terms in URL path)\n\n### Feature Table\n\n\n| Feature | Type | Source | Freshness | Description |\n|---------|------|--------|-----------|-------------|\n| query_length | Int | Query | Real-time | Number of tokens in query |\n| query_intent | Categorical | Intent classifier | Real-time | navigational / informational / transactional |\n| query_entity_count | Int | NER model | Real-time | Named entities detected in query |\n| doc_authority | Float | Link graph | Weekly | PageRank-style domain authority (0-1) |\n| doc_freshness_days | Int | Crawler | Daily | Days since last document update |\n| doc_word_count | Int | Document processor | At ingestion | Total words in document body |\n| doc_spam_score | Float | Spam classifier | Weekly | Probability of being spam (0-1) |\n| bm25_score | Float | BM25 engine | Real-time | Lexical relevance score |\n| embedding_similarity | Float | Bi-encoder | Real-time | Cosine similarity between query and doc embeddings |\n| title_match | Float | Matching engine | Real-time | Fraction of query terms in doc title |\n| url_match | Binary | Matching engine | Real-time | Whether query terms appear in URL |\n| query_doc_overlap | Float | Matching engine | Real-time | Jaccard similarity of query and doc token sets |\n\n\n---\n\n# 4. Model\n\n### Baseline Approach\n\nStart simple: BM25 retrieval followed by a logistic regression ranker trained on hand-crafted features.\n\nBM25 retrieves the top 1,000 documents using lexical matching. It's fast, well-understood, and works well for queries that use the same words as the documents. The logistic regression ranker takes the features from the table above and learns a linear combination. This baseline is surprisingly strong. At web scale, BM25 with a good ranker achieves NDCG@10 around 0.35-0.38.\n\nThe baseline breaks on semantic queries. \"Best way to handle exceptions in Python\" won't match a document titled \"Python Error Handling Guide\" because the key terms differ. It also struggles with short, ambiguous queries like \"apple\" (the fruit or the company?) because it has no way to model intent beyond term statistics.\n\n### Advanced Approach\n\nThe production system uses two ML models in sequence: a bi-encoder for semantic retrieval and a cross-encoder for ranking.\n\n#### **Bi-encoder (retrieval)**\n\nTwo separate transformer networks encode queries and documents into 768-dimensional vectors. The two towers do not share weights. Queries are short (3-5 tokens) and written in a different style than documents (often imperative fragments vs full prose), so letting each encoder specialize on its own length and vocabulary distribution gives better retrieval quality than a shared encoder.\n\nAt query time, only the query encoder runs. Document vectors are precomputed and stored in an ANN index. The top-K nearest vectors (by dot product) become retrieval candidates. This adds a semantic retrieval path alongside BM25.\n\n\n```mermaid\nflowchart LR\n subgraph Bi-Encoder\n QT[Query
Text]:::primary --> QTE[Query
Transformer]:::teal\n QTE --> QV[\"q_vec
(768-d)\"]:::primary\n DT[Document
Text]:::primary --> DTE[Document
Transformer]:::teal\n DTE --> DV[\"d_vec
(768-d)\"]:::primary\n QV --> DOT[\"dot(q, d)\"]:::orange\n DV --> DOT\n DOT --> SCORE1[Retrieval
Score]:::green\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **ANN index design**\n\nBi-encoder retrieval is only useful if the index can return top-K neighbors from ~10B vectors in under 10ms. Brute-force cosine search is ruled out by the math: one query against 10B vectors is 7.6 trillion float-float multiplies. The system uses an approximate nearest neighbor (ANN) index, and the choice of algorithm plus compression scheme determines whether the design works at all.\n\n\n| Approach | Recall@100 | Latency | Memory | Fit for This System |\n|----------|-----------|---------|--------|---------------------|\n| Brute force (exact) | 100% | Seconds | Full (30 TB) | No |\n| HNSW (graph-based) | 95-99% | <2ms | Full vectors + graph | Excellent quality, but 30+ TB in RAM is cost-prohibitive |\n| IVF + flat | 85-92% | 5-10ms | Full vectors + centroids | Moderate memory savings, lower recall |\n| HNSW + PQ (product quantization) | 90-95% | 2-5ms | 1-2 TB (16-32x compression) | Production sweet spot |\n| IVF + PQ | 80-90% | 3-8ms | 1-2 TB | Faster index build, lower recall |\n\n\nThe production design uses **HNSW over product-quantized vectors**. Each 768-dim float32 vector is split into 96 sub-vectors of 8 floats each, and each sub-vector is replaced by an 8-bit codebook index trained via k-means. That compresses 3 KB per document to roughly 96 bytes, shrinking the index from 30 TB to just under 1 TB.\n\nDistance computations use precomputed lookup tables, so the compressed search is actually faster than the raw float version, not slower. The recall cost is real (90-95% vs 95-99% for full HNSW) but acceptable because the downstream cross-encoder can recover from minor retrieval noise.\n\nThe index is built offline from the bi-encoder's document embeddings and served read-only. Writes go through a separate live tier (covered in the Index Freshness deep dive). Cluster memory is sized at 2-3x the compressed index to hold the HNSW graph, PQ codebooks, and metadata. With the compressed index at ~1 TB and sharded across 20-30 nodes (detailed below), each node holds a manageable 50-100 GB slice.\n\n#### **Cross-encoder (ranking)**\n\nA single transformer takes the concatenated query-document pair as input and produces a relevance score from the CLS token representation passed through a small feed-forward network. Because the query and document tokens attend to each other at every transformer layer, the model captures fine-grained interactions between query terms and document terms (like matching \"handle exceptions\" to \"error handling\"), something a bi-encoder can't do because its query and document representations never interact during encoding.\n\n\n```mermaid\nflowchart LR\n QD[\"[CLS] query [SEP]
document [SEP]\"]:::primary --> BERT[Cross-Encoder
Transformer]:::teal\n BERT --> CLS[CLS
Embedding]:::teal\n CLS --> CES[CE Relevance
Score]:::orange\n DF[Dense Features
doc_authority, freshness,
bm25, title_match, ...]:::primary --> MERGE[Feature
Concat]:::teal\n CES --> MERGE\n MERGE --> GBM[Gradient-Boosted
Ranker / MLP]:::orange\n GBM --> SCORE2[Final
Score]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe cross-encoder score alone is not the final ranking signal. The features from §3 (domain authority, freshness, spam score, BM25, title match, URL match, and so on) carry information the cross-encoder can't see from raw text: link graph structure, document age in days, prior spam judgments.\n\nIn production, the CE score is concatenated with these hand-crafted features and fed to a final ranker, usually a gradient-boosted tree model (LightGBM or XGBoost) or a small MLP trained on the combined input. The CE handles semantic fit; the outer ranker handles everything else. This two-level setup also makes it cheap to add new features without retraining the transformer.\n\nThe trade-off between these two architectures is the core of this design:\n\n\n| Property | Bi-Encoder | Cross-Encoder | Late-Interaction (ColBERT) |\n|----------|-----------|---------------|----------------------------|\n| Query-document interaction | None (independent encoding) | Full cross-attention | Token-level MaxSim at query time |\n| Inference cost per pair | ~0.1ms | ~5-10ms | ~0.5-1ms |\n| Can precompute document side | Yes (single vector) | No | Yes (one vector per token) |\n| Scalable to billions of docs | Yes (via ANN) | No (too slow) | Partial (larger index, per-token vectors) |\n| Relevance quality | Good | Excellent | Very Good |\n| Role in pipeline | Retrieval (find candidates) | Ranking (sort candidates) | Middle stage or alternative ranker |\n\n\nThe bi-encoder sacrifices quality for speed. The cross-encoder sacrifices speed for quality. Late-interaction models like ColBERT sit between them: document tokens are encoded offline, and at query time the system computes lightweight token-to-token similarities (MaxSim) rather than a full transformer pass.\n\nThe trade-off is a much larger index (one vector per document token instead of one per document). The production pipeline here uses the bi-encoder + cross-encoder combination. ColBERT-style models are a credible alternative when the cross-encoder becomes a latency bottleneck, discussed more under Deep Dives.\n\n### Training\n\n#### **Bi-encoder training**\n\nTrain with contrastive loss: given a query, the model should produce higher similarity for relevant documents than for irrelevant ones.\n\nPositive pairs come from click logs (queries paired with clicked documents where dwell time exceeded 30 seconds).\n\nNegative pairs come from two sources: random negatives (documents sampled uniformly from the corpus) and hard negatives (documents that look relevant but aren't). \n\nHard negatives are critical, as they teach the model to distinguish semantically similar but irrelevant documents.\n\nPure random negatives are too easy because most random documents have nothing to do with the query, so the loss saturates quickly and the model never learns fine distinctions.\n\nHard-negative mining is typically iterative (often called ANCE-style after the ANCE paper). The first training round uses BM25 top-K non-clicked documents as hard negatives. Once the bi-encoder is trained, its own top-K non-relevant retrievals (documents it scores highly but that weren't clicked) become the next round's hard negatives, and training repeats. Each iteration pushes the model to correct its own blind spots. \n\nA common refinement is to distill hard-negative labels from the cross-encoder: the CE labels each candidate's true relevance, and candidates it rates low but the bi-encoder rates high become hard negatives. Three to five mining rounds usually close most of the gap between random-negative training and full hard-negative training.\n\n\n```python\n# Contrastive loss with in-batch negatives\ndef contrastive_loss(query_vecs, doc_vecs, temperature=0.05):\n # query_vecs: [batch_size, 768], doc_vecs: [batch_size, 768]\n similarity = torch.matmul(query_vecs, doc_vecs.T) / temperature\n labels = torch.arange(similarity.shape[0]) # diagonal = positives\n return F.cross_entropy(similarity, labels)\n```\n\n\n**Cross-encoder training.** Train with a pairwise or listwise loss. For each query, take several documents with different relevance grades (from human judgments or click signals) and train the model to rank them correctly. LambdaMART-style listwise losses work well here because they directly optimize NDCG.\n\nTraining data volume: the bi-encoder needs millions of query-document pairs to learn good representations. The cross-encoder needs fewer examples (hundreds of thousands) because it sees the full query-document interaction and learns more from each example.\n\n---\n\n# 5. Serving\n\n### Inference Pipeline\n\nHere's how a single query flows through the system:\n\n\n```mermaid\nflowchart LR\n Q[Query]:::primary --> |\"1. 5ms\"| QU[Query
Understanding]:::teal\n QU --> |\"2. 15ms\"| BM25[BM25
Retrieval
500 docs]:::orange\n QU --> |\"2. 10ms\"| ANN[ANN
Retrieval
500 docs]:::orange\n BM25 --> |\"3. 5ms\"| MERGE[Merge &
Dedup
~800 docs]:::teal\n ANN --> MERGE\n MERGE --> |\"4. 10ms\"| FF[Feature
Fetch]:::teal\n FF --> |\"5. 40ms\"| RANK[Cross-Encoder
Ranking]:::orange\n RANK --> |\"6. 10ms\"| RR[Re-rank &
Filter]:::teal\n RR --> |\"7.\"| RES[Top 10
Results]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nSteps 2a and 2b (BM25 and ANN retrieval) run in parallel, which is key to meeting the latency budget. The cross-encoder ranking step is the most expensive component, taking about 40ms to score ~800 candidates. Batching candidates and running inference on GPUs keeps this feasible.\n\nA query cache sits in front of the pipeline and short-circuits popular head queries. Web search traffic follows a heavy power law: the top few thousand queries account for 20-40% of total volume, and their top-10 results change slowly. \n\nCaching these responses in Redis or Memcached for a short TTL (typically 1-5 minutes for news-sensitive queries, longer for stable ones) cuts backend load substantially and improves p50 latency for the most common queries. Personalized and location-sensitive queries must either skip the cache or cache per-user-segment, so the cache key includes intent class and coarse user context.\n\n### Batch vs Real-Time\n\n\n| Component | Mode | Why |\n|-----------|------|-----|\n| Document encoding (bi-encoder) | Batch | 10B documents, precompute once, update incrementally |\n| ANN index build | Batch (weekly) + incremental (daily) | Full rebuild is expensive, incremental handles new docs |\n| Inverted index update | Near-real-time | New documents should be searchable within minutes |\n| Query encoding (bi-encoder) | Real-time | Must encode at query time, ~2ms per query |\n| Cross-encoder ranking | Real-time | Depends on query-document pair, can't precompute |\n| Feature computation (static) | Batch | Domain authority, spam scores change slowly |\n| Feature computation (dynamic) | Real-time | BM25 score, embedding similarity are query-dependent |\n\n\nThe biggest operational challenge is keeping the ANN index fresh. A full rebuild of a 10B-document index takes 6-8 hours on a GPU cluster. To handle new documents between rebuilds, use a small \"live\" index that holds the last 24 hours of documents and merge results from both indexes at query time.\n\n### Latency Budget\n\n\n| Component | Budget (p99) | Notes |\n|-----------|-------------|-------|\n| Query understanding | 5ms | Spell check, intent classification, lightweight models |\n| BM25 retrieval | 15ms | Inverted index lookup, sharded across nodes |\n| ANN retrieval | 10ms | HNSW search + query encoding |\n| Candidate merge + dedup | 5ms | Set union, remove duplicate URLs |\n| Feature fetch | 10ms | Parallel lookups to feature store |\n| Cross-encoder ranking | 40ms | GPU inference, ~800 candidates batched |\n| Re-ranking + filters | 10ms | Business rules, diversity, freshness boost |\n| Network overhead | 5ms | Internal RPCs between services |\n| **Total** | **~100ms** | Leaves 100ms headroom below the 200ms p99 target |\n\n\nThe budget intentionally targets ~100ms median rather than the full 200ms requirement. The extra headroom absorbs tail variance: GC pauses, shard stragglers, cold feature-store reads, and transient network hiccups are all common in a distributed system, and each can easily add 20-50ms at the 99th percentile. Budgeting to the SLA directly would mean routine tail violations.\n\nThe cross-encoder ranking step dominates the budget. If latency becomes an issue, the first lever is reducing the candidate set from 800 to 400, which halves ranking time at the cost of some recall. The second lever is distilling the cross-encoder into a smaller model, trading a bit of relevance quality for speed.\n\n### Sharding and Distribution\n\nNeither the inverted index nor the ANN index fits on a single node at this scale. The 3-5 TB inverted index and the 1-2 TB compressed ANN index are both sharded horizontally, and every query fans out to all shards of both indexes.\n\n#### **Sharding by document ID vs by term**\n\nTwo strategies exist for an inverted index. Document-ID sharding splits the corpus into N partitions (roughly 10B / N documents per shard) and replicates the full inverted index within each partition. Every query hits every shard but each shard searches a smaller document set. Term sharding routes each term to a specific shard that owns its postings list. \n\nDocument-ID sharding is the standard choice for web search: query fan-out is wide but per-shard work scales well, load balances naturally, and adding capacity means adding shards rather than re-routing terms. The same partition scheme is used for the ANN index so that document-level features co-locate with both retrieval paths.\n\n#### **Query fan-out**\n\nThe query router sends the incoming query to every shard in parallel. Each shard returns its local top-K (K=50-100 is typical for a 20-30 shard cluster). A root aggregator merges all shard results, sorts by local score, and keeps the global top-K before handing off to ranking. Network fan-out is the latency floor: with 25 shards and a 10ms per-shard budget, the query completes in roughly `max(shard_latency) + merge_time`, not `sum`, assuming enough network bandwidth to run in parallel.\n\n\n```mermaid\nflowchart TD\n Q[Query]:::primary --> R[Query
Router]:::teal\n R --> S1[Shard 1
Inverted + ANN]:::orange\n R --> S2[Shard 2
Inverted + ANN]:::orange\n R --> S3[Shard ...]:::orange\n R --> SN[Shard N
Inverted + ANN]:::orange\n S1 --> |top-100| AGG[Root
Aggregator]:::teal\n S2 --> |top-100| AGG\n S3 --> |top-100| AGG\n SN --> |top-100| AGG\n AGG --> |global top-1000| RANK[Ranking
Stage]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **Tail latency and stragglers**\n\nFan-out exposes the system to tail-latency amplification: a single slow shard slows the whole query. Two techniques mitigate this. \n\nFirst, each shard is replicated 2-3 times and the router uses request hedging: after a small delay (typically the p95 of shard latency), send the same request to a replica and take whichever response arrives first.\n\nSecond, shards may return partial results if they exceed their time slice (a best-effort contract), and the aggregator tolerates a few missing shards rather than blocking. Losing 1 of 25 shards drops recall by ~4% in the worst case, which is acceptable when the alternative is a timeout.\n\n#### **Replication and fault tolerance**\n\nEvery shard runs on at least three replicas across availability zones. The router load-balances reads across replicas and fails over on timeout. Index updates are applied to a leader and asynchronously replicated; staleness between leader and follower is measured and bounded (typically under a few seconds for the live tier).\n\n#### **Capacity planning**\n\nAt 50K peak QPS, 25 shards, and 2.5-3x replication, roughly 75-100 serving nodes handle retrieval. This sits behind the 50-100 GPU nodes for query encoding and cross-encoder ranking, plus a feature-store tier and a caching tier. The total online-serving footprint lands at roughly 200-300 nodes, cost-dominated by the GPUs.\n\n---\n\n# 6. Deep Dives\n\n### Query Understanding\n\nRaw user queries are messy. They contain typos, ambiguous terms, and implicit intent. The query understanding module transforms the raw query into a form that the retrieval and ranking stages can work with effectively.\n\n\n```mermaid\nflowchart LR\n RAW[Raw Query]:::primary --> SPELL[Spell
Correction]:::teal\n SPELL --> INTENT[Intent
Classification]:::orange\n INTENT --> EXPAND[Query
Expansion]:::teal\n EXPAND --> ENTITY[Entity
Recognition]:::orange\n ENTITY --> PROC[Processed
Query]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n**Spell correction** uses a combination of edit distance, language models, and query logs. If \"pythn tutorail\" has been corrected to \"python tutorial\" in thousands of past sessions, the system learns that mapping directly. For novel misspellings, a character-level model proposes corrections.\n\n**Intent classification** determines what the user wants. A query like \"amazon\" is navigational (the user wants amazon.com). \"Best noise-canceling headphones 2025\" is informational. \"Buy AirPods Pro\" is transactional. The retrieval strategy differs: navigational queries rely heavily on URL matching and domain authority, while informational queries need strong semantic matching.\n\n\n| Query Type | Example | Retrieval Strategy |\n|------------|---------|-------------------|\n| Navigational | \"youtube\" | URL match, domain authority, prioritize exact match |\n| Informational | \"how does DNS work\" | Semantic retrieval, content quality, freshness |\n| Transactional | \"buy running shoes\" | Product pages, price features, merchant quality |\n| Local | \"coffee shops near me\" | Location-aware ranking, map integration |\n\n\n**Query expansion** adds related terms to improve recall. For \"ML deployment strategies,\" the system might add \"model serving,\" \"inference optimization,\" and \"MLOps\" to the query. This can be done through a learned model (predict expansion terms from query embeddings) or through mining co-occurrence patterns in query logs. \n\nA more recent approach uses a small LLM to rewrite the query into a more expressive form or to generate several paraphrases that are searched in parallel and whose results are merged. LLM rewriting handles ambiguity and long-tail phrasing well, at the cost of added latency (a 1-3B-parameter rewriter adds 10-30ms) and occasional hallucinated terms that drift off topic. \n\nThe risk is the same either way: expanding \"jaguar speed\" with \"car\" and \"animal\" retrieves irrelevant results for both intents. Careful expansion, ideally conditioned on the predicted intent, avoids this.\n\n**Entity recognition** identifies named entities (people, companies, locations) and links them to a knowledge graph. When a user searches \"Tim Cook net worth,\" recognizing \"Tim Cook\" as Apple's CEO allows the system to boost documents from authoritative sources and fetch structured data (like a knowledge card).\n\nQuery understanding must stay under 5ms, so all these components use lightweight models: small distilled transformers or even logistic regression classifiers trained on query logs.\n\n### Hybrid Retrieval Fusion\n\nRunning BM25 and ANN retrieval in parallel is only half the story. Each path returns a ranked list of candidates, and the system has to combine them into a single list before ranking. \n\nThe score distributions are fundamentally incompatible: BM25 produces unbounded positive floats (often 5-50 for reasonable matches), while cosine similarity is bounded in [-1, 1] and most live results cluster in [0.5, 0.9]. Adding them directly, or even normalizing and adding them, is unstable because the scales shift with corpus size, query length, and document length.\n\nThree fusion strategies solve this:\n\n#### **Reciprocal Rank Fusion (RRF)**\n\nIgnore the raw scores entirely and use the rank positions. For a document `d` appearing at rank `r_bm25` in the BM25 list and `r_ann` in the ANN list, the combined score is:\n\n\n```python\ndef rrf_score(rank_bm25, rank_ann, k=60):\n # k dampens the contribution of lower-ranked documents\n score_bm25 = 1 / (k + rank_bm25) if rank_bm25 else 0\n score_ann = 1 / (k + rank_ann) if rank_ann else 0\n return score_bm25 + score_ann\n```\n\n\nRRF has a few nice properties: it is score-scale invariant, tuning-free (the `k=60` constant from the original paper works across most corpora), and robust to one retriever failing (a missing rank contributes zero). It is the default choice when a principled tuning setup is not in place, and it often matches or beats tuned linear combinations.\n\n#### **Linear score combination**\n\nNormalize each path's scores (min-max or z-score per query) and take a weighted sum: `final = alpha * norm_bm25 + (1 - alpha) * norm_ann`. The weight `alpha` is tuned offline against a held-out set. This can outperform RRF when the system is well-calibrated per query type, but drift in either retriever's score distribution silently degrades quality.\n\n#### **Learned fusion**\n\nTreat rank_bm25, rank_ann, score_bm25, score_ann, and a handful of query features as inputs to a small gradient-boosted model that outputs a single fusion score. This handles query-dependent weighting (navigational queries want more BM25, informational queries want more semantic) automatically but adds another model to train and monitor.\n\n\n| Approach | Tuning Required | Handles Scale Drift | Query-Dependent Weighting | Recommended For |\n|----------|----------------|---------------------|---------------------------|-----------------|\n| Reciprocal Rank Fusion | None | Yes | No | Default, strong baseline |\n| Linear combination | Per-corpus alpha | Only with re-tuning | No (unless alpha by intent) | Mature systems with stable distributions |\n| Learned fusion | Full training pipeline | Yes | Yes | Large systems with diverse query types |\n\n\nThe design here uses RRF at launch, with the option to swap in learned fusion once enough labeled data accumulates and the team has confidence that the added complexity pays for itself. One subtle but important detail: RRF needs both retrievers to return their top-K *before* fusion, which means the candidate fetch is `2K` documents, not `K`. The \"Merge & Dedup ~800 docs\" step in the latency budget reflects this.\n\n### Index Freshness\n\nNew web pages are published every second. News articles need to be searchable within minutes of publication; social and trending content within seconds. A full rebuild of the ANN index takes 6-8 hours on a GPU cluster, far too slow to serve the freshness requirement directly. The system bridges the gap with a two-tier index design.\n\n\n```mermaid\nflowchart LR\n NEW[New/Updated
Documents]:::primary --> LIVE[(Live Tier
~100M docs
last 24-48h)]:::orange\n HIST[Historical
Corpus]:::primary --> BASE[(Base Tier
~10B docs
weekly rebuild)]:::teal\n Q[Query]:::primary --> R[Retrieval
Layer]:::teal\n R --> LIVE\n R --> BASE\n LIVE --> M[Merge &
Dedup]:::teal\n BASE --> M\n M --> RANK[Ranking]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **Base tier**\n\nHolds the full 10B-document corpus in the PQ-compressed HNSW index described earlier. Rebuilt weekly end-to-end. Optimized for scale and steady-state quality.\n\n#### **Live tier**\n\nA much smaller index (100-200M documents) covering everything ingested in the last 24-48 hours. Uses full-precision HNSW without quantization, because the size is small enough to fit in RAM uncompressed and the extra recall is valuable for fresh content. Rebuilt every 1-2 hours. Writes to the live tier can happen in near-real-time (minutes from document publication to searchability).\n\nAt query time, both indexes are searched in parallel, their results are merged and deduplicated by URL, and the union is passed to the cross-encoder. When the weekly base rebuild completes, documents that were in the live tier migrate to the base tier and the live tier shrinks back to the new cutoff window.\n\n#### **Deletion and updates**\n\nBoth tiers support document deletion via tombstones rather than physical removal. A URL that appears in both tiers takes the most recent version based on a crawl timestamp. This pattern avoids having to rebuild either index for routine updates and keeps staleness bounded.\n\n#### **The cost**\n\nThe system now serves two retrieval paths per retrieval type, so fan-out doubles. The live tier is small enough that its query latency is negligible (~2-3ms), but the complexity budget is real: more moving parts, more versions of the bi-encoder to keep aligned between tiers, and careful attention to embedding versioning (discussed in Monitoring).\n\n### Relevance Feedback\n\nThe system has access to billions of user interactions, but turning raw clicks into training signal is harder than it sounds.\n\n**Implicit feedback** comes from user behavior:\n\n- **Clicks** indicate interest, but are biased by position. Users click the first result 30-40% of the time regardless of quality. A document in position 1 gets 10x the clicks of the same document in position 5.\n- **Dwell time** measures engagement. Over 30 seconds suggests the user found what they needed. Under 5 seconds (pogo-sticking) suggests they didn't.\n- **Reformulations** signal dissatisfaction. If a user searches \"python error handling,\" clicks nothing, then searches \"python try except tutorial,\" the first query likely had poor results.\n- **Session termination** without reformulation suggests success.\n\nThe biggest challenge with click data is **position bias**. Users interact with higher-ranked results more often, regardless of actual relevance. If you train directly on clicks, the model learns to reproduce the existing ranking rather than improve it.\n\nThree approaches handle position bias:\n\n\n| Approach | How It Works | Trade-off |\n|----------|-------------|-----------|\n| Position feature | Include result position as a feature during training, set to 0 at inference | Simple, but assumes bias is additive |\n| Inverse propensity weighting | Weight each click by 1/P(click|position), downweighting easy positions | Principled, but variance increases for low positions |\n| Randomization | Occasionally shuffle results to collect unbiased data | Gold-standard data, but hurts user experience |\n\n\nIn practice, a combination works best: use inverse propensity weighting for the bulk of training data, and supplement with a small randomized dataset (1-5% of traffic) for calibration.\n\n**Explicit feedback** from human annotators provides unbiased relevance grades but is expensive ($0.50-2.00 per judgment) and doesn't scale to cover the tail of queries. Use it primarily for evaluation and to calibrate the click-based training pipeline.\n\nThe feedback loop closes when improved models produce better rankings, which generate better click data, which trains better models. This virtuous cycle is powerful but can also amplify biases. If the model consistently ranks a category of documents lower, they get fewer clicks, which makes the model rank them even lower. Monitoring for these feedback loops is essential.\n\n### Personalization\n\nNot all users searching the same query want the same results. A software engineer searching \"tree\" likely wants data structures. A botanist wants the plant. Personalization injects user context into the ranking to resolve these ambiguities.\n\n**Personalization signals:**\n\n\n| Signal | Source | Staleness | Impact |\n|--------|--------|-----------|--------|\n| Search history (last 24h) | Query logs | Real-time | High for session continuity |\n| Long-term interests | Aggregated click history | Weekly | Medium for topic preference |\n| Location | IP / GPS | Real-time | High for local queries |\n| Language preference | Browser settings | Static | High for multilingual |\n| Device type | Request headers | Real-time | Low, but affects result format |\n\n\nThe simplest approach is adding user features to the ranking model. Concatenate a user embedding (learned from click history) with the query-document features and let the ranker learn when personalization helps. The user embedding can be a time-decayed average of document embeddings the user has clicked on, giving more weight to recent interactions.\n\nA more sophisticated approach learns a user-specific query embedding. Instead of encoding just the query text, the query encoder takes both the query and a summary of the user's recent search context, producing a personalized query vector. This shifts results toward the user's interests at the retrieval stage, not just the ranking stage.\n\n**When personalization hurts.** For navigational queries (\"gmail login\") and fact-seeking queries (\"population of France\"), personalization adds noise. The right answer doesn't depend on who's asking. The system should detect these query types (via intent classification) and reduce or disable personalization for them.\n\nA good design decision is to make personalization a re-ranking signal rather than a retrieval signal, at least initially. This limits the blast radius: if personalization goes wrong, the base results are still good. Once the system is well-calibrated, you can introduce personalized retrieval for ambiguous queries.\n\n### Safety and Content Filtering\n\nRelevance and safety pull in different directions. The most \"relevant\" document for some queries is also misleading, harmful, or illegal to surface. The system enforces safety at two points: during ingestion, and during re-ranking. The spam score in the feature table is one signal, but production search applies a broader policy layer.\n\n**Ingestion filters** remove documents that are clearly out of scope: malware-hosting domains, confirmed phishing sites, adult content for non-opted-in queries, and content that violates platform policy. These checks run during document processing before embeddings are computed, keeping the corpus clean.\n\n**Re-ranking filters** handle content that made it into the index but should be down-ranked or removed for specific queries: misinformation on medical or civic topics, NSFW content without explicit intent, and safety-sensitive topics that require authoritative sources. A set of classifiers (medical, legal, adult, violence) score each candidate. Scores above a threshold either demote the document or replace it with a safer alternative, depending on the query's sensitivity class.\n\n**Authoritative-source boosts.** For health, finance, elections, and similar high-stakes topics, the system boosts results from sources with strong editorial review (major news outlets, government sites, academic institutions). This is policy, not pure relevance, and it lives entirely in re-ranking so that the retrieval and cross-encoder stages stay focused on topical match.\n\nThe trade-off is transparency. Aggressive filtering can suppress legitimate results, and quiet demotion is hard to audit. Production systems keep a full decision log (what was filtered, which classifier fired, what the threshold was) and sample filtered queries for human review.\n\n---\n\n# 7. Evaluation and Iteration\n\n### Offline Evaluation\n\nBuild a test set from human relevance judgments, covering a mix of query types: head queries (frequent, well-served), torso queries (moderate frequency), and tail queries (rare, often poorly served). The distribution matters because models often perform well on head queries but poorly on the tail, and the tail is where semantic search adds the most value over BM25.\n\n#### **Constructing the evaluation set**\n\nQuery sampling deserves real care. A uniform sample from query logs heavily oversamples the head (the top 1,000 queries are a huge fraction of traffic) and under-represents the tail, which is where the model actually breaks. \n\nThe standard fix is stratified sampling: draw separate samples from head, torso, and tail bands (by frequency quantile), plus targeted samples for navigational, informational, transactional, and local intents. Aim for 3,000-10,000 labeled queries, each with 20-50 judged documents, which is enough statistical power to detect NDCG differences of 0.01-0.02.\n\n#### **Labeling**\n\nAnnotators rate each (query, document) pair on a 5-point relevance scale (Perfect, Excellent, Good, Fair, Bad). Every query gets at least three annotators, and disagreements are resolved by a senior reviewer. \n\nTrack inter-annotator agreement (Cohen's kappa or similar) as a quality signal: kappa below 0.5 means the guidelines are ambiguous, and the labels are noisier than the model's differences. Refresh the test set quarterly to catch distribution shift, and version it so past experiments remain reproducible.\n\nCompute NDCG@10 on the full test set and on slices:\n\n\n```python\nfrom sklearn.metrics import ndcg_score\n\n# Per-query NDCG computation\nndcg_scores = []\nfor query_id, group in test_set.groupby(\"query_id\"):\n true_relevance = group[\"human_label\"].values\n predicted_scores = group[\"model_score\"].values\n ndcg = ndcg_score([true_relevance], [predicted_scores], k=10)\n ndcg_scores.append({\"query_id\": query_id, \"ndcg\": ndcg, \"type\": group[\"query_type\"].iloc[0]})\n\n# Slice analysis\nresults = pd.DataFrame(ndcg_scores)\nprint(results.groupby(\"type\")[\"ndcg\"].mean())\n```\n\n\n**Output:**\n\n\n```plaintext\nquery_type\nhead 0.52\ntorso 0.44\ntail 0.38\nnavigational 0.61\ninformational 0.42\n```\n\n\nIf NDCG on tail queries is significantly lower than head queries, you likely need more diverse training data or better query expansion for rare queries.\n\n### Online Evaluation\n\nA/B testing in search requires care. Standard randomized experiments work, but two pitfalls are specific to search:\n\n#### **Novelty effects**\n\nUsers initially click more on new result layouts or orderings simply because they're different, not because they're better. Run experiments for at least 2 weeks to let novelty wear off.\n\n#### **Interleaving**\n\nInstead of showing different users different result sets, interleave results from the control and treatment systems into a single result page and measure which system's results get more clicks. Interleaving detects ranking quality differences with 10-100x fewer queries than traditional A/B tests, making it the preferred method for search quality experiments.\n\nGuard against regressions by monitoring secondary metrics during any experiment: zero-result rate, latency p99, and crash rate. An improvement in NDCG that comes with a 20ms latency increase might not be worth shipping.\n\n#### **The offline-online gap**\n\nA recurring headache in search is that offline NDCG improves but online metrics (CTR, session success, reformulation rate) refuse to move, or occasionally move the wrong way. There are a few reasons this happens. Human judges rate documents on topical relevance, but users also weigh freshness, readability, layout, and authority signals that judges may under-weight. \n\nThe judgment distribution doesn't match the traffic distribution, so an improvement concentrated on the tail can show up strongly offline and invisibly online. And some online metrics are noisy enough that only large changes clear the significance bar. \n\nThe practical response is to track both kinds of metrics side by side during every experiment, and to treat offline NDCG as a guardrail and filter rather than the single source of truth. When the two disagree, trust the online signal and investigate why the offline set missed the effect.\n\n### Monitoring\n\nProduction search systems degrade silently. The queries don't stop coming, but relevance erodes gradually as the world changes and the model stays fixed.\n\n#### **What to monitor:**\n\n- **Retrieval recall.** Sample queries periodically, run them through the full ranking pipeline on the complete corpus (expensive), and check if the retrieval stage captures the top-ranked documents. A drop in recall means the ANN index or the bi-encoder is going stale.\n- **Embedding drift.** Track the average cosine similarity between query embeddings and clicked document embeddings over time. A decreasing trend means the embedding space is losing alignment with user intent.\n- **Query distribution shift.** Monitor the distribution of query intents, lengths, and topics. If a new category of queries emerges (a trending event, a new product launch), the model may not handle it well.\n- **Latency p99 by component.** Track latency for each stage individually. A latency spike in the cross-encoder often means the candidate set grew unexpectedly.\n- **Click entropy.** If users increasingly click on results at position 3+ instead of position 1, it suggests the ranking is degrading.\n\nSet up automated alerts for these metrics and retrain models when offline evaluation shows degradation. For the bi-encoder, retraining every 1-2 weeks on fresh click data is typical. The cross-encoder can be retrained less frequently (monthly) because it sees the full query-document pair and is less sensitive to distribution shift.\n\n#### **Embedding versioning and migration**\n\nRetraining the bi-encoder produces a new embedding space that is mathematically incompatible with the old one: the new query vectors cannot be compared against old document vectors and still produce meaningful distances. This means every bi-encoder retrain invalidates the entire 30 TB document index. The migration pattern is a shadow rebuild:\n\n1. Freeze the new bi-encoder checkpoint and record its version.\n2. Re-encode the full corpus into a new ANN index running in parallel with the old one.\n3. While the rebuild is in progress (6-8 hours on a GPU cluster), serving continues to use the old index with the old query encoder. The new index is dark.\n4. Once the new index is built and validated against a canary traffic slice, cut over atomically: both query encoding and retrieval switch to the new version in one deploy.\n5. Hold the old index online for a day or two as a rollback target, then retire it.\n\nThe live tier (from the Index Freshness deep dive) goes through the same cutover. Because the live tier rebuilds every 1-2 hours anyway, its migration is cheap. The expensive part is the base-tier re-embedding, which runs on a dedicated GPU pool separate from the serving fleet so training load doesn't impact serving latency. \n\nA common mistake is to mix old and new embeddings in the index (for example, serving some new documents with the new encoder and older documents with the old one); the resulting score distributions are silently inconsistent and recall degrades in ways that don't show up in aggregate metrics. Keep each index pure to a single embedding version.","pageType":"ml-system-design"}

Design a Semantic Search Engine

Ashish Pratap Singh

Get Premium