Imagine searching your company's internal documents for "How do we recover from a database outage?"

The most useful document might be titled "Disaster Recovery Runbook." It might never mention the exact words in your query, but it contains precisely the information you're looking for. A traditional keyword search engine might miss that document. A semantic search engine should find it.

That's the problem we're going to solve.

In this article, we'll design a production-scale semantic search engine from the perspective of a Machine Learning Engineer. We'll walk through how to approach this problem in an actual ML system design interview, from clarifying requirements and choosing models to designing the retrieval pipeline, training, evaluation, scaling, and monitoring.

More importantly, we'll understand the trade-offs behind each major decision.

Clarifying the Problem

The interviewer asks: "Design a semantic search engine."

Before jumping into architecture, the first thing we would do is clarify what we're searching, who will use it, and what a successful result looks like.

Semantic search can mean very different things depending on the product. Searching products on an e-commerce platform is different from searching research papers, and both are different from searching private company documents. The underlying ML techniques may be similar, but the constraints and ranking objectives can change significantly.

Here is how that first part of the conversation might go:

For this interview, let's assume we're building an enterprise search engine over internal company documents. This gives us a concrete product to design while covering most of the important concepts behind modern semantic search.

Functional Requirements

Next, we would clarify the core functionality.

For the initial design, we'll focus on text documents. We won't support image search, voice search, or answer generation. Those are reasonable extensions, but they're not essential to solving the core problem.

Keeping the scope focused allows us to spend more time on the ML decisions that actually determine search quality.

Non-Functional Requirements

Now let's discuss the non-functional requirements.

Notice how these requirements create competing objectives. Larger models may improve relevance but increase latency. More sophisticated indexes may improve search speed but consume additional memory.

A strong design must balance all of them.

Estimating the Scale

Let's establish some numbers. Assume 10 million documents, split into roughly 50 million passages (we'll explain why this chunking is necessary shortly), and 10 million searches per day. Traffic isn't evenly distributed, so we plan for roughly 1,000 queries per second at peak.

The exact numbers aren't important. What matters is recognizing that document embedding happens mostly during ingestion, while query embedding happens for every search. This asymmetry will heavily influence our ML architecture.

Defining Success

Before choosing a model, we would ask the interviewer: "How are we measuring the quality of search results?"

A reasonable product objective is to help users find the information they need with minimal effort.

One important distinction is that embedding similarity is not our final business metric. Two documents can have very similar embeddings without being equally useful to the user. Ultimately, we want to optimize relevance to the user's information need, not merely similarity in vector space.

That distinction becomes especially important when we introduce reranking.

Defining the Search API

Let's define a simple API.

The client sends a search request containing the query, optional filters, and the maximum number of results.

The service returns a ranked list of results with an excerpt for each.

Notice that the request has no user ID field. The authenticated user's identity comes from a trusted authentication mechanism, not an arbitrary user ID supplied in the request body. We'll use that identity to enforce document permissions.

This API is intentionally simple. The interesting part is everything that happens between receiving the query and returning the ranked results.

Starting With a Baseline

An important interview principle is to establish a baseline before introducing a complex ML system.

For this problem, we would begin with a traditional keyword search engine using BM25, which ranks documents by term frequency, term rarity across the collection, and document length. BM25 is fast, interpretable, and particularly effective for exact terminology. But it struggles when the query and document express the same idea using different words.

For example, someone searching for "slow website" may want documentation about "high request latency." A purely lexical system might not connect those expressions.

That's the main limitation we'll address with semantic retrieval. However, we'll retain BM25 as part of our final design rather than replacing it entirely.

High-Level ML Approach

At a high level, we would propose a multi-stage retrieval and ranking architecture.

Documents are embedded ahead of time, while each query runs approximate nearest-neighbor search and BM25 in parallel before a more powerful ranking model orders the combined candidates. This architecture separates fast candidate retrieval from more expensive relevance scoring. That separation is the key to supporting both high search quality and low latency.

Two Different Pipelines

Let's divide the system into two major pipelines.

The first is the offline or asynchronous ingestion pipeline. It processes documents, generates embeddings, and builds the search indexes.

The second is the online serving pipeline. It accepts user queries, retrieves candidates, ranks them, and returns results.

Why separate them?

Embedding millions of documents is expensive, but we don't need to repeat that work for every query. We calculate document embeddings ahead of time and reuse them, so only the incoming query is embedded during search, which is what the dotted arrow in the diagram represents. This keeps heavy computation off the request path and lets us scale ingestion and search independently.

Let's design the ingestion pipeline first.

Collecting Documents

Our documents may come from several sources. These could include internal wikis, document repositories, issue trackers, engineering runbooks, and knowledge-management systems.

We introduce a set of connectors responsible for collecting content from those sources. Each connector extracts the document's content and metadata, including its source identifier, title, timestamps, and access permissions.

For an initial import, the connector may scan the entire collection. Afterward, it should ideally consume incremental changes through webhooks, change-data-capture mechanisms, or periodic synchronization.

Every document receives a stable identifier. This identifier is important because we'll need to connect passages, embeddings, and search results back to the original document. It also lets us update or delete documents without confusing them with newly created content.

Ingestion Architecture

Now let's build the ingestion pipeline.

A connector publishes each change to a message queue, and workers then extract, chunk, embed, and index it. We keep the original document in a canonical document store or rely on the source system as the source of truth.

Why use a queue? Because ingestion traffic can arrive in bursts, and embedding generation may be slower than document extraction. The queue decouples those stages, supports retries, and allows us to scale workers independently.

Text Extraction and Cleaning

Before generating embeddings, we need to prepare the document text.

Enterprise documents are rarely clean. They may contain HTML, Markdown, tables, navigation menus, duplicated headers, code blocks, or formatting artifacts.

Our preprocessing pipeline converts each document into a structured representation. However, we should avoid excessive text normalization. Technical identifiers, error codes, punctuation, and capitalization can carry useful meaning. For example, "HTTP 404" and "HTTP 500" describe very different problems. And in an engineering search engine, a code snippet or configuration value may be exactly what the user needs.

The goal isn't to make every document look the same. It's to preserve meaningful content while removing noise.

Why We Need Chunking

Next comes an important design decision: chunking.

Suppose a document contains 30 pages covering database architecture, replication, backup policies, and incident recovery.

If we generate one embedding for the entire document, we're forcing all those topics into a single vector. That vector may represent the overall subject, but it can lose important details. A query about backup restoration might not retrieve the document, even though one section answers the question perfectly.

Instead, we split documents into smaller passages and embed each passage separately.

As the diagram shows, each section now gets its own vector. The recovery section can match the recovery query directly, which gives us more precise retrieval. Chunking also helps keep text within the embedding model's supported input length.

But chunking introduces another problem. Smaller passages provide more focused representations, while larger passages preserve more context. We need to choose a reasonable balance.

Choosing the Chunk Size

For our initial design, we'd start with passages of roughly 300 to 500 tokens, with a small overlap between adjacent chunks. These numbers are starting points, not universal rules. We should validate them experimentally.

We'd also prefer structure-aware chunking over blindly splitting every fixed number of tokens. We should respect section boundaries, paragraphs, and code blocks whenever possible. For especially long sections, we can split further with limited overlap.

The best chunking strategy depends on the document distribution and the types of queries users ask. That's something we'll evaluate rather than assume.

Making Chunks Self-Contained

Let's look at a practical example.

Suppose a document is titled "Database Disaster Recovery." One section is called "Restoring From Backup."

If we embed only the text inside that section, the passage may not contain the words "database" or "disaster recovery." That missing context could hurt retrieval.

A simple improvement is to prepend relevant document context to each passage:

The first line carries the document title and section heading, and the passage content follows. We preserve the title and heading as metadata as well. This helps the embedding model understand what the passage is about.

We should avoid attaching excessive context to every chunk, because repeated text increases storage and computation. The objective is to make passages sufficiently self-contained for retrieval, without overwhelming their actual content.

Document and Passage Metadata

Each passage needs more than an embedding.

Beyond the purposes in the table, this metadata supports debugging. If a user reports an irrelevant search result, we should be able to identify the exact passage, embedding model version, and document version involved.

One important separation is that embeddings capture semantic meaning. They should not be treated as a substitute for authoritative metadata or access permissions.

Handling Duplicate Content

Enterprise knowledge bases often contain duplicated or near-duplicated content. A troubleshooting guide might appear in several team wikis. A copied document may exist under multiple titles. And a document might contain repeated disclaimers or templates.

Without handling duplicates, the search results could be dominated by essentially identical passages.

We can start with exact deduplication using content hashes. For near-duplicates, we can use techniques such as MinHash or embedding similarity to identify likely matches.

However, we shouldn't automatically merge documents just because their content looks similar. They may belong to different teams or have different permissions. Instead, we can group related content and reduce duplication during ranking.

This preserves source identity while improving the diversity of results.

Introducing Embeddings

Now we reach the core ML component. An embedding model transforms text into a dense vector so that passages with related meanings land near each other. "The application is responding slowly" and "Users are experiencing high request latency" share few words, but a good model should place them close together and far from a vacation policy. Embeddings don't understand meaning perfectly. They learn statistical patterns from training data, so our job is to select and evaluate a model that captures the relevance relationships our users care about.

Choosing an Embedding Model

There are several ways to generate text embeddings. We could train a model from scratch. We could fine-tune an existing encoder. Or we could begin with a pretrained embedding model.

For an initial production system, we would start with a strong pretrained retrieval model. Training from scratch would require substantial data and computation, and it's unlikely to be justified before we understand the baseline.

When selecting a pretrained model, we'd consider these factors:

For technical documentation, we should also evaluate how well the model handles specialized terminology. A model that performs well on general semantic similarity benchmarks may still struggle with internal acronyms, product names, or error messages.

So we'd shortlist a few candidates and compare them on a representative dataset from our domain.

Why a Bi-Encoder?

For the first stage of semantic retrieval, we would use a bi-encoder, which encodes the query and each passage independently. The main advantage is that document embeddings can be precomputed during ingestion, so at query time we only embed the query and compare it against 50 million stored vectors. That makes the bi-encoder suitable for large-scale retrieval. We'll use a more expensive model later, but only after narrowing the candidate set.

For scoring, let's use normalized embeddings with cosine similarity, which is equivalent to a dot product on unit-length vectors. One subtle but important point: the similarity function must match what the model was trained to optimize. Some retrieval models are designed for dot-product scoring without normalization, so the correct preprocessing and similarity function are part of the model contract.

Asymmetric Query and Document Encoding

Queries and documents serve different roles. A query might be a short question like "How do I roll back a deployment?", while the relevant document is a detailed, multi-paragraph procedure. Many retrieval models are trained for asymmetric search and expect different prefixes for each side, such as query: and passage:. The exact prefixes depend on the model, so we'd follow the selected model's intended encoding procedure rather than treating all text as interchangeable. This detail can have a large impact on retrieval quality.

Versioning Embeddings

One production detail that's easy to overlook is embedding versioning.

Suppose we improve the embedding model and deploy a new version. Can we immediately use the new query encoder with all the existing document vectors?

Usually, we shouldn't. Even if both models output 768-dimensional vectors, the new model's dimensions don't necessarily mean the same thing as the old one's, so comparing vectors across versions produces unreliable results.

So every stored embedding should carry its model version, and an upgrade means building a new document index and switching query encoding only when that index is ready. We'll come back to this rollout in the deployment section.

Choosing a Vector Index

At this point, we've generated embeddings for our 50 million passages. Comparing one query against all of them on every request, at around 1,000 queries per second, would be expensive even with GPUs. So we'll use approximate nearest-neighbor search, which trades a small amount of retrieval accuracy for a large improvement in speed.

For our initial design, we'd choose HNSW because it offers a strong latency-recall trade-off and supports incremental inserts, which our freshness requirement needs. Its graph adds memory on top of the raw vectors, and its parameters trade recall against latency and memory. We shouldn't claim one setting is optimal. Instead, we'd benchmark configurations, build a latency-versus-recall curve, and pick the cheapest one that meets our retrieval-quality target. We'd also benchmark it against an IVF-PQ index, which compresses vectors at some cost in recall. If HNSW meets our requirements at an acceptable cost, we'd favor the simpler operational choice. If memory becomes the dominant constraint, compression becomes much more attractive.

Calculating Vector Storage

Let's estimate the storage requirements.

We have 50 million passages. Assume each embedding contains 768 dimensions. With 32-bit floating-point values, each dimension takes four bytes.

That gives us approximately 154 gigabytes of raw vector data, or about 77 gigabytes with 16-bit values. But the ANN graph, identifiers, metadata, and replication all add to that footprint. So we'd measure index memory on a representative sample and extrapolate with headroom rather than sizing from the raw embedding calculation alone.

This estimate also shows why embedding dimensionality and compression are important infrastructure decisions, not just modeling choices.

Building the Vector Index

Let's return to our architecture.

An indexing service writes passage vectors into a vector search engine, either a managed vector database or a system built around a library such as Faiss. The exact vendor isn't the critical decision. What matters is an index that supports efficient ANN retrieval, incremental writes, filtering, persistence, and distributed serving.

As the diagram shows, the vector index holds only a stable passage identifier and the vector. After ANN search, we use that identifier to fetch the passage text and metadata, and we keep full document content in dedicated storage. This separation helps us manage updates and control serving costs.

Keeping Documents Fresh

Now consider a document update. An engineer edits a runbook and changes the recovery procedure. Our search engine should eventually retrieve the updated version rather than the old instructions.

The source connector detects the change and publishes an event. The processing pipeline extracts the new content, creates updated passages, generates their embeddings, and updates the indexes.

We should use document versions and idempotent processing so retries don't produce duplicate records. We also need to handle out-of-order events. An older update must not overwrite a newer version.

For example, suppose version 7 of a runbook is processed after version 8 because of a retry. The indexing worker compares versions and drops the stale write:

For our initial design, let's target a p95 document-to-searchable freshness of five minutes. We can measure that by comparing the source update timestamp with the time the new version becomes searchable.

Later, we'll discuss how to make updates visible without interrupting search traffic.

Handling Deletions

Deletions deserve special attention.

Suppose a document is deleted or a user's access is revoked. We cannot simply wait for a large background index rebuild. The content must stop appearing in unauthorized search results promptly.

We would separate logical deletion from physical cleanup.

We update the access or deletion state first so the serving system immediately rejects the affected document, with a tombstone or denylist as an additional fast block while indexes catch up. This approach avoids coupling security enforcement to the speed of ANN index maintenance. It also means cached results and stored excerpts must respect the same deletion and permission rules.

A stale vector is undesirable. A stale vector that exposes private information is unacceptable.

Walking Through a Search Request

Now that the documents are indexed, let's design the online serving path.

A user searches for: "How can I recover a failed database deployment?"

Semantic and keyword retrieval run in parallel, and access constraints apply before reranking. Now let's examine the decisions inside each stage.

Query Preprocessing

The first step is query preprocessing.

We normalize basic formatting, such as excessive whitespace and encoding issues. We may also identify language, detect empty queries, and validate filter parameters.

But we would be conservative about rewriting the query. For example, lowercasing might be harmless for ordinary English text, but technical identifiers can be case-sensitive. Stemming or removing words may also change meaning.

Consider the query "Disable automatic failover." Removing a word like "disable" would fundamentally change the user's intent.

We can introduce query rewriting or spelling correction later if evaluation shows that it helps. For the initial system, we'd preserve the original query for the embedding model while applying the appropriate analyzers for lexical search.

Simplicity matters, especially when debugging unexpected retrieval behavior.

Understanding Different Query Types

Not every query should be treated identically.

One possible improvement is a lightweight query classifier that identifies these different query types. We could then adjust retrieval weights or candidate budgets.

However, we shouldn't introduce a separate intent model immediately. We'd start with a robust hybrid retriever and add query-dependent routing only if offline and online evaluations justify the complexity.

Generating the Query Embedding

Next, the Search Service sends the processed query to an embedding inference service. The service uses the same retrieval model version as the indexed document embeddings, returns a 768-dimensional vector, and applies normalization if the model requires it. We only run the embedding model once per query, because the 50 million document embeddings were already calculated during ingestion. Since every semantic search request depends on this service, it needs batching, efficient inference runtimes, and auto-scaling.

Semantic Candidate Retrieval

The vector search engine uses the query embedding to perform approximate nearest-neighbor search.

Let's say we request the top 100 semantically similar passages. The engine returns their identifiers and similarity scores.

These are candidates, not final results. A high similarity score doesn't guarantee that the passage answers the question. A query about fixing a database outage might retrieve an article on common outage causes when the recovery runbook is more useful. So candidate retrieval should find relevant material broadly, and a more precise model later decides which candidates deserve the top positions.

Why Semantic Retrieval Alone Isn't Enough

Here's an important question the interviewer might ask: "Why don't we just use the top vector search results?"

Because semantic similarity has limitations. Embedding models can struggle with exact numbers, error codes, uncommon identifiers, and subtle negations.

Consider two documents:

These sentences share most of their words and context. An embedding model may place them close together even though their operational meanings are opposite.

Similarly, searching for a specific function name or configuration key may require exact matching.

Semantic search is good at connecting related concepts, but it isn't always reliable for precise constraints. So we should not build a production enterprise search engine on dense retrieval alone. We'd combine semantic and lexical retrieval.

Adding BM25 Retrieval

In parallel with vector search, we run a BM25 query against a traditional inverted index. This is what finds the passages containing an exact error code like "ORA-12541" quickly. We can also give additional importance to certain fields, because a match in the document title may be more useful than the same term appearing incidentally in the body. Our lexical retriever returns its own top 100 candidates.

Now we have two lists. One is based on semantic similarity, and the other is based on lexical relevance. The next challenge is combining them.

Combining the Two Retrievers

A simple approach would be a weighted average of the two scores, but BM25 and vector similarity scores operate on different scales. A BM25 score of 12 isn't directly comparable to a cosine similarity of 0.8, and even after normalization the distributions vary by query. So for the initial system, we'd use Reciprocal Rank Fusion (RRF), which combines ranked lists using each passage's position rather than its raw score, typically summing 1 / (60 + rank) across retrievers. A passage that ranks highly in both lists gets a strong combined score, while a passage found by only one retriever can still make the cut. That matters here because the two methods catch different types of relevant content. Before fusion, we deduplicate candidates by stable passage identifier, and later we can tune the constant and retrieval depths on validation data.

Metadata Filtering

Now imagine the user wants only documents created within the last six months, or only documentation owned by the infrastructure team. These are hard filters. We shouldn't depend on embeddings to represent them. The search engine must enforce them using indexed metadata, and that introduces an important ANN trade-off.

If we retrieve the top 100 vectors and apply a restrictive filter afterward, we might discard nearly all the candidates, even when relevant documents exist elsewhere. So we'd push eligible filters into retrieval whenever the search engine supports it. For a highly selective filter, exact search over the small filtered set may even be cheaper than filter-aware ANN. The best strategy depends on filter selectivity and index capabilities.

Access Control and Multi-Tenancy

Because this is enterprise search, permissions are a core requirement.

Suppose two employees ask the same query. One has access to a confidential project document. The other does not. They must receive different authorized result sets.

We'd enforce tenant isolation at the storage and retrieval layers. Within a tenant, document permissions may depend on users, groups, or source-specific access rules.

Where possible, we push permission-aware filters into candidate retrieval. But we also perform an authoritative authorization check before returning results or excerpts, as shown at the end of the diagram. This protects against stale index metadata and permission changes.

For highly dynamic permissions, we may need additional candidate retrieval to avoid underfilled result sets.

The critical rule is that ranking can decide which authorized documents are most relevant. It must never decide whether an unauthorized document is allowed to appear.

Why We Need Reranking

At this stage, we have a combined list of candidate passages, but their ordering is still based on approximate retrieval scores. We'd introduce a second-stage ranking model that examines the query and passage together, so it can distinguish a document explaining why deployments fail from one providing the exact rollback procedure. That costs more compute, so we rerank only a small candidate set produced by the fast retrievers.

Each stage narrows the set so the next, more expensive stage only sees the most promising passages. This is a classic multi-stage search architecture.

Cross-Encoder Reranking

For our first reranking model, we'd choose a cross-encoder. It reads the query and candidate passage together as one input, so query tokens can attend to passage tokens, which generally gives a richer relevance judgment than comparing two independent embeddings. The downside is that nothing can be precomputed: the model must process each query-passage pair.

If we rerank 40 candidates, that means evaluating 40 pairs per query. At 1,000 peak queries per second, that's potentially 40,000 pair evaluations per second. So the cross-encoder must be small enough, optimized enough, and provisioned appropriately for our latency budget.

Choosing the Reranking Depth

How many candidates should we rerank? We could rerank the top 10, top 50, or top 200.

Reranking more candidates gives the model more opportunities to identify truly relevant documents. But the computation increases with the number of candidates.

For our initial design, we'd start by reranking the top 40 merged passages. Then we'd measure what happens when increasing or decreasing that number.

We also need to consider candidate coverage. If a relevant document never enters the candidate pool, the reranker cannot recover it. That means improving the reranker won't fix poor first-stage recall.

Keeping these responsibilities separate makes the system easier to evaluate and improve.

Reranker Alternatives

Cross-encoders are a strong starting point, but they aren't the only option. A learning-to-rank model over features such as BM25 score, semantic similarity, title match, and freshness would be much cheaper to serve but can't read the text deeply. Late-interaction models like ColBERT keep token-level vectors at a higher storage cost, and an LLM judge is usually far too expensive for the main search path. We'd start with a compact cross-encoder and move to a more complex architecture only when measured results justify its serving cost.

From Passages to Documents

There's another important detail. Our search engine retrieves passages, but users expect documents.

Suppose five passages from the same runbook rank among the top ten results. We probably don't want to show the same runbook five times.

Instead, we group candidate passages by their parent document. We can use the highest-scoring passage as a simple document relevance signal. A more sophisticated approach may combine evidence from multiple passages. But we need to avoid rewarding long documents merely because they contain more chunks.

Once we've ranked documents, we select the most relevant passage from each one as the excerpt.

This gives users focused results while preserving links to the original documents. It also helps prevent large documents from dominating the entire results page.

Improving Ranking Diversity

Imagine a query about database replication. Our top results might all come from five nearly identical versions of the same tutorial. Each result could be individually relevant, but the page would be less useful than one covering replication fundamentals, implementation details, and failure recovery. Techniques such as Maximal Marginal Relevance (MMR) balance query relevance against similarity to results already selected. However, for a highly specific navigational query, the best exact match should remain first. So we'd treat diversification as a controlled reranking improvement rather than applying it aggressively to every query.

Generating Search Snippets

Now we need to present the results. Each result should contain a useful title and excerpt.

Because we retrieve passages directly, we already have a strong starting point for snippets. We can display the highest-ranked passage or a focused excerpt around the most relevant content.

For lexical matches, we may highlight matching terms. For semantic matches, highlighting is less straightforward because the query and passage may use different words. A simple system can show the passage without artificial highlighting. Later, we could introduce a passage selection or highlighting model.

But we should keep the displayed text faithful to the source. Search results shouldn't invent explanations that aren't present in the document.

The Complete Serving Flow

Let's connect everything.

The diagram adds the candidate counts to the flow we walked through earlier: 100 candidates from each retriever, fused with RRF, and 40 passed to the reranker before authorization, grouping, and excerpts.

Fast retrieval reduces a massive document collection to a small candidate set, and a more expensive ML model then spends computation only where it can improve the final ranking.

Building a Latency Budget

Let's check whether this design can meet our 300-millisecond p95 target. We need to budget latency across the full request path.

For an illustrative starting point, here is one way to allocate it:

These are design targets, not measured guarantees. Actual tail latency depends on queueing, network overhead, model inference, and the slowest dependencies. We would validate the overall p95 under realistic concurrent load.

The key insight is that the expensive reranking stage should operate only on a limited candidate set.

Optimizing Model Inference

At peak traffic, ML inference may become the main bottleneck. The query embedding model is relatively inexpensive because we run it once per query. The reranker is more demanding because it processes 40 query-passage pairs per search. To keep it within its ~100 ms budget, we'd choose the smallest model that meets the relevance target, use optimized runtimes and quantization, batch requests when it doesn't hurt latency, and cap the passage length sent to the reranker. If that still isn't enough, a ranking cascade can help: a cheaper model scores 100 fused candidates, and the cross-encoder reranks only the top 20. The right configuration depends on measured quality gains per unit of computation.

Caching

Popular queries such as "VPN setup" or "incident response policy" repeat many times, so caching can reduce computation and latency.

Because users have different permissions, any cached candidate or result must be scoped to the tenant and authorization context, with version-aware invalidation when indexes or models change.

We'd begin with query embedding caching and carefully scoped retrieval caching. Then we'd add broader result caching only when its security and freshness behavior is well defined.

Sharding and Replication

As our corpus and traffic grow, a single search node will no longer be enough, so we partition the index across shards. A search coordinator fans each query out to the relevant shards, and each shard returns its top candidates for the coordinator to merge into a global list. We also replicate shards to improve availability and distribute read traffic.

For multi-tenant search, tenant-aware partitioning may reduce unnecessary cross-tenant retrieval. But we should avoid creating tiny shards for every small tenant. Large tenants might require dedicated shards, while smaller tenants can share infrastructure with strict isolation. The exact strategy depends on tenant sizes, query patterns, and operational constraints.

Scaling the Ingestion Pipeline

Search serving isn't our only scaling concern. Suppose an enterprise imports millions of documents in a single day. The normal ingestion rate may be low, but the backfill workload can be enormous. Because the pipeline is asynchronous, the message queue absorbs bursts, text-processing and embedding workers scale independently, embedding workers batch passages on accelerators, and indexing workers limit write rates so they don't overwhelm the search cluster.

We'd also separate bulk backfills from urgent incremental updates. Otherwise, a large import could delay a critical runbook update for hours.

Separate queues or priorities, as shown above, preserve freshness for ordinary updates while backfills run on spare capacity.

Handling Failures

Let's discuss what happens when a component fails.

Every fallback should preserve the same authentication and authorization guarantees. The ML components improve relevance, but we shouldn't make the entire product unavailable whenever an optional ranking stage fails.

Graceful degradation gives us a more resilient search experience.

Do We Need to Train a Custom Model?

Now let's move to the ML development side.

The interviewer might ask: "Would you train your own embedding model?"

Our answer would be: not immediately. We'd begin with pretrained retrieval models and measure their performance on our domain. If the results are already strong, fine-tuning may not justify the additional complexity.

But suppose our organization uses many internal acronyms, product names, and specialized technical concepts. A general-purpose embedding model may fail to understand these relationships. That's when domain-specific fine-tuning becomes valuable.

The challenge is collecting appropriate training data. For retrieval, we need examples connecting user queries with relevant and irrelevant documents. Let's look at how to construct that dataset.

Collecting Training Data

The most valuable training signal is evidence of what users actually find relevant: search queries, clicked documents, query reformulations, and explicit feedback. If a user searches for "database rollback procedure" and opens a runbook, that pair might be a positive example. But clicks are only weak relevance signals, as the table below shows.

So we'd combine interaction logs with human relevance judgments and curated question-document pairs from internal documentation. The goal is training data that reflects actual search intent rather than merely predicting clicks.

Training the Bi-Encoder

For the bi-encoder, we'd use contrastive learning on query-passage pairs. A positive pair links a query like "How do I restore a deleted database?" to the recovery procedure that answers it, and the model learns to score that passage above the negatives. An InfoNCE-style loss with in-batch negatives is the usual starting point, because the positives for other queries in the same batch become free negatives.

Random negatives such as an article about employee benefits teach the model very little, so we add hard negatives. For the query "How do we recover an expired TLS certificate?", a document about generating new certificates without describing production recovery is semantically close but doesn't answer the question. We can mine such candidates from BM25, dense retrieval, or the existing search system by taking highly ranked passages that aren't labeled relevant. But we must verify they are genuinely negative. A supposedly negative passage may contain a valid answer, and training the model to push it away creates contradictory supervision. We'd also keep a balanced mixture of easy and hard negatives, since that tends to train more stably.

Training the Reranker

We can also fine-tune our cross-encoder to score query-passage pairs directly. We'd lean toward pairwise ranking, where the model learns that a more relevant passage should score above a less relevant one for the same query, because our actual objective is ranking rather than an absolute relevance probability. We should build training examples from the candidates our retrieval system actually produces. Otherwise, the reranker may perform well on artificial examples but struggle with the difficult candidates it encounters in production.

Avoiding Feedback Bias

There is a major problem with training directly on search clicks. Users click documents near the top more often regardless of relevance (position bias), and they can never click a relevant document the current engine fails to retrieve (selection bias). Training only on observed clicks can therefore reinforce the current ranker's mistakes. To break that loop, we'd collect independent human judgments, run carefully designed exploration experiments, and use propensity weighting for click-based learning. We would also distinguish between clicks, long engagements, repeated searches, and explicit positive feedback, because these signals have different meanings.

The Model Training Pipeline

Let's connect the training workflow.

Interaction data must be privacy-compliant, and every run is recorded with experiment tracking: data version, configuration, metrics, and artifacts. We shouldn't automatically deploy whichever model achieves the lowest training loss. A model becomes a deployment candidate only when it meets relevance, latency, cost, and safety requirements.

Building an Offline Evaluation Dataset

Before deploying a new model, we need a reliable offline evaluation process.

We'd create a held-out collection of representative queries. For each query, human evaluators judge the relevance of candidate documents. We might use graded labels:

A representative evaluation set should include common queries, rare queries, exact identifiers, ambiguous searches, and domain-specific terminology.

We also need to avoid data leakage. The evaluation queries and judgments shouldn't simply duplicate training examples.

For a changing document corpus, we should preserve the document snapshot used for each evaluation. Otherwise, the ground truth may become inconsistent as documents are edited or deleted.

Evaluating Retrieval and Ranking

Each stage gets its own metric. The first stage's job is to find relevant candidates, so we'd track Recall at 100 for the hybrid candidate generator: if a query has five known relevant passages and our top 100 contain four, Recall at 100 is 80 percent. Higher candidate recall gives the reranker more opportunities to select good results. We need to keep this separate from ANN recall, which measures how closely approximate vector search reproduces exact nearest-neighbor search. That tells us about index approximation quality, not whether the neighbors are actually relevant to the user.

For the final ranked results, we'd make NDCG at 10 a primary offline ranking metric, because it rewards placing highly relevant results near the top and uses our graded labels. Mean Reciprocal Rank is a useful companion for queries where the user needs one correct document. Measuring each stage on its own job tells us whether a regression came from retrieval or from ranking.

Evaluating More Than Accuracy

Average relevance metrics can hide serious failures.

Suppose the model improves NDCG overall but becomes much worse on exact error-code queries. That could frustrate the engineers who depend on those searches.

So we would break evaluation down by query type:

We'd also evaluate the quality of results under restrictive filters. A model might perform well without filters but struggle when searching a small subset of the corpus.

Finally, we should evaluate security and freshness independently. A relevant result that violates access permissions is still a system failure.

Search quality is multidimensional, and the evaluation process should reflect that.

Online A/B Testing

Offline metrics don't tell us everything about user behavior. Before a major rollout, we'd run an online A/B experiment comparing the current search system with the new retrieval or ranking pipeline.

A click isn't always a success, so click-through rate stays a secondary signal. Only after validating both quality improvements and operational guardrails would we gradually increase rollout.

Deploying a New Embedding Model

Deploying a new retrieval model requires special care.

Suppose we fine-tune the embedding model and improve offline relevance. We cannot safely replace only the query encoder while continuing to use incompatible old document embeddings.

Instead, we'd build a new vector index using the updated model. During this process, the existing index continues serving production traffic. New or updated documents should be incorporated into both indexes, or their changes replayed before the switch.

Once the new index is complete, we validate it with shadow queries and quality checks, then gradually route traffic to the new query encoder and matching index. The model and index must switch as a compatible pair, and if quality or latency regresses, we roll back to the previous version. This is effectively a blue-green deployment for semantic retrieval.

Monitoring the Search System

Once the system is live, we'd split model monitoring into several categories.

One important point is that semantic search quality can degrade even when every service appears healthy. Latency can be flat and error rates can be zero while the results quietly get worse, for example after a shift in the kinds of documents being added or the queries being asked.

That's why technical monitoring and relevance monitoring must work together.

Final Design

Let's summarize the architecture we've designed.

The diagram brings together the three parts of the system. Ingestion turns documents into embedded passages, indexing stores them in vector, lexical, and metadata stores, and serving runs hybrid retrieval, fusion, reranking, and authorization for every query. Relevance evaluation, model versioning, caching, monitoring, and safe deployment support all three.

The table below summarizes the key decisions we made along the way:

The most important lesson is that designing a semantic search engine isn't just about choosing an embedding model or vector database. It's about building a complete retrieval system that delivers relevant, fast, secure, and consistently reliable results.