Retrieval and generation work much like they do in a RAG system for a single company. The difference is that thousands of tenants share the same pipelines, indexes, and model capacity, and their sizes vary a lot. One tenant might have 40 help articles; another, 30 million support tickets.

This range in size affects most of the design. The platform has to keep each tenant's data separate, stop one busy tenant from slowing down the others, let tenants configure and pay for what they use, and move a tenant with 100 million chunks to a new embedding model without taking it offline.

In this chapter, we'll design a multi-tenant RAG platform. We'll start with the requirements and high-level architecture, then go deep on choosing between one index per tenant and a shared index, isolation guarantees, noisy neighbors and billing, and re-indexing large tenants without downtime.

Let's start by clarifying the requirements.

"Design a multi-tenant RAG platform" can describe anything from a single shared index to a product serving thousands of businesses, each with its own data and settings. Before getting into the design, we need to agree on the scope.

Here is how the conversation with the interviewer might go:

These answers set the main constraints for the design. Let's turn them into functional and non-functional requirements.

Functional Requirements

Non-Functional Requirements

Quality Bar

An answer is good when every claim is supported by a cited chunk from the tenant's data. When the data doesn't contain the answer, the assistant should say so instead of guessing.

Quality has to hold for each tenant individually. A platform-wide average of 90% grounded answers can hide a tenant whose connector broke last week and who now gets almost nothing right.

Tenant size drives storage and placement decisions, so the estimate starts there. Assume each chunk holds about 500 tokens, or roughly 2 KB of text.

The largest single tenant holds about 150 million chunks. Just 1% of tenants own half the data. That skew rules out treating every tenant the same way.

Storage

Keeping 8 TB of full-precision vectors in memory is expensive. A common approach is to keep quantized vectors and the graph in memory for search, then rescore the top candidates using full-precision vectors from disk.

Query Traffic and Token Cost

Assume 10 million queries per day across all tenants.

Each query sends about 6,000 input tokens to the model: 1,000 for platform and tenant instructions, 4,000 for eight retrieved chunks, and 1,000 for the question and recent conversation. The answer averages 400 output tokens.

At an illustrative price of $3 per million input tokens and $15 per million output tokens:

Generation is the largest cost in the system by a wide margin, so the platform needs per-tenant metering and model tiers to keep it in check. A tenant on a cheap plan sending a million queries a day could cost the platform more than it pays.

Ingestion

If 1% of chunks change each day, the platform re-embeds 20 million chunks, or 10 billion tokens. At $0.02 per million tokens for a small embedding model, that's about $200 per day.

Re-embedding the largest tenant from scratch means 75 billion tokens, about $1,500. At 5,000 chunks per second, it takes around 8 hours. The embedding cost is modest, but the rebuild has to happen without downtime for that tenant and without starving other tenants of embedding capacity.

The platform exposes two groups of APIs: one for managing data and configuration, and one for answering questions.

Here's a query request:

The response streams tokens, followed by a final event with citations and usage:

Two details in this contract matter for isolation.

First, the request body never contains a tenant_id. The platform derives the tenant from the API key. If tenant ID were a request parameter, a bug or a malicious caller could pass someone else's.

Second, end-user identity arrives as a token signed by the tenant's backend. The platform trusts the tenant to vouch for its own users, then uses the groups in that token to filter documents by permission.

We'll build the design one requirement at a time, starting with getting tenant data into the system.

Ingestion Path

Documents arrive through uploads or connector syncs. Every step carries the tenant ID with the data, and every store is partitioned by it.

Raw files land in object storage under a tenant-specific prefix. A message goes onto that tenant's ingestion queue, where workers parse the file, split it into chunks, and attach metadata such as the document ID, source permissions, and timestamps.

The embedding service turns each chunk into a vector, and the index writer stores it in both a vector index and a keyword index, since hybrid search handles exact terms like error codes and product names better than vectors alone.

Workers also save the parsed text. This pays off later: when we change the embedding model or chunking strategy, we can rebuild the index from stored text instead of re-crawling every source.

Connectors sync incrementally. They track a cursor or change feed for each source, so a tenant with a million Confluence pages doesn't trigger a full re-crawl every few minutes.

Query Path

When a question arrives, the platform has to work out who is asking, check that they have quota, retrieve from the right data, and generate an answer.

The flow for a single question:

Control Plane

The two paths above handle data. A set of control-plane services decides how each tenant is configured, where its data lives, and how much it can use.

The control plane and the data plane change at different speeds. Config and placement change a few times a day. Queries arrive hundreds of times per second. Keeping them separate means a slow config database doesn't add latency to every query.

The platform serves thousands of tenants with a small set of shared models. Running separate models per tenant would multiply GPU and operational cost without much benefit, since each tenant's knowledge comes from retrieval, not model weights.

Generation models are offered as tiers, such as a fast, inexpensive model and a more capable one. Tenants pick a tier, and the LLM gateway handles routing, fallbacks, and provider rate limits. Some platforms let enterprise tenants bring their own provider API key, which moves model cost onto the tenant's account.

The embedding model is a platform decision, not a tenant setting. Vectors from different embedding models live in different spaces and can't be compared, so every chunk in an index and every query against it must use the same model version. Letting each tenant pick any embedding model would fragment the index layout and multiply re-indexing work. Offering one default and perhaps one premium option keeps this manageable.

The reranker is shared. A cross-encoder scoring 50 candidates adds roughly 50 to 200 ms, and it lets the platform send fewer, better chunks to the generation model, which reduces cost.

Fine-tuning per tenant is rarely worth it here. RAG already supplies tenant knowledge, and tenant instructions handle tone and format. Training a shared model on tenant data also raises consent and leakage concerns, because a model can memorize and reproduce training text. If the platform fine-tunes at all, it should use data tenants have explicitly opted in.

Prompt structure affects cost. Placing the platform instructions first, then tenant instructions, then retrieved chunks and the question gives each tenant a stable prompt prefix. Providers that support prefix caching can then reuse the computation for the first 1,000 tokens across that tenant's requests.

The platform stores four kinds of data: tenant metadata, document metadata, the search indexes, and usage records.

A few choices are worth explaining.

content_hash on documents lets ingestion skip unchanged files during a sync. Connectors often report a document as modified when only its metadata changed, and re-embedding it would waste capacity.

acl_groups is copied onto every chunk so retrieval can filter by permission inside the index. Looking up permissions after retrieval would be too slow and would return too few results when most of the top matches are filtered out. The trade-off is that a permission change in the source system has to update every chunk of that document.

index_versions records which embedding model and chunking configuration produced each version of a tenant's index. The query path reads it to decide which model should embed the question.

event_id on usage events makes metering idempotent. If a metering event is retried after a network failure, the billing aggregator deduplicates it instead of charging the tenant twice.

Four questions make this problem harder than a single-tenant RAG system: how to lay out the index across tenants of very different sizes, how to guarantee isolation, how to stop noisy neighbors, and how to re-index a large tenant without downtime.

7.1 One Index per Tenant or a Shared Index?

This is the central design decision, and the right answer depends on tenant size.

Approach 1: A Shared Index with a Tenant Filter

All chunks from all tenants go into one large index, and each query filters by tenant_id. It's simple to operate and uses resources efficiently.

The problem shows up in vector search. An approximate nearest neighbor (ANN) index like HNSW finds similar vectors by walking a graph. A small tenant with 20,000 chunks makes up 0.001% of a 2-billion-chunk index.

If the search finds the global top 100 neighbors and then filters by tenant (post-filtering), it will almost always return nothing for that tenant. If it filters during the graph walk (pre-filtering), it skips almost every node it visits, latency climbs, and recall drops because the tenant's nodes aren't well connected to each other in the graph.

A shared index with a plain metadata filter works for keyword search, but breaks vector search for small tenants.

Approach 2: One Index per Tenant

Each tenant gets its own index. Isolation is clean, deleting a tenant means dropping one index, and re-indexing a tenant doesn't touch anyone else.

The cost is 5,000 separate indexes. Each one carries fixed overhead in memory, file handles, and cluster metadata, and most of them are tiny. In some vector databases, this per-collection overhead becomes costly at tens of thousands of collections, and cluster operations such as rebalancing and upgrades get slower as the count grows.

Approach 3: Partitioned Shared Index

Tenants share a cluster, but data is physically grouped by tenant inside it, so a query only searches its own tenant's partition. Several vector databases support this directly. At the time of writing, examples include Pinecone namespaces, Milvus partition keys, Weaviate's multi-tenancy with a shard per tenant, and Qdrant's tenant-aware payload indexes.

Small partitions don't even need an ANN graph. A brute-force scan of 20,000 vectors with 1,024 dimensions is about 20 million multiply-adds, which takes a few milliseconds and gives perfect recall.

Recommendation: Tier by Tenant Size

Tenant sizes range from 40 documents to 150 million chunks, so a tiered layout uses each approach where it fits best.

Small tenants share shards, with each tenant in its own partition and searched by brute force or a small graph. Medium tenants also share shards but get their own ANN partition. Large tenants and tenants paying for isolation get dedicated indexes, sized and scaled for their traffic.

The placement service records where each tenant lives, and the query path looks it up on every request. When a tenant grows past a threshold, the platform moves it to the next tier using the same re-indexing process described in Section 7.4.

7.2 Isolation Guarantees

Isolation is the requirement with the highest stakes. If one tenant sees another tenant's data, the platform risks contractual and legal exposure and loses customer trust, so the design can't rely on any single check.

The approach is defense in depth: several independent layers, each of which would prevent a leak even if another layer had a bug.

The tenant comes from authentication, never from the request body. A shared data access layer builds every index query and adds the tenant scope and permission filter itself. Application code can't issue a query without a tenant context, which rules out a whole class of bugs where a developer forgets the filter.

The partition or dedicated index provides physical separation. After retrieval, a final check confirms that every chunk's tenant_id matches the request's tenant before anything goes into the prompt. If the check fails, the request errors out and pages someone.

The LLM is never part of this enforcement. Telling the model "only use documents from tenant A" protects nothing if tenant B's chunk is already in the context window.

Leak Paths Outside the Index

The index is the obvious place to enforce isolation, but data flows through several other components:

Deletion

Deleting a document has to remove it from the raw store, the parsed text, both indexes, caches, and eventually backups. Many vector indexes mark deleted vectors with a tombstone and remove them physically during compaction, so a deleted chunk may still exist on disk for a while. It must never be returned in search results, though.

For tenants that require it, per-tenant encryption keys make account deletion verifiable. Each tenant's data is encrypted with its own key, and destroying the key renders the remaining copies, including those in backups, unreadable.

Testing Isolation Continuously

Code review alone can't confirm isolation, so the platform also tests it continuously. It runs synthetic canary tenants that contain documents with unique marker strings, then continuously sends queries from other tenants designed to retrieve them. If a marker ever shows up in another tenant's results, that's a critical alert.

7.3 Noisy Neighbors, Quotas, and Billing

Tenants compete for four shared resources: ingestion workers and embedding capacity, index search capacity, LLM provider throughput, and query concurrency. Without limits, one tenant importing 50 million documents or running a load test can raise latency for thousands of others.

Fair Ingestion

A single shared ingestion queue processes work in arrival order. If a large tenant enqueues 50 million documents, every other tenant's updates wait behind them, and the 5-minute freshness target fails for everyone.

Per-tenant queues with a fair scheduler fix this. Workers pull from tenant queues in weighted round-robin order, with a concurrency cap per tenant, so a small tenant's single document update gets processed promptly even while a large backfill is running.

Separating bulk and incremental work into two lanes adds another safeguard. Initial imports and re-indexing go into the bulk lane, which gets whatever capacity remains after the incremental lane is served. Freshness for day-to-day edits stays predictable, and large imports finish when capacity allows.

Query Quotas

The quota service keeps per-tenant token buckets in a fast store like Redis, with limits for requests per second, tokens per minute, and concurrent streams. The gateway checks request limits before doing any work.

Token limits are harder, since the output length isn't known in advance. The LLM gateway reserves an estimate based on input tokens plus the configured maximum output tokens, then reconciles with the actual count when the stream finishes.

When a tenant exceeds its limit, the platform returns HTTP 429 with a Retry-After header, or, if the tenant has enabled it, routes the request to a cheaper model tier instead of rejecting it.

Large tenants on dedicated indexes can't slow down shared shards. On shared shards, per-tenant concurrency caps ensure that one busy tenant can only use a bounded fraction of search capacity.

Model Capacity

Provider rate limits apply to the platform's account, not to individual tenants. If the platform has a limit of 20 million tokens per minute, one tenant could consume most of it. The LLM gateway divides that capacity into per-plan shares and lets tenants borrow unused capacity, but only up to a ceiling, so a burst from one tenant can't use the whole limit.

Metering and Billing

Every billable action emits a usage event: input and output tokens by model, embedding tokens during ingestion, and storage measured in GB-days. Events flow through a stream into an aggregator that produces per-tenant usage totals, and those totals feed both the usage API and invoicing.

The same events power quota alerts. A tenant approaching its monthly budget gets a warning, and the platform can apply a hard cap if the tenant has configured one.

7.4 Re-indexing Large Tenants Without Downtime

The platform needs to rebuild a tenant's index for several reasons: switching to a better embedding model, changing the chunking strategy, or moving a tenant from shared shards to a dedicated index. A small tenant can be rebuilt in seconds. The 150-million-chunk tenant takes about 8 hours, and its users expect answers throughout.

The approach is a blue-green rebuild with an alias. Queries go through an alias that points to the active index version. The new version is built alongside the old one, and the alias switches only after the new version is complete and validated.

The process runs in five steps:

The query path must embed each question with the model that built the active version. It reads the alias, looks up the version in index_versions, and uses that version's embedding model. A query embedded with the new model and searched against the old index would return poor results without any error, so this lookup can't be skipped.

Two operational limits keep the rebuild from hurting other tenants. The backfill runs in the bulk ingestion lane with a rate cap, and the tenant temporarily needs double storage while both versions exist, which the placement service has to account for.

For a platform-wide embedding model upgrade, the same process runs tenant by tenant. It starts with a few internal and canary tenants, then proceeds in waves, watching per-tenant quality metrics after each wave.

With 5,000 tenants, each with different data, the platform can't rely on hand-labeled test sets for every tenant.

Offline Evaluation

The platform uses three layers of offline evaluation:

Online Evaluation

In production, the platform tracks feedback ratings, citation clicks, the rate of "I don't know" answers, and, for support use cases, how often users escalate to a human. An LLM judge scores a sample of answers for groundedness.

All of these metrics are tracked per tenant, with alerts on sudden drops. A spike in "I don't know" answers for one tenant often points to a broken connector or an expired source credential rather than a model problem, and per-tenant alerts catch it the same day.

Safety

Tenant documents are untrusted input. A support platform indexes tickets written by the tenant's customers, and any of them could contain text like "ignore previous instructions." Retrieved chunks go into the prompt clearly marked as reference data, and output checks look for signs that instructions in a chunk took over.

Since this platform only reads data and answers questions, a successful injection has limited impact. The risk grows considerably if tenants later add tools that take actions, which would need the stricter permission controls used for agents.

Within a tenant, end users may try to extract documents they shouldn't see. Permission filtering in the index handles this: chunks a user isn't allowed to read never reach the model, so no phrasing of the question can get them out.

Tenants can also turn on PII redaction for answers and logs, which matters for tenants in regulated industries.

Scaling to 10x

At 50,000 tenants, 20 billion chunks, and 100 million queries per day, several parts of the design come under pressure.

Idle tenants waste memory. Many tenants query rarely, but their partitions still occupy RAM. Tiered storage helps: active tenants stay in memory, while idle tenants' indexes move to SSD or object storage and load on their first query. The trade-off is a cold-start delay of a second or more for the first query after a quiet period. Some vector databases support offloading inactive tenants natively.

Shared shards need rebalancing. As tenants grow at different rates, some shards fill up and get hot. The placement service needs automated rebalancing that moves tenants between shards with the same alias-based process used for re-indexing.

The control plane becomes a hot path. Config and placement lookups on every query need aggressive caching, and the quota store needs sharding by tenant.

Model capacity becomes the main constraint. At 10x, generation would cost about $72 million per month at list prices. Provisioned throughput, committed-use discounts, or self-hosting open-weight models for the cheaper tier become worth evaluating.

Reducing Cost

Generation dominates the bill, so that's where savings come from. Two changes make a large difference:

Together, these changes cut the cost per query from $0.024 to about $0.008, a reduction of roughly two-thirds. Prefix caching for tenant instructions and a per-tenant response cache for repeated questions add further savings, especially for support tenants where the same questions come up many times a day.

Each change needs per-tenant evaluation before rollout. A routing rule that works well for a tenant's FAQ-style questions may hurt a tenant whose users ask complex legal questions, and only per-tenant metrics show the difference.

A multi-tenant RAG platform runs the same retrieval and generation pipeline as a single-company system, but thousands of tenants with very different sizes share it. Most of the design work goes into keeping tenants separate, keeping them from affecting each other, and handling the few very large tenants differently from the many small ones.