Most companies keep their knowledge in many different tools. The onboarding guide might live in Confluence, the travel policy in a Google Drive PDF, the fix for a recurring deployment error in an old Jira ticket, and the answer to "which team owns the billing service?" in a Slack thread. Finding an answer often means searching several places or asking a colleague.

In this chapter, we'll design a Q&A assistant over company documents. We'll start with the requirements and high-level architecture, then go deep on access control at retrieval time, incremental sync and deletes, retrieval quality, grounded answers and citations, and evaluating faithfulness.

Let's start by clarifying the requirements.

"Design a Q&A assistant over company documents" leaves a lot open, from which sources it reads to who is allowed to see what. That's why it's worth clarifying the scope before getting into the design.

Here is how a requirements discussion might unfold in an interview:

This conversation reveals several constraints that will influence our design. Let's formalize these into functional and non-functional requirements.

Functional Requirements

Non-Functional Requirements

Actions such as filing tickets, image understanding beyond text extraction, and answering external customers are out of scope.

Uptime and latency matter, but they don't show whether the answers are correct, so we also need quality metrics. We track metrics at four levels.

Abstention needs to be measured in both directions. An assistant that says "I don't know" to half the questions it could have answered is technically safe, but employees will stop relying on it. The evaluation set needs answerable and unanswerable questions so you can tune the trade-off between the two.

These numbers tell us where the hard parts are. Let's start with traffic.

Twenty-five queries per second is modest for the API servers. The more relevant number is token throughput to the model provider, which we'll calculate shortly.

Next, the corpus.

This fits comfortably on a small cluster of index nodes.

Embedding the corpus is also inexpensive. At an illustrative price of $0.02 per million tokens, embedding about 16.5 billion tokens (including overlap) costs around $330. If 2% of documents change each day, re-embedding them costs a few dollars per day.

The initial backfill is limited mainly by the source APIs. SaaS tools throttle API calls, and documents need to be downloaded, parsed, and sometimes OCR'd. At a sustained 50 documents per second across all connectors, 10 million documents take about 200,000 seconds, or roughly 2.3 days. The design needs to keep that backfill from blocking normal updates.

Finally, the generation cost per query:

Two observations come out of this. First, generation dominates the cost, so the cheapest query is one that never reaches the large model. Second, 8.3 million input tokens per minute at peak is above the default rate limits many providers give a new account, so capacity has to be arranged with the provider ahead of launch.

The system has three paths. The ingestion path pulls documents and permissions from the sources and keeps the index current. The query path answers a question in a couple of seconds. The feedback path captures traces and ratings so we can measure and improve quality.

The top half of the diagram runs continuously in the background. The bottom half runs once per question. They meet at the index and the ACL store, which is where most of the interesting design decisions live.

The Ingestion Path

Each source gets a connector that knows how to list documents, fetch content, read permissions, and detect changes. Connectors emit change events into a queue rather than processing documents inline, so a slow parser or an embedding outage doesn't stall change detection.

Workers consume the queue and run each document through the same pipeline:

A separate identity service syncs users and groups from the identity provider, such as Okta or Microsoft Entra ID, along with source-specific groups like Confluence groups and Slack channel membership.

The Query Path

When an employee asks a question, the request moves through these steps:

Here is the latency budget for the time to first token:

That leaves some headroom under the 2-second target. Most of it goes to the model, so the steps before generation need to stay fast. Skipping the rewrite for first-turn questions, which are usually standalone, saves 300 ms on the most common request.

The next sections cover the parts of this design that need more work, starting with permissions.

Suppose an employee asks, "What's the plan for the reorg next quarter?" A draft reorg document exists in a restricted HR folder. If any part of that document reaches the model's context, the model may summarize it, and the leak has already happened. Instructions in the prompt such as "don't reveal restricted content" do not protect against this. Access control has to be enforced before content enters the context window.

There are three ways to do it.

Approach 1: Filter After Retrieval

Retrieve the top results without considering permissions, then check each one against the source system's API and drop the ones the employee can't access.

This is always up to date, because it asks the source directly. But it has two problems. Every query triggers several live API calls, which adds latency and quickly runs into rate limits. Also, if the employee can only see a small slice of the corpus, the top 50 results may all be filtered out, leaving nothing to answer with even though relevant documents they can access exist further down the ranking.

Approach 2: Separate Index per User or Group

Build a dedicated index for each user or group so that search only ever touches allowed content.

This gives strong isolation, but it doesn't fit how company permissions work. Employees belong to hundreds of overlapping groups, so a single shared document would have to be copied into thousands of indexes, and one permission change could mean rewriting data in many places.

Approach 3: Store Permissions in the Index and Filter During Search

At ingestion, store each chunk's allowed principals as metadata. At query time, expand the employee into their full set of principals and pass it as a filter to both the keyword and vector search. The search engine only returns chunks where the two sets overlap.

Filtering during search is the right default for this system. Making it correct takes a few more decisions.

Store groups, not expanded users. If a document is shared with group:eng-platform, store that group ID on the chunks rather than the 400 user IDs inside it. When someone joins or leaves the group, only the identity service needs to update. If you expand groups at index time, one membership change can force you to rewrite millions of chunks.

Expand the user at query time. The identity service keeps a precomputed map from each user to all their principals, including nested groups, and caches it with a short TTL. An employee in 300 groups produces a filter with about 300 IDs, which keyword engines and most vector databases handle efficiently.

Compute the effective permission in the connector. Source permission models are complicated. Confluence combines space permissions with page restrictions inherited from parent pages. Jira has project permission schemes and issue-level security. Each connector must translate its source's rules into a flat list of allowed principals. A mistake in this translation code can expose documents to the wrong people, so it deserves its own tests.

Decide how to treat link-shared documents. A Google Drive file shared as "anyone in the company with the link" is technically accessible to everyone. But its owner may have assumed that only people who received the link would ever find it. Indexing it for everyone makes it discoverable in a way it wasn't before. A safer default is to restrict such documents to explicitly listed users and groups unless the company decides otherwise.

Watch for selective filters. Approximate nearest neighbor indexes like HNSW are fast because they explore only a small part of a graph. When the filter excludes most of the corpus, the search may visit many nodes it then has to discard and return too few results. Several vector databases handle this by estimating the filter's selectivity and switching to an exact scan over the allowed subset when it is small. Test this with your most restricted users, not your average ones.

Handling Permission Changes Between Index Updates

Approach 3 is only as fresh as the index metadata. If someone removes Bob's access to a document at 10:00 and the index updates at 10:04, Bob could retrieve it at 10:02.

To shrink that window, permission changes flow through a fast lane that updates the ACL store first, usually within seconds. After reranking, the orchestrator re-checks the final eight candidates against the ACL store before they enter the prompt. That check is a key-value lookup and adds only a few milliseconds. For the most sensitive sources, such as HR or legal spaces, you can also verify the final candidates against the source API directly and accept the extra latency.

The system should fail closed. If the identity service is unreachable and the cached principals have expired, the request should fail rather than run an unfiltered search.

Leaks Outside Retrieval

Retrieval isn't the only place where restricted content can escape. A response cache shared between users can return an answer built from documents the second user can't see. Traces viewed by the engineering team contain full prompts with retrieved text. A "share this answer" link can expose content to someone without access. Each of these needs the same permission discipline as the index: cache keys scoped by access, access-controlled trace viewers with redaction, and shared answers re-checked against the viewer's permissions.

Permissions also change constantly, and so does content. That brings us to the second deep dive: keeping the index in sync with the sources.

A naive approach re-crawls every source each night. With 10 million documents and rate-limited APIs, a full crawl takes days, so a nightly crawl can't even finish before the next one starts. The system has to process changes incrementally.

Detecting Changes

Sources offer different ways to detect changes, and each has a weakness. A reliable connector usually combines all three.

Many APIs expose a change cursor. Google Drive's changes API returns a page token you store and resume from, and Microsoft Graph supports delta queries for SharePoint and OneDrive. Polling with these cursors every few minutes catches what webhooks miss.

Reconciliation lists all document IDs in the source and compares them to the IDs in the index. Anything in the index that no longer exists in the source is a missed delete. Without this step, deleted documents can stay in the index indefinitely.

Processing a Change

Every change event becomes a job keyed by document ID. The pipeline is designed so that processing the same event twice, or out of order, is harmless.

Events from all three detection methods are normalized and grouped by document. A page edited 20 times in ten minutes becomes a single job that fetches the latest version, so a busy page doesn't trigger 20 rounds of embedding.

Jobs then go into separate lanes:

Without separate lanes, connecting a new SharePoint site with two million documents would fill the queue and delay every permission revocation behind it. Each lane also has per-source rate limits with backoff, because exceeding a SaaS API's quota can get the connector blocked for everyone.

For content changes, the pipeline compares a hash of the new text with the stored one. If only metadata changed, it skips embedding. If the text changed, it re-chunks the document and compares hashes at the chunk level, so editing one paragraph of a 40-page document re-embeds only the affected chunks.

Ordering and Versioning

Events can arrive out of order. If an older update is processed after a newer one, it can overwrite the index with stale content. To prevent this, every job carries the source's version number or modification timestamp, and the writer ignores any job older than what's already stored. Partitioning the queue by document ID also keeps updates for the same document in sequence.

When a document's content changes, the writer adds the new chunks tagged with the new version, then removes the chunks from the old version. For a moment, both versions may exist in the index. A query-time filter on the current version from the document store hides the old chunks during that window.

Handling Deletes

Deletes need extra care, because a deleted document that still shows up in answers is both a quality bug and, sometimes, a compliance issue.

With permissions and freshness handled, the index contains the right content for each employee. The next question is whether retrieval actually finds the right passage.

Many bad answers from a Q&A assistant trace back to retrieval. If the right passage isn't in the context, the model either abstains or, worse, answers from a passage that's close but wrong. Company documents make retrieval harder than general web text in a few specific ways.

Internal vocabulary. Questions contain ticket IDs like PLAT-4521, error codes, product codenames, and team acronyms. Embedding models often represent these poorly because they rarely appeared in the model's training data. Keyword search matches them exactly.

Paraphrased questions. "Can I expense a home office chair?" should match a section titled "Remote Work Equipment Reimbursement," even though they share almost no words. Vector search handles this well.

This is why the design uses hybrid search. Both searches run in parallel with the same permission filter, and their results are merged with Reciprocal Rank Fusion (RRF), which scores each chunk by summing 1 / (k + rank) across the result lists, with k commonly set to 60. RRF uses only ranks, so it doesn't need the two systems' scores to be on the same scale.

The reranker is a cross-encoder that reads the question and each candidate passage together and scores how well the passage answers it. It's more accurate than the first-stage searches, which score the query and passage separately, but too slow to run over the whole corpus. Running it on the top 50 candidates adds roughly 50 to 200 ms, and it often gives a noticeable improvement in answer quality for that cost.

Chunking decides what a "passage" is. Splitting on a fixed token count can cut a table in half or separate a heading from its content. A better approach splits along document structure: sections under headings for wiki pages, the description and resolution fields for tickets, and whole threads for Slack. Prepending each chunk with its document title and section path, such as "Travel Policy > International Travel > Per Diem," gives both the embedding and the model context that a bare paragraph lacks.

Metadata boosts fix a problem that pure relevance can't. A company might have both "Travel Policy 2023" and "Travel Policy 2025," and both would be equally relevant to a question about per diem rates. Boosting recent documents, demoting pages marked archived or deprecated, and preferring official policy spaces over personal notes keeps the assistant from citing a policy that no longer applies.

Even with good retrieval, the model can still misuse what it receives. That's the focus of the next deep dive.

The model's job is narrow: answer the question using only the provided passages, and cite them. The prompt makes that contract explicit.

Each source includes a title, update date, and URL, so the model can reason about recency and the client can render citation links. Keeping the system instructions identical across requests also makes them eligible for prompt caching, which reduces input cost.

Abstaining Before Generation

Many unanswerable questions can be detected before calling the large model. If the reranker's top score falls below a threshold, nothing retrieved is relevant enough to answer from. In that case, the assistant skips generation, responds with "I couldn't find this in the documents you have access to," and lists the three closest documents in case one of them helps.

This gate improves trust and cuts cost at the same time, since abstained queries never reach the large model. The threshold is tuned on the evaluation set by balancing correct abstentions against wrongly refused answerable questions.

The gate doesn't catch everything. A question about the 2026 bonus structure may retrieve a highly relevant 2025 bonus document that doesn't contain the answer. The prompt instruction handles this second case, where the passages are on topic but incomplete.

Verifying Citations

A citation that exists isn't necessarily a citation that supports the claim. Verification happens at two levels.

The structural check is cheap and runs on every response. It confirms that each cited number refers to a source that was actually in the prompt, and that every factual sentence has a citation. This catches invented source numbers and uncited claims.

The support check asks whether the cited passage actually backs up the sentence. A small natural language inference model or an LLM judge compares each sentence with its cited passage. Running it synchronously on every response adds latency and cost, so a practical setup runs it on completed sentences in parallel with streaming for high-risk spaces like HR and legal, and asynchronously on a sample of everything else. Unsupported sentences in the synchronous path get a visible warning or are removed. The asynchronous results feed the quality dashboards.

Checks like these only catch problems in a single response. To know whether the assistant is getting better or worse over time, we need systematic evaluation.

Every change to this system can affect answer quality without causing any visible error: a new chunking strategy, a different embedding model, a prompt tweak, or a model version upgrade. Evaluation is what lets you make those changes with confidence.

Building the Evaluation Set

Start with real questions. Helpdesk tickets and channels like #ask-it or #ask-hr are a good source, since they contain questions employees actually asked, often with the correct answer in the replies. Subject-matter experts label each question with the expected answer and the documents that support it. A few hundred labeled questions are enough to start.

The set should deliberately include:

Evaluating Each Stage Separately

An end-to-end score tells you something broke but not where. Evaluating retrieval and generation separately points to the stage that needs fixing.

To measure faithfulness, an LLM judge splits the answer into individual claims and checks each against the cited passages. The faithfulness score is the fraction of claims that are supported. The judge itself must be calibrated: have humans label a few hundred claims, and check that the judge agrees with them before trusting its scores.

Testing for Permission Leaks

Permission correctness is easier to test than faithfulness because it's binary. Create canary documents in each source, each containing a unique random string and shared only with a dedicated test group. Then run queries as test users outside that group, both in CI and on a schedule in production. If a canary string ever appears in retrieved chunks or answers, that's a high-severity incident. This catches bugs in connector permission logic, which are hard to find through code review alone.

Online Signals

After deployment, ratings and behavior provide continuous signals. Thumbs-down ratings, employees rephrasing the same question right away, and answers whose citations nobody clicks all point to potential problems. A daily sample of traces goes through the LLM judge, and confirmed failures are added to the evaluation set so the next release is tested against them.

One change deserves special mention. Switching embedding models requires re-embedding the entire corpus, because vectors from different models aren't comparable. The safe approach is to build the new index alongside the old one, compare both on the evaluation set and on shadow traffic, and switch only once the new index is better.

With the deep dives done, let's return to the numbers from the estimation.

Scaling

The API servers and orchestrator are stateless and scale horizontally. The index holds about 33 million chunks, which a small number of shards with replicas can serve. Replicas matter more than shards here, because they let the index handle peak query load and survive a node failure. Ingestion workers scale independently based on queue depth, so a large backfill adds workers without touching the query path.

Reducing Cost

At around $150,000 per month, most of the spend is generation. The main levers are:

Handling Failures

A Q&A assistant over company documents is built on RAG, and much of the design work covers the parts around the model: permissions, freshness, retrieval quality, and evaluation.