Traditional e-commerce search works well when shoppers know the product name. If they type "Nike Pegasus 41", a keyword index finds it in milliseconds.

It works less well when shoppers describe what they need. A query like "waterproof hiking boots for wide feet, under $150" combines a category, two product attributes, a fit requirement, and a price limit in one sentence. A keyword engine treats all of it as text to match, so it might return a $220 boot whose description happens to mention "wide" and "waterproof".

A conversational version of that search might go like this:

Each turn depends on the ones before it. "Lighter" only makes sense relative to the boots already on screen, and "the second one" refers to a specific result.

Unlike a chat assistant, this system returns a ranked grid of real products rather than a paragraph of text. The LLM helps interpret what the shopper wants, but prices, sizes, and stock must come from the catalog, and the results need to load about as fast as a normal search page.

In this chapter, we'll design a conversational product search system. We'll start with the requirements and high-level architecture, then go deep on turning free text into structured filters, combining LLM output with the ranking model, the latency budget per stage, and managing conversation state.

Let's start by clarifying the requirements.

Shopping assistants can mean several different products, from a chatbot that recommends a few items to a full search experience. So the first step is agreeing on what we're building.

A requirements discussion for this problem might look like this:

The conversation gives us the constraints we need. Let's write them down as functional and non-functional requirements.

Functional Requirements

Non-Functional Requirements

Quality Bar

It helps to agree on how quality will be measured, since "good results" is vague. We'll track three things:

Traffic

LLM Tokens and Cost

Every turn needs at least one LLM call to understand the query. That call includes fixed instructions, the attribute schema for the likely category, the current conversation state, and the new message.

At an illustrative small-model price of $0.15 per million input tokens and $0.60 per million output tokens:

That is for query understanding alone. If we also generate a short message for every turn, the total grows by another 50% or so. This math pushes us toward three decisions early: use a small model, cache aggressively, and skip the LLM when a query doesn't need it. We'll come back to these in the cost section.

Storage

The vector index fits comfortably across a handful of shards with replicas for throughput. Conversation state is small and short-lived, so an in-memory store like Redis works well.

The API is organized around a conversation. Each shopper message creates a new turn.

Start a conversation

Send a turn

The filter_edits field carries direct chip changes, such as removing the "waterproof" chip. When a turn contains only filter edits, we skip the LLM entirely, since the change is already structured.

Response (server-sent events)

The response streams in stages so the grid can render before any generated text is ready:

Two details matter here. The interpretation event lets the UI show filter chips immediately, which makes the system's understanding visible and easy to correct. And prices in the results event come from the catalog, never from generated text.

Let's build the design one requirement at a time, starting with a single search and adding pieces as we go.

Step 1: Understand a Single Query and Retrieve Products

The simplest version takes the shopper's text, turns it into a structured query, retrieves candidates, and ranks them.

Query understanding converts free text into a structured search request: a category, hard filters like price_max and width, and a remaining text query for everything that isn't a filter.

Hybrid retrieval runs a lexical search (BM25) and a vector search in parallel, both restricted by the hard filters. Lexical search catches exact terms like brand names and model numbers. Vector search catches descriptions that don't share words with the query, such as "boots for long day hikes" matching products described as "all-day comfort trail footwear". The two candidate lists are merged, for example with reciprocal rank fusion, into roughly 500 candidates.

Ranking scores those candidates with a learning-to-rank model trained on clicks, add-to-carts, and purchases.

The LLM's role is limited. It never sees the 50 million products and never chooses which products appear. It only translates language into a query that a search engine can execute.

Step 2: Support Multi-Turn Refinement

To handle "only brown ones", the system needs to remember what was searched before. We add a conversation state store that holds a structured search state for each conversation.

On each turn, query understanding receives the current state and the new message, and returns an updated state. After the second turn in our example, the state looks like this:

Storing structured state instead of the full transcript keeps the LLM input short and makes each turn's change easy to inspect. A few of the last messages are still kept, because references like "the second one" need the list of products that was actually shown.

Step 3: Keep Filters Accurate and Fresh

Filters are only as accurate as the data behind them. We need a catalog pipeline that keeps the index in sync, and a final check before results are returned.

Step 4: Add Personalization

We add a user feature store with signals such as preferred brands, past purchase price ranges, and known sizes. The ranking model uses these as features.

Personalization should influence ranking, not add filters the shopper didn't ask for. If a shopper usually buys size 10 but didn't say so, the ranking can boost products that have size 10 in stock. Turning it into a hard filter would hide results the shopper may want, for example when shopping for someone else.

Step 5: Generate a Message and Handle Failures

Finally, a response generator writes the short message above the grid. Most turns don't need it, so it runs only when there's something useful to say: a clarifying question, an explanation of relaxed filters, or a brief comparison the shopper asked for.

Every LLM call also gets a timeout. If query understanding doesn't respond within its budget, the system falls back to plain keyword search on the raw text, plus any chips the shopper already has.

The Complete Design

The request path runs from the Search API through query understanding, retrieval, ranking, and the availability check. The data path at the bottom keeps the index current. The "Needs LLM?" router is a lightweight classifier that sends simple keyword queries like "airpods pro" straight to retrieval, which saves both latency and money on a large share of traffic.

This design uses several models, and only some of them are LLMs.

Why Not Let the LLM Pick the Products?

One option is to retrieve 50 products, put them in the LLM context, and ask it to choose the best ten. At this scale, it has three problems. A 50-product context can add thousands of input tokens and a second or more of latency. The LLM has no access to the click and purchase history that tells us which products shoppers actually prefer. And when the LLM writes about products, it can state a wrong price or a feature the product doesn't have.

So the LLM's job stays narrow: understand language and produce a structured query. Search and ranking systems that are already fast and well-measured do the rest.

Prompting, Fine-Tuning, and Distillation

A practical path is to start by prompting a capable model with the attribute schema and a set of examples. This gets a working system quickly and produces labeled data.

Once traffic builds up, we can fine-tune a smaller model on logged queries, using outputs from the larger model that were reviewed or corrected by humans. A fine-tuned small model can match the larger model on this narrow task while cutting latency and cost several times. It also becomes a candidate for self-hosting if API costs grow.

The embedding model follows the same idea. General-purpose embeddings are a reasonable start, but fine-tuning on pairs of queries and purchased products usually improves retrieval for shopping-specific language like "wide toe box" or "zero drop".

Product Document

Each product is indexed with its variants, because stock and size live at the variant level. "Size 11 in stock" is a filter on variants, not on the product.

Most search engines (Elasticsearch, OpenSearch, Vespa) support nested documents or parent-child structures so that a filter like color = brown AND size = 11 AND in_stock = true matches a single variant rather than three different ones.

Facet Catalog

The facet catalog lists, for each category, which attributes exist and which values are valid. Hiking boots have width and waterproof. TVs have screen_size and refresh_rate. Query understanding uses this catalog to know what it can filter on, and validation uses it to reject values that don't exist.

Conversation State and Traces

Linking interactions to turn IDs is easy to overlook, but it's what lets us train the ranker and evaluate query understanding using real outcomes.

7.1 Turning Free Text into Structured Filters

Filter extraction comes first in the deep dives because ranking can't fix a wrong filter. If a filter is wrong, the right products were never retrieved.

Approach 1: Rules and Dictionaries

A traditional query parser uses regular expressions for prices ("under $150") and dictionaries for brands, colors, and sizes. It's fast and predictable, and it handles common patterns well.

It struggles with phrasing outside its rules. "Something I can wear to a beach wedding" has no dictionary terms. Negations like "not leather" and relative refinements like "cheaper" need context that rules can't track across turns.

Approach 2: LLM with a Schema

An LLM receives the facet schema for the category, the current state, and the new message, and returns JSON. Structured output or constrained decoding guarantees the JSON is well-formed and that fields match the schema.

This handles nuanced phrasing and multi-turn references well. However, a well-formed output can still be wrong. The model might produce color: "espresso", which isn't a filterable value, or set price_max: 150 when the shopper said "around $150".

Approach 3: LLM Extraction with Validation and Grounding

A practical approach combines the LLM with deterministic checks:

The hard versus soft distinction is central. "Under $150", "size 11", and "waterproof" are hard constraints, so they become filters. "Lightweight", "stylish", or "around $150" are soft preferences, so they become ranking boosts or text terms. Treating soft preferences as filters can produce empty or oddly narrow results.

Relative refinements need special handling. When a shopper says "cheaper", the model doesn't know what cheaper means in dollars. One reasonable approach is to let the LLM emit an operation like {"op": "decrease", "field": "price"} and let code compute a new price_max from the prices currently shown, for example the 30th percentile. Arithmetic should be done in code, not by the model.

The full flow for one turn looks like this:

Handling Zero Results

Strict filters will sometimes return nothing, for example wide, waterproof, brown boots under $150 in size 15. Instead of showing an empty page, the system relaxes constraints in a defined order: soft preferences first, then the hard constraint the shopper stated least recently. The response message tells the shopper what changed, such as "No exact matches in size 15, so these include sizes 14 and 15 in regular width." Being explicit about relaxed filters keeps the shopper's trust.

Recommendation

Use Approach 3. The LLM handles language, and code handles validation, normalization, and arithmetic. For head queries that repeat often, the validated output can be cached so most of them never reach the LLM.

7.2 Combining LLM Output with the Ranking Model

Once the LLM has interpreted the query, we need to decide how much influence it has on the final order of products.

Approach 1: The LLM Reranks the Top Results

Retrieve 50 candidates and ask the LLM to reorder them. This can capture subtle intent, like matching "beach wedding" to linen and light colors.

The costs are high: a second or more of extra latency, thousands of tokens per query, sensitivity to the order in which candidates appear, and no knowledge of what shoppers actually buy.

Approach 2: The LLM Only Produces the Query

The LLM produces filters and a text query, and the existing learning-to-rank model orders the results. It's fast, and the ranker already uses features like click-through rate, conversion rate, ratings, price relative to the category, and personalization.

The weakness is that subtle intent captured by the LLM, such as "for long day hikes", may get lost if it only survives as a few text terms.

Approach 3: LLM Signals as Ranking Features

Keep the ranker in charge, but give it signals derived from the LLM:

Recommendation

Use Approach 3 as the default. Most of the LLM's value moves offline, where cost and latency are easier to control, and the online ranker stays fast and grounded in real shopper behavior. If evaluation shows a clear gain for complex queries, an LLM rerank of only the top 10 can be added for that slice of traffic, and it should run after the grid has already rendered so it never blocks the first results.

7.3 Latency Budget per Stage

With a 1-second p95 target for the first results, every stage needs a budget. The LLM call is the largest single cost, so the rest of the pipeline has to be tight.

Stage budgets don't add up perfectly at p95, since slow requests in one stage aren't always slow in others, but the table gives each team a clear target. The roughly 350 ms of headroom covers network variation and LLM latency spikes.

Several techniques keep the pipeline within budget:

On a first turn, query understanding and a speculative keyword search run side by side:

When query understanding finishes in time, its filtered retrieval wins. When it times out, the speculative results are ranked and shown, so the shopper still sees products within budget.

7.4 Managing Conversation State

Multi-turn conversations raise a question that single searches don't: when should earlier constraints still apply?

Consider a shopper who searches for hiking boots under $150, refines to brown, and then types "what about rain jackets?" Carrying price_max: 150 and color: brown into a rain jacket search might be helpful or might be wrong. A brown filter almost certainly shouldn't carry over, while the budget might.

We handle this with a few explicit rules:

Keeping these rules in code, rather than leaving them all to the LLM, makes behavior consistent and easy to test with scripted conversations.

Offline Evaluation

We evaluate each stage separately, since an end-to-end score alone can't tell us which stage caused a failure.

Online Evaluation

New models and ranking changes are released through A/B tests. The primary metrics are add-to-cart rate and conversion. Supporting metrics include zero-result rate, reformulation rate (shoppers rephrasing because results missed), fallback rate, and p95 latency per stage.

Safety and Trust

Scaling to 10x

At 10 times the traffic, roughly 30,000 QPS during sales, three areas need attention.

LLM throughput and cost. Hosted API rate limits and costs become the main constraint. A fine-tuned small model served on our own GPUs, with a high cache hit rate in front of it, becomes worth the operational effort.

Index write load. During sales, inventory updates can arrive at thousands per second. Updating the main search index for every change causes heavy segment merging in Lucene-based engines and hurts query latency. A common fix is to move fast-changing fields like stock into a separate in-memory availability service that retrieval and ranking consult, and to update the main index in batches.

Retrieval capacity. The index is sharded by product ID, and replicas scale with read traffic. Since sale events are predictable, capacity can be added ahead of time.

Graceful Degradation

When the system is overloaded, features are turned off in a defined order:

Search keeps working at every step, and only the conversational features degrade.

Reducing Cost

The router, the cache, and conditional message generation can together cut the number of LLM calls by half or more, depending on how traffic is distributed. Distillation and prompt caching reduce the cost of the calls that remain.

Conversational product search uses an LLM to understand shoppers, while proven search and ranking systems decide which products appear. That division keeps latency, filter accuracy, and cost under control.