A product recommendation system decides which items to put in front of a shopper on an e-commerce surface: the Amazon home page, the eBay "sponsored" rail, the product detail page's "frequently bought together" row, the cart page's "customers also bought" strip, the post-purchase replenishment email. The catalog has hundreds of millions of SKUs, inventory and prices move minute by minute, and the signal that actually matters, a purchase, is rare and delayed by hours or days. That combination makes it a different problem from video recommendation even though the architecture looks superficially similar: the target shifts from engagement to purchase intent, the catalog is live rather than archival, and the system has to care about margin, inventory, and returns, not just watch time.

The first thing to pin down in an interview is which surface is being designed. A home page rail with no query and a cart page rail with three items already in the basket are both called "product recommendations" and they share infrastructure, but the ranking objective, the retrieval pool, and the evaluation story are noticeably different. Get the surface wrong and every trade-off downstream is mis-calibrated.

Clarifying Questions

Candidate: "Which surface are we designing for: the home page, product detail page, cart page, search results, post-purchase email, or all of them?"

Interviewer: "Home page and product detail page, with the cart page as a stretch deep dive. Assume search ranking is a separate team's problem."

Candidate: "What's the catalog size and how fast does it change?"

Interviewer: "About 500 million SKUs active in the catalog. Sellers add roughly 1 million new SKUs per day. Inventory and price updates arrive continuously, tens of millions of updates per hour during peak."

Candidate: "User scale and traffic?"

Interviewer: "300 million daily active users. Peak is about 50,000 recommendation requests per second globally. Assume logged-in only; logged-out traffic gets a simpler popularity-by-geo page that you don't need to cover."

Candidate: "Latency budget?"

Interviewer: "200 milliseconds p99 end to end from gateway to first byte. Your ranking pipeline owns about 130 milliseconds of that."

Candidate: "What signals are available for training and serving?"

Interviewer: "Clickstream, add-to-cart events, checkouts, refunds and returns, reviews and ratings, browse history, search queries, full order history, and a rich catalog feed with title, brand, category tree, price, image, text description, and seller metadata. Inventory and pricing are live services you can call."

Candidate: "Does inventory get treated as a hard constraint?"

Interviewer: "Out-of-stock in the user's region is a hard filter. Low-stock is a soft signal that the ranker can use, but you can't promise something you can't ship."

Candidate: "What about regulated or restricted items, alcohol, prescription products, region-locked SKUs?"

Interviewer: "Hard filter. Age-gated items are filtered by user profile, region-locked items by shipping address. Treat this as an allow-list check before the final slate goes out."

Candidate: "Is there an attribution window for purchase labels?"

Interviewer: "Use 7 days for the purchase label with a 30-day return window. A purchase that was returned inside 30 days gets the label flipped or heavily down-weighted. You'll need to pick an attribution model: last-click, last-recommendation, or fractional credit."

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

The single biggest framing decision is what to predict. Clicks are cheap and abundant and almost always the wrong target. Training a ranker on clicks rewards the same thing that rewards clickbait on a content site: flashy images, oddly low prices, and titles that overstate what the product does. The user arrives on the product page, realizes the thing doesn't fit their need, and bounces. The ranker looks good on offline click metrics and ships, and three weeks later someone notices that conversion rate is down.

Purchase is the real target, but a purchase is rare (roughly 1 in 50 clicks converts on most catalogs) and delayed (median several hours, long tail up to days). Training a single-head model purely on purchase labels leaves most of the loss function staring at zeros. The standard move is multi-task learning: a shared representation feeds several heads, each with its own label and its own loss, and the final ranking score is a calibrated blend.

The heads that matter for product rec are pClick, pAddToCart, pPurchase, and pReturn. pClick and pAddToCart give the model dense signal early in training. pPurchase is the sparse but high-value label. pReturn is a negative signal that prevents the ranker from chasing purchases that get refunded. The score at serve time is a weighted combination that gets tuned on long-horizon A/B tests:

Expected GMV turns the purchase probability into an expected-revenue estimate, which is what e-commerce cares about. The return penalty uses the same GMV factor because a $500 return is worse than a $10 return. The weights are not free parameters; they are set by business policy and re-tuned a few times a year.

Retrieval is framed as a nearest-neighbor problem over a learned embedding space, the two-tower pattern. One tower maps the user and context to a vector; the other maps SKUs. Co-purchase pairs are the primary positive examples, with in-batch softmax and hard negatives mined from impressions that were shown but didn't convert.

Metrics

Offline Metrics:

Online Metrics:

Guardrails:

The system runs three loops with different cadences, the same pattern that shows up in most large-scale personalization stacks. A slow offline loop retrains the embedding and ranker models daily, weekly for the heavier models. A medium streaming loop keeps content embeddings for new SKUs, popularity counters, inventory state, and ANN index deltas fresh within minutes. A fast online loop assembles the slate per request in under 130ms.

The online loop reads session context first because the retrieval pool mix depends on it. A user who just added a coffee maker to the cart needs complement retrieval (filters, descaler, pods); a user who just viewed a coffee maker needs substitute retrieval (other coffee makers at different price points). A user on the home page with an empty session gets the general personalization mix. The candidate generator dispatches the right mix of retrieval sources, each returning a few hundred candidates, which the pre-ranker trims to the top few hundred for the ranker.

The offline loop is where purchase labels get assembled, joined against features as they existed at request time (point-in-time correct), and fed into training. The two-tower and ranker retrain on different cadences because they have different churn characteristics. The two-tower embeddings drift slowly and a weekly retrain with daily incremental updates is enough. The ranker reacts to short-term catalog shifts, seasonal effects, and promotions, so daily retrain plus hourly calibration is more typical.

The streaming loop is what makes the system feel live. Content embeddings for new SKUs are computed within minutes of listing activation, which keeps cold-start recall from collapsing. Inventory and price updates flow through to an online cache that the inventory filter reads on every request; stale inventory data is the single most common cause of out-of-stock impressions.

Data Flow

Every user event funnels through a single Kafka topic per event type. A feature writer materializes user-level and item-level aggregates into both an offline store (for point-in-time training joins) and an online store (Redis or a key-value database) for serving. A label writer maintains the training tables with attribution applied, and crucially, re-writes purchase labels when returns arrive days later. Point-in-time correctness matters: training must see features as they were at the moment of the impression, not as they are now, or the model learns leakage.

Scale Numbers

Data Sources

Six upstream systems produce the raw data. The clickstream has every product impression, hover, and click, stamped with session id, user id, and surface. The cart service emits add-to-cart and remove-from-cart events. The order service emits checkouts with order id, SKU list, prices, and payment method. The returns service emits refund events with reason codes. The catalog feed carries every SKU's title, brand, category path, image URL, text description, seller id, and attributes; sellers can update any field at any time. The inventory and pricing service is a live API; the rec system reads a cached snapshot, refreshed every 30-60 seconds.

Review data lives in a separate system and flows through on a slower cadence. Search query logs are available but usually owned by the search team; the rec system pulls daily aggregates, not raw queries.

Feature Engineering

Features split into five families. User features describe the shopper's long-term behavior: lifetime GMV, category affinity vectors, price-band affinity (percentiles of prices paid), brand loyalty, return rate, time-since-last-purchase. Item features describe the SKU: price, margin, historical CTR and CVR, average rating, review count, return rate in category, age in catalog, current inventory level, promotion state. Context features describe the request: surface (home vs PDP vs cart), device, time of day, region, referrer, and anchor SKU if the request is on a product detail page. Cross features combine user and item: user-category affinity times item category, user-brand loyalty times item brand, price-band match, repeat-purchase indicator (has this user bought this exact SKU before). Session features describe the current session: last-N viewed SKUs, currently-in-cart SKUs, session duration, clicks-so-far, whether the session started from a search or a direct link.

The session features are the ones that distinguish e-commerce personalization from content personalization. A user's long-term taste matters, but so does the fact that they looked at three Italian espresso machines in the last five minutes. The most recent dozen events in a session carry more predictive weight than the prior three months of history for many decisions.

Feature Table

Baseline Approach

The first version should be item-to-item collaborative filtering, the approach Amazon used when it first published on recommendations and which still works remarkably well on purchase data. For each pair of SKUs, compute how often they appear in the same order (or the same session for a denser signal) and normalize by popularity to avoid the "everything co-occurs with batteries" problem. At serve time, look up the top co-purchase neighbors of the user's recent SKUs and return the union.

This is stronger than it sounds. Purchase co-occurrence is a surprisingly clean signal; if ten thousand users bought a French press and also bought a burr grinder, the French press users tomorrow probably want the burr grinder. A price-bucketed popularity fallback covers new users. The combined system hits a respectable baseline CVR that any deep-learning contender has to beat before shipping.

The baseline has obvious limits. It can't generalize beyond SKUs that have co-occurrence data, so new products cold-start poorly. It can't personalize for users at the individual level, only at the "people who viewed X" level. And it doesn't capture context: the same user on Monday morning and Saturday evening gets the same recs.

Advanced Approach

The production system uses a two-stage architecture, retrieval followed by ranking, with the retrieval stage blending several sources and the ranking stage running a multi-task neural network.

Retrieval has four parallel pools. A two-tower neural retriever covers general personalization: the user tower consumes long-term and session features and outputs a 128-dim embedding; the item tower consumes catalog features and outputs a 128-dim embedding; the dot product approximates the log-odds of a future purchase. The trained item embeddings live in a sharded ANN index (HNSW or ScaNN) and the user vector is computed per request. A co-purchase graph retriever covers the baseline's strength, especially on PDP rails where the anchor SKU is known. A trending-pool retriever covers new and fast-moving SKUs that haven't been seen enough times for the two-tower to place well. A cold-start content retriever uses content embeddings (computed from title, category, and image) to handle the first hour of a SKU's life before any purchase signal arrives.

The ranker is a multi-task network with a shared bottom and four prediction heads.

The shared bottom pattern is replaced by a Mixture-of-Experts gate in mature systems because the four tasks compete for capacity in a shared bottom. With MMoE, each head learns its own weighted combination of experts, which reduces negative transfer between, say, pClick (which rewards flashy thumbnails) and pPurchase (which rewards matches to user intent). The gating is worth the parameter cost on catalogs this large.

The final score is the blend shown earlier, computed in Python at the top of the scoring stage:

The margin floor is the kind of thing that feels hacky in isolation and turns out to be load-bearing in production. Without it, the ranker learns to boost low-margin loss leaders because they drive pClick and pATC; with it, the ranker has to find purchases that are both likely and profitable.

Training

Label construction has three wrinkles that don't exist on a pure engagement system. First, attribution: when a user clicks a recommended SKU, wanders off, comes back the next day via search, and buys, does the original recommendation get credit? Most production systems use last-touch attribution with a 7-day window for the recommendation surface, which is generous enough to capture cross-session purchases but tight enough to exclude coincidental conversions. Second, returns: a label written on day zero has to be flipped or down-weighted if a return arrives on day 14. The training pipeline re-writes labels up to the return-window boundary, which means today's training data is not frozen until 30 days later. In practice, teams train on day-T data where T is at least 7 days ago for a stable purchase label and accept the staleness.

Third, negative sampling. Purchases are rare, and a naive negative sampler pulls random SKUs from the catalog, which produces easy negatives that the model solves quickly and stops learning from. The standard mix is two-thirds in-batch softmax (other users' positives in the same batch) and one-third hard negatives mined from impressions that were viewed and clicked but not purchased. Hard negatives force the model to distinguish genuine purchase intent from passing interest.

Training cadence splits by component. The two-tower retriever retrains weekly; embeddings drift slowly and retraining too often causes instability in the ANN index. The ranker retrains daily with a calibration refresh every few hours during peak retail seasons. Content embeddings for cold-start SKUs are computed on-demand by a batched inference pipeline that reads the catalog feed delta every few minutes.

Inference Pipeline

A request lands at the session classifier, which reads session state and decides which surface is being served and what the retrieval mix should be. A home page request gets a roughly 60% two-tower, 20% trending, 15% co-purchase, 5% cold-start mix. A PDP request, where there's an anchor SKU, flips heavily toward co-purchase (60%) and leans substitutes-heavy. The mix selector dispatches retrieval in parallel across all four pools and unions the results.

The pre-ranker is a small model, often a shallow MLP or a gradient-boosted tree, whose job is to take the 2000-or-so retrieved candidates down to the top 400. Running the full ranker on 2000 candidates would blow the latency budget; running it on 400 fits comfortably. Feature fetch happens after the pre-ranker for exactly this reason: fetching expensive features for 2000 candidates is wasteful if 80% of them get dropped in the next stage.

The inventory filter is a hard gate. It reads the local inventory cache, which is refreshed every 30-60 seconds from the live inventory service, and drops any SKU that's out of stock in the user's shipping region. The safety allow-list is another hard gate: restricted items, region-locked SKUs, age-gated items for underage accounts, and any SKU currently under a seller-quality hold. Only after these two gates does the ranker see the candidates.

Diversity re-ranking is the last stage. A naive ranker will happily return 30 nearly-identical coffee makers because they all have high pPurchase scores. A diversity re-ranker, usually Maximal Marginal Relevance or a DPP-based slate optimizer, trades a small amount of score for category and brand variety. Production systems typically enforce no more than two SKUs from the same seller in the top 10 and no more than three from the same category in the top 30.

Batch vs Real-Time

Latency Budget

The biggest cost is the ranker forward pass. A MMoE with three experts and four heads on a few hundred candidates is enough work that batching and warm GPUs are not optional. Running the ranker on CPUs is possible for smaller catalogs but not at 50K QPS.

6.1 Purchase vs Browse Signals

Labels are where product rec diverges most sharply from content rec. Clicks are abundant but misleading; purchases are sparse but truthful; add-to-cart and review events live in between. The label design question is how to combine them without letting the abundant signals drown out the truthful ones.

The first failure mode is training on clicks and evaluating on purchases. The offline click metrics look great, the model ships, purchase rate is flat, and someone spends two weeks figuring out what went wrong. What went wrong is that the model learned to predict what gets clicked, and what gets clicked is not what gets bought. The shopper clicks because the thumbnail is striking and bounces because the reviews are terrible. Training directly on purchase labels is the right move, but pure purchase labels are too sparse to train a deep ranker from scratch; the gradient dies.

The standard fix is multi-task training with a loss that weighs purchase heavily and uses the denser signals as auxiliaries:

With w_purchase typically 3-5x larger than w_click. The shared bottom learns a representation that's useful for all four heads; the heads themselves specialize. The pReturn head is interesting because returns are rare and noisy, but even a weakly-trained pReturn head that can flag "this category has 40% return rate and this user has a history of returning things that look like this" adds measurable revenue when plugged into the blend.

The label lifecycle above is the other subtlety. A click is logged in seconds. Add-to-cart follows in minutes. Purchase follows in minutes to hours. Returns, when they happen, arrive days or weeks later. The label pipeline has to handle each event type with its own latency budget and re-write purchase labels when returns arrive inside the return window.

6.2 Inventory-Aware Ranking

A recommendation for something the user can't buy is worse than no recommendation. It wastes a slot, trains the user to distrust the surface, and if it happens in the checkout flow it can kill the session. Inventory awareness is part of ranking, not a post-hoc filter.

There are three approaches to integrating inventory signal, and mature systems use all three at different stages. A hard filter drops any SKU with zero stock in the shipping region; this runs at the end of the pipeline and is non-negotiable. A soft penalty down-weights low-stock items in the ranker, because a SKU with two units left will stock out before the slate can convert meaningfully; this runs inside the ranking stage. A demand-shaping adjustment pulls low-inventory items out of trending and popularity pools entirely, so the system doesn't amplify demand for something it can't fulfill; this runs in the retrieval stage.

The substitute boost deserves more attention. When a user arrives on a PDP for a SKU that's low-stock or out-of-stock, the right move is not to hide the page (the user already searched for it) but to surface near-substitutes prominently in the rail. The substitute retriever uses content embeddings to find SKUs in the same category with similar specs and price band, filtered by availability. This recovers revenue that would otherwise be lost.

Production teams also run a conversion-weighted inventory policy: a SKU with 3 units and 90% conversion rate might get pulled from recommendations faster than a SKU with 3 units and 5% conversion rate, because the expected time-to-stockout is much shorter for the former. The math is straightforward; wiring it through to retrieval takes some care.

6.3 Cross-Sell, Upsell, and Session Context

The cart page is the surface where product rec stops looking like content rec and starts looking like merchandising. A user with a coffee maker in their cart should see filters, descaler, pods, and a milk frother, not other coffee makers. A user who just viewed a coffee maker should see other coffee makers at different price points plus a handful of bundles. A user who just completed a purchase of a coffee maker should see replenishment items (pods, beans) in their post-purchase email a month later.

The retrieval mix has to change based on session state, and the signals that drive retrieval change with it.

The co-order graph is the cross-sell workhorse. Co-purchase within a single order is a much cleaner signal than co-purchase across sessions; the former captures "these things are bought together on purpose," the latter also captures "people who like this also like this." For cart-page cross-sell, the co-order graph beats the general co-purchase graph every time. It also captures complement relationships that a two-tower can't easily learn because the two-tower is trained on user-SKU pairs, not SKU-SKU-in-same-order triples.

Upsell is the opposite move: finding a substitute in the same category at a higher price band that improves margin. The rule is simpler than it sounds. Retrieve the top substitutes by content similarity, filter to the same category and user price-band affinity plus one tier, and boost items with higher margin and higher review score. Upsell works on users who show price-band elasticity (they've bought across multiple price bands before) and fails on users who consistently buy at the same price band; a user who always buys the cheapest option in a category should not get aggressive upsell, or they bounce.

Post-purchase replenishment deserves a note. A user who buys coffee beans every three weeks should get a replenishment nudge at the three-week mark, not the one-week mark. The model that drives this is a separate head, often a survival model over inter-purchase times per category, that feeds into email and push notification queues rather than on-site rails. It's usually owned by a different team and integrated loosely, through shared features in the feature store.

6.4 Cold Start for Products and Users

Cold start is continuous, not an edge case. A million new SKUs arrive every day. A substantial fraction of daily active users are infrequent shoppers with little history. The system has to serve sensible recommendations for both from day zero.

For a new SKU, the system has title, category, brand, image, and text description but no purchase history. A content tower computes an embedding from those fields and writes it to a dedicated cold-start ANN index within minutes of listing activation. The cold-start retriever returns candidates from this index, which places the new SKU near existing SKUs that share its content profile. The SKU inherits, in effect, the neighborhood of its content-similar products. Once the new SKU accumulates a meaningful number of purchase events (roughly 100 is the usual threshold), it graduates to the main two-tower index, which has learned embeddings informed by actual behavior.

The content tower matters a lot more for products than for videos, because product metadata is richer. A product has a canonical title, a category tree, a brand, a set of structured attributes (size, color, material), and usually multiple images. A video has a title and a thumbnail and not much else. That richness means the cold-start content tower for products can be surprisingly good, which in turn means the system doesn't collapse to pure popularity during the first hour of a new SKU's life.

Cold-start for users is the other half. A brand-new user has no purchase history, no category affinity, no price-band profile. The first three clicks are worth more than the next three hundred because they're all the model has. The system bootstraps by treating the first session as exploration, leaning heavily on trending plus category-popularity-by-geo, and updating the user's embedding aggressively within the session based on the SKUs they interact with. A lightweight online learner (often a factorization machine or a session-level nearest-neighbor lookup) sits between raw clicks and the full two-tower, updating a user vector every few seconds as new events arrive.

The fifth row captures a subtle case. A user who returns after six months of inactivity is not quite a new user and not quite a returning user. Their historical embedding is stale, their category affinities may have shifted, their price band may have changed. Some systems decay the embedding weights by recency and blend with a cold-start signal for the first few sessions back; this is where returning-user conversion rates live or die.

Offline Evaluation

Offline evaluation on rec systems is structurally hard because the logged data was generated by a previous policy, which biases what got seen in the first place. Naive evaluation of a new ranker on old impressions suffers from position bias, selection bias, and feedback-loop bias. The practical approach uses three tools in combination.

Replay-based evaluation runs the new ranker against the logged impressions and checks whether the model would have ranked actually-purchased items higher than actually-non-purchased items. Inverse-propensity-weighted variants correct for position bias by weighting each logged impression by the inverse of the probability that the old policy would have shown it at that slot. This gives an unbiased estimator of the new ranker's expected behavior, but with high variance when propensities are small. Counterfactual replay with doubly-robust estimators combines IPS with a model-based reward prediction and usually gives tighter bounds.

Offline metrics worth reporting per release: Recall@1000 on held-out purchase days (retrieval health), revenue-weighted nDCG (ranking health), calibration error per head (blend math health), and catalog coverage (fraction of catalog surfaced to at least one user per day, which is an exploration metric).

Online Evaluation

A/B testing on e-commerce has one twist that isn't obvious. Users don't make purchase decisions in isolated events; they make them over sessions and days. Assigning a user to a variant mid-session, right after they added something to the cart, can confuse the experiment: the cart was influenced by the old model, the completion was influenced by the new one. The safer assignment is session-level (a user is assigned to a variant at session start and stays there for the session) or day-level for slower-moving experiments on the home page.

The primary metric is net revenue per session. The secondary metrics are CVR, basket size, and 30-day repeat purchase rate. The guardrails are return rate, average order margin, out-of-stock impression rate, and p99 latency. Any regression on a guardrail disqualifies the variant regardless of how much primary metric lift it shows.

Ramping from 10% to 100% gradually matters because e-commerce has large intra-week and intra-month seasonality. A model that wins in the first week of a ramp can lose in the second week when a promotion launches or the category mix shifts. Running a full ramp across a month of varied traffic is how production teams catch this.

Monitoring

The production monitoring stack watches four kinds of drift. Data drift is straightforward: feature distributions that differ between training and serving indicate a pipeline bug or a behavior shift. Catalog drift is specific to product rec: new categories appearing (the platform launches pet supplies), existing categories shifting in volume (holiday season spikes electronics), or price bands shifting (inflation moves a whole price distribution). Conversion drift is the worry-inducing one: if pClick-to-pPurchase correlation starts dropping, the ranker is learning to drive clicks that don't convert, and that usually means the training data got polluted.

Coverage drift is the final one. If the system starts surfacing the same thousand SKUs to everyone, the long-tail of the catalog is dying. This is an exploration failure and it shows up in seller satisfaction before it shows up in revenue. A coverage dashboard tracking "what fraction of active SKUs appear in at least one top-30 slate per day" is the usual canary; when it drops below 15% or so, someone needs to look at the retrieval mix.

The next chapter moves from commerce to social recommendation, designing the "People You May Know" surface that suggests connections on a social graph. The shift is from purchase intent to relationship prediction, and the graph structure becomes central in a way that it isn't here.