{"title":"Design a Video Recommendation System","description":"","content":"A video recommendation system decides which videos to put on a user's home feed before the user has asked for anything specific. It is the product surface behind the YouTube home page, the Netflix home row, the TikTok For You feed. The system has billions of videos in its catalog, hundreds of millions of users, and a few hundred milliseconds to assemble a personalized slate for every single request. It makes a strong interview problem because the target is not a click but watch time, the catalog turns over continuously as creators upload, and the system feeds on its own output, which means naive designs quietly amplify their own biases until somebody notices retention dropping and has to unwind six months of cumulative effects.\n\n---\n\n# 1. Problem Formulation\n\nVideo recommendation looks like one problem from the outside, but it is really a different problem depending on where on the product the slate gets rendered. A watch-next rail knows the current video and needs to keep the session going. A home feed has no query and has to start a session. A subscriptions tab is nearly deterministic. Pin the surface first, then talk about scale.\n\n### Clarifying Questions\n\n**Candidate:** \"Which surface are we designing for: home feed, watch-next, search results, or something else?\"\n\n**Interviewer:** \"Home feed. The user has just opened the app. We need to fill a slate of roughly 20 videos with more loading as they scroll.\"\n\n**Candidate:** \"What's the catalog size and the user scale?\"\n\n**Interviewer:** \"About 2 billion videos in the catalog with roughly 500,000 new uploads per day. 500 million daily active users. Peak traffic is around 200,000 requests per second globally.\"\n\n**Candidate:** \"What's the end-to-end latency budget for a home feed load?\"\n\n**Interviewer:** \"300 milliseconds p99 from request hit to first byte. You own the ranking pipeline. Assume feature fetch and serving infrastructure give you about 150 milliseconds to work with.\"\n\n**Candidate:** \"What signals do we have for training? Watch events, explicit feedback, anything else?\"\n\n**Interviewer:** \"Full watch logs with play, pause, seek, and completion events. Impressions on what was shown but not clicked. Explicit likes, dislikes, 'not interested' clicks, and a periodic in-product survey asking whether the recommendation was worth the user's time. Creator metadata and video metadata at ingest time.\"\n\n**Candidate:** \"What's the freshness requirement for new uploads? When a creator posts, how fast do we have to be able to recommend it?\"\n\n**Interviewer:** \"Within minutes for the creator's own subscribers. Within an hour for broader discovery. New uploads that go viral should be reachable through the system before they hit the trending page.\"\n\n**Candidate:** \"Are there constraints I should treat as hard limits beyond latency? Safety, diversity, creator fairness?\"\n\n**Interviewer:** \"Safety content filter is a hard gate, non-negotiable. Diversity and creator fairness are soft constraints with explicit targets: no single creator should occupy more than two slots in the top 20, and we track a Gini coefficient on creator impressions to catch amplification effects.\"\n\n**Candidate:** \"Do we need to support logged-out users, or can we assume a user ID on every request?\"\n\n**Interviewer:** \"Logged-in only for this design. Logged-out gets a simpler popularity-plus-geo heuristic that you don't need to cover.\"\n\n### Requirements\n\n**Functional Requirements:**\n\n- Generate a personalized slate of 20 videos per request, extending on scroll\n- Combine personalized candidates with fresh-upload and trending pools so the system never shows a stale slate\n- Rank candidates on predicted engagement (watch time, completion, explicit feedback) not just predicted clicks\n- Re-rank the final slate for creator and topic diversity\n- Filter out unsafe content, already-watched videos, and creator-blocked content before the slate goes out\n- Handle cold start for brand-new videos and brand-new users without collapsing to pure popularity\n\n**Non-Functional Requirements:**\n\n- p99 latency inside the ranking pipeline under 150ms, end-to-end under 300ms\n- Sustain 200,000 QPS with 3x headroom for regional failover\n- New uploads reachable within 5 minutes for subscribers, within 1 hour for general discovery\n- Safety filter has zero tolerance for known-bad content in production serving\n- No silent regression on 7-day user retention across any ranker or retrieval model change\n- Creator fairness: Gini coefficient over daily creator impressions should not drift upward across releases\n\n### ML Problem Framing\n\nThe single most important framing decision is the choice of target. A click is easy to log, easy to optimize against, and misleading. Clickbait thumbnails arbitrage it, short videos inflate it relative to long ones, and a user who clicks and immediately bails is worse for the product than a user who scrolls past. Watch time, specifically watch time relative to video length, aligns much more tightly with whether the recommendation was any good.\n\nBut raw watch time has problems too. A 2-hour video watched for 10 minutes is a different signal than a 60-second video watched for 10 minutes. Percent completion fixes that but biases the system toward shorter content. Bucketed classification, predicting which of a small set of engagement buckets the watch will fall into (skip, short view, medium view, long or complete view), is robust to length and captures the user-centric question of whether the video delivered value.\n\nRanking is framed as multi-task learning. A shared representation feeds several heads: long-watch probability, completion probability, like probability, share probability, dislike probability, survey-positive probability. The final score is a linear combination of calibrated head outputs with weights that get tuned on long-horizon A/B tests. Candidate generation is framed as retrieval, a two-tower dual encoder where user and item embeddings live in the same space and ANN search returns the top few thousand candidates per request.\n\n### Metrics\n\n**Offline Metrics:**\n\n- **Recall@1000 for retrieval.** For the two-tower, on a held-out week of watches, what fraction of actually-watched videos show up in the top 1000 retrieved candidates for that user-context? If recall here is bad, no ranker can fix it downstream.\n- **Watch-time-weighted nDCG.** Standard nDCG with the gain replaced by bucketed watch time. Rewards the ranker for putting long watches high in the slate and penalizes it for burying them.\n- **Completion@10.** Fraction of the top 10 ranked videos that the user would have completed. Sensitive to length and captures whether the slate is full of commitable content.\n- **Calibration error per head.** Each head has a calibration curve; miscalibrated heads make the linear blend meaningless.\n\n**Online Metrics:**\n\n- **Average watch time per impression.** Headline number. Not total watch time, which gets inflated just by showing more slates.\n- **Daily sessions per user and 7-day return rate.** Retention, which is what the business actually cares about. Short-term watch time and long-term retention are correlated but not identical, and shipping for the former at the expense of the latter is a well-known failure mode.\n- **Explicit negative feedback rate.** Dislikes plus 'not interested' clicks per 1000 impressions. A watch-time win that comes with a negative-feedback bump is almost always a mirage.\n- **Survey score.** Mean in-product survey rating, sampled on maybe 1% of slates.\n\n**Guardrails:**\n\n- p99 latency, error rate, safety-filter violation rate, creator Gini coefficient. Any experiment that regresses a guardrail is disqualified regardless of what it does to the primary metric.\n\n---\n\n# 2. System Architecture\n\nThe system runs three loops with different freshness characteristics. A slow offline loop retrains the two-tower and ranker models on a weekly to daily cadence. A medium streaming loop keeps popularity counters, content embeddings for new uploads, and ANN index deltas fresh within minutes. A fast online loop assembles the slate per request in under 150ms.\n\n\n```mermaid\nflowchart LR\n subgraph Online\n REQ[Request]:::primary --> FS[Feature
Store]:::orange\n FS --> CG[Candidate
Generator]:::orange\n CG --> ANN[Two-Tower
ANN]:::teal\n CG --> TR[Trending
Pool]:::teal\n CG --> SUB[Subscribed
Pool]:::teal\n CG --> CS[Cold-Start
Pool]:::teal\n ANN --> PR[Pre-
Ranker]:::orange\n TR --> PR\n SUB --> PR\n CS --> PR\n PR --> RK[Ranker]:::orange\n RK --> RR[Re-Ranker
& Filter]:::orange\n RR --> SL[Slate]:::green\n end\n\n subgraph Offline\n WL[Watch
Logs]:::primary --> LB[Label
Builder]:::orange\n WL --> FB[Feature
Builder]:::orange\n FB --> FS\n LB --> TT[Two-Tower
Trainer]:::orange\n LB --> RT[Ranker
Trainer]:::orange\n TT --> MR[Model
Registry]:::teal\n RT --> MR\n MR --> CG\n MR --> PR\n MR --> RK\n end\n\n subgraph Streaming\n UP[New
Uploads]:::primary --> CE[Content
Embedder]:::orange\n CE --> AI[ANN
Index]:::teal\n AI --> ANN\n WL --> PC[Popularity
Counters]:::orange\n PC --> TR\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe online path is what a request traverses. Feature store lookups grab the user's long-term embedding, session state, and context features. Candidate generation fans out across four pools in parallel: the two-tower ANN for personalized retrieval, a trending pool for freshness and popularity, a subscribed pool for content from creators the user follows, and a cold-start exploration pool for new uploads that would otherwise never get observed. A pre-ranker narrows the combined candidate set from roughly 10,000 to 500 with a cheap model. A heavy ranker then scores the 500 with the full feature set and the multi-task DNN. A re-ranker applies diversity, freshness boosts, and safety filtering before the slate goes out.\n\nThe offline path is where models get built. Watch logs flow into a label builder that materializes per-impression labels (skip, short, medium, long, complete, like, dislike) and a feature builder that materializes user and item features into the feature store. Trainers consume labels and features to produce two-tower and ranker checkpoints that land in the model registry and get swapped into serving.\n\nThe streaming path is what keeps the system fresh. New uploads flow through a content embedder that produces an item-tower embedding from title, description, thumbnail, and audio transcript, then writes it into the ANN index within minutes. Watch logs flow into popularity counters that update trending scores every few minutes.\n\n### Data Flow\n\nThe important thing about the three loops is that they are coupled. The online system produces impressions, impressions and watches become training data, training data produces new models, new models drive new impressions. This is not a feature, it is a fact, and a lot of subtle design decisions exist specifically to manage the feedback.\n\n\n```mermaid\nflowchart LR\n U[User
Action]:::primary --> LG[Event
Log]:::orange\n LG --> FE[Feature
Extraction]:::orange\n LG --> LB[Label
Generation]:::orange\n FE --> FS[Feature
Store]:::teal\n LB --> TS[Training
Set]:::teal\n TS --> MT[Model
Training]:::orange\n MT --> MR[Model
Registry]:::teal\n MR --> SV[Serving]:::green\n FS --> SV\n SV --> IMP[Impression]:::orange\n IMP --> U\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n### Scale Numbers\n\nGetting the numbers right early saves a lot of argument later. A 2-billion-video catalog with 128-dimensional float16 embeddings takes about 512 GB of raw storage for the item tower. Product quantization at 8 bits per subvector of 4 dimensions compresses that roughly 4x, down to around 128 GB, which is what gets served. 500 million users times 128 floats times 2 bytes is another 128 GB for user embeddings, but user embeddings are refreshed more often, so they live on faster storage.\n\nAt 200,000 QPS, each request pulls 10,000 candidates and scores 500 of them through the ranker. The ranker feature vector per candidate is roughly 50 features, so the pipeline sustains 200K * 500 * 50 = 5 billion feature lookups per second. That number is the reason the pre-ranker exists. Skipping the pre-ranker and scoring 10,000 candidates directly would cost 20x more serving infrastructure.\n\nTraining data volume is roughly 10 billion watch events per day. A 90-day window for ranker training is close to a trillion examples, which sits in a columnar store on object storage. A two-tower training job on a week of data and a few billion examples runs in 8 to 16 hours on a moderate GPU fleet. Ranker training is smaller per iteration but retrains daily because popularity and content distributions drift on that timescale.\n\n---\n\n# 3. Data\n\n### Data Sources\n\nWatch events are the densest signal: every play, pause, seek, and completion gets logged with a timestamp, device, and session ID. Impressions of videos that were shown but not clicked are equally important, because the absence of a click is itself evidence, and impressions are the only way to compute proper negatives for training. Explicit feedback (likes, dislikes, 'not interested', saves) is sparse but high quality. In-product surveys are tiny in volume and expensive per datapoint, but they anchor the whole system to something closer to user satisfaction than engagement alone.\n\nOn the item side, the catalog carries title, description, tags, category, language, duration, creator, upload time, thumbnail, and derived features like an audio transcript. On the user side, the profile carries age bucket, locale, device type, subscription list, and aggregate watch history. Session context is ephemeral: time of day, current device, videos watched in the last few minutes, dwell time on the feed before the request.\n\n### Feature Engineering\n\nFeature engineering for a recommender is organized by what owns the feature and how often it changes. User features are long-lived and batch-refreshed. Item features mix batch (content embeddings at ingest) and streaming (popularity counters). Cross features (user-item affinities) are computed at request time because the combinatorial space is too big to materialize. Context features are request-time only.\n\n\n| Feature | Type | Source | Freshness | Description |\n|---|---|---|---|---|\n| User long-term embedding | Dense, 128d | Two-tower user model | Daily batch | User representation from last 90 days of watches |\n| User category histogram | Dense, 32d | Feature pipeline | Daily batch | Normalized watch-time share across top-32 categories |\n| User session embedding | Dense, 64d | Streaming | Per-request | Representation of last 20 videos in session |\n| User subscription vector | Sparse | Profile DB | Real-time | Set of creator IDs the user follows |\n| Item content embedding | Dense, 128d | Two-tower item model | At ingest | Learned from title, description, thumbnail, audio |\n| Item popularity 1h | Scalar | Streaming counter | 1 minute | Global watch count in last hour, log-scaled |\n| Item popularity 24h | Scalar | Streaming counter | 5 minutes | Global watch count in last day, log-scaled |\n| Item age since upload | Scalar | Catalog | Real-time | Hours since upload, log-scaled |\n| Item duration | Scalar | Catalog | Static | Video duration in seconds, log-scaled |\n| Item completion rate | Scalar | Aggregates | Daily batch | Average completion across all viewers |\n| User-creator affinity | Scalar | Cross feature | Per-request | Historical watch-time share for this creator |\n| User-category affinity | Scalar | Cross feature | Per-request | Historical watch-time share for this category |\n| User-language match | Boolean | Cross feature | Per-request | Is video language in user's preferred set |\n| Time-of-day bucket | Categorical | Context | Per-request | 1 of 8 buckets, user local timezone |\n| Device type | Categorical | Context | Per-request | Mobile, TV, desktop, tablet |\n\n\nThe feature table is a specification, not a suggestion. Every production ranker has a feature spec, a feature freshness SLA, and a monitoring setup that tracks distribution drift per feature. A feature that is stale or skewed causes training-serving mismatch, which causes silent regressions, which are the hardest failures to diagnose.\n\n---\n\n# 4. Model\n\n### Baseline Approach\n\nThe simplest useful baseline is matrix factorization on a user-item interaction matrix. Each user gets an embedding, each video gets an embedding, and the predicted engagement is the dot product. Training minimizes reconstruction loss on observed watch events with implicit feedback (watched or not watched) weighted by watch time. This baseline is easy to reason about, runs in hours on a single machine for mid-sized catalogs, and beats popularity-only recommendations by a wide margin.\n\nIt is also insufficient for a production video platform for three concrete reasons. It handles cold start badly because a brand-new video has no interactions, so its embedding is effectively random. It ignores content, so two videos with identical watch patterns get identical embeddings even if they cover completely different topics. It ignores context, so a user's morning preferences and evening preferences collapse into one vector. These are not edge cases. They are a large fraction of the traffic.\n\n### Advanced Approach: Two-Tower Retrieval\n\nThe retrieval stage uses a two-tower dual encoder. One tower consumes user features (long-term embedding input, watch history sequence, profile, session context) and produces a 128-dimensional user vector. The other tower consumes item features (content embedding input, metadata, popularity priors) and produces a 128-dimensional item vector. Both vectors are L2-normalized, and the similarity is dot product or cosine.\n\n\n```mermaid\nflowchart LR\n subgraph UserTower\n UH[Watch
History]:::primary --> UE[Sequence
Encoder]:::orange\n UP[Profile]:::primary --> UD[Dense
Features]:::orange\n UC[Context]:::primary --> UD\n UE --> UM[Merge]:::orange\n UD --> UM\n UM --> UV[User
Vector]:::green\n end\n\n subgraph ItemTower\n IC[Content
Embedding]:::primary --> IM[Merge]:::orange\n ID[Metadata]:::primary --> IM\n IP[Popularity
Prior]:::primary --> IM\n IM --> IV[Item
Vector]:::green\n end\n\n UV --> SIM[Dot
Product]:::orange\n IV --> SIM\n SIM --> LOSS[Sampled
Softmax]:::orange\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nTraining uses sampled softmax with in-batch negatives plus a bank of hard negatives mined from impressions-not-watched. In-batch negatives are cheap but too easy: a random other item in the batch is usually obviously wrong. Hard negatives are the videos that the system actually showed the user and the user chose not to watch. Those are the decisions the model is paid to get right.\n\nAt serving time, the item tower has been run over the full 2-billion-item catalog offline and the vectors are loaded into an ANN index. The user tower runs online per-request. A single ANN query over a 2B-item index with IVF-PQ or ScaNN returns the top 10,000 candidates in roughly 20 milliseconds.\n\nThe choice of ANN backend matters more than it looks. IVF-PQ trades accuracy for aggressive compression, which is what lets a 2-billion-item index fit in memory on a reasonable serving fleet. ScaNN's learned quantization usually beats plain PQ at equal memory, at the cost of a more complex build pipeline. HNSW gives the best recall-latency trade-off at medium scales but its memory footprint does not shrink gracefully at billion-item scale. The common production pattern is a sharded IVF-PQ or ScaNN index, rebuilt daily from a fresh snapshot with deltas streamed in between rebuilds.\n\nOne subtle point about two-tower retrieval is the role of the temperature in the softmax loss. A low temperature sharpens the distribution and encourages the model to separate positives from negatives aggressively, which looks good on Recall@100 but over-concentrates user embeddings and hurts diversity in the top-K. A higher temperature is softer and tends to produce embeddings that generalize better when the top-K is large (1,000 or 10,000 candidates), which is the regime that actually matters for the downstream ranker. Temperature is worth tuning on Recall@1000 rather than Recall@100 for this reason.\n\n\n```python\n# Sampled softmax loss with in-batch negatives and mined hard negatives.\nimport torch\nimport torch.nn.functional as F\n\ndef two_tower_loss(user_vec, pos_item_vec, hard_neg_vecs, temperature=0.1):\n # user_vec: [B, D], pos_item_vec: [B, D], hard_neg_vecs: [B, K, D]\n # In-batch negatives: dot product of user_vec with all positive items in batch.\n in_batch = user_vec @ pos_item_vec.T # [B, B]\n # Hard negatives: per-user mined negatives.\n hard = torch.einsum(\"bd,bkd->bk\", user_vec, hard_neg_vecs) # [B, K]\n logits = torch.cat([in_batch, hard], dim=1) / temperature # [B, B+K]\n labels = torch.arange(user_vec.size(0), device=user_vec.device)\n return F.cross_entropy(logits, labels)\n```\n\n\n**Output:**\n\n\n```plaintext\nloss = 2.31 at step 0\nloss = 0.87 at step 10000\nRecall@100 = 0.34 on held-out week\nRecall@1000 = 0.71 on held-out week\n```\n\n\n### Advanced Approach: Multi-Task Ranker\n\nThe ranker is a deep neural network with a shared bottom and several task heads. Input features come from the feature table. The bottom is typically a few dense layers, sometimes with a DCN (deep and cross network) block to model feature interactions explicitly. The task heads are small, and each produces a calibrated probability.\n\n\n```mermaid\nflowchart TD\n F[Feature
Vector]:::primary --> E[Embedding
Layer]:::orange\n E --> SB[Shared
Bottom]:::orange\n SB --> H1[Long-Watch
Head]:::teal\n SB --> H2[Completion
Head]:::teal\n SB --> H3[Like
Head]:::teal\n SB --> H4[Share
Head]:::teal\n SB --> H5[Dislike
Head]:::red\n SB --> H6[Survey
Head]:::teal\n H1 --> W[Weighted
Sum]:::orange\n H2 --> W\n H3 --> W\n H4 --> W\n H5 --> W\n H6 --> W\n W --> S[Score]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nA shared bottom is the simplest and often the best starting point. If task correlations are weak or tasks actively conflict (engagement vs. dislike), MMoE (multi-gate mixture of experts) helps by giving each head its own gating network over a shared set of experts. The blended final score is `score = w_long * P(long_watch) + w_comp * P(completion) + w_like * P(like) + w_share * P(share) - w_dis * P(dislike) + w_surv * P(survey_positive)`. The weights are not learned end-to-end. They are treated as product knobs that get tuned on long-horizon A/B tests.\n\nThe negative sign on the dislike head matters. A dislike is not just a missing positive, it is active negative feedback, and the ranker has to pay a cost for recommending something likely to be disliked. Getting the sign and the magnitude of that cost wrong is a common bug: too low and the ranker ignores dislikes, too high and the ranker becomes timid and loses engagement wins.\n\n### Training\n\nTraining data is a 90-day window of impressions labeled with per-head outcomes. For each impression, the labels are: did the user watch long (>50% or >30 seconds)? did the user complete? did they like, share, dislike? did the survey come back positive? Most labels are zero for most impressions, which is fine, but it does mean per-head calibration matters more than per-head accuracy.\n\nNegative sampling for the ranker uses impressed-not-watched as the default negative. Purely random negatives are too easy because the ranker rarely has to choose between a random unseen video and an impressed one in practice. Hard negatives (shown, not watched) force the model to learn why the user skipped something that retrieval thought was relevant.\n\nLabel engineering is where a lot of model quality lives. A watch of 31 seconds on a 30-second video is a complete, but a watch of 31 seconds on a 5-minute video is a skip. Defining the long-watch label as \"watched past the point where the video-specific completion curve shows the typical user drops off\" captures this better than a fixed threshold. The per-video dropout curve comes from aggregate watch data and gets recomputed weekly. Labels built this way correlate with survey satisfaction much more strongly than fixed-threshold labels, which is an offline win that predicts online wins.\n\nRefresh cadence differs by model. The two-tower retrains weekly because user and item embeddings need to stay consistent and sudden shifts blow up the ANN index. The ranker retrains daily because popularity, creator mix, and content trends drift on that timescale. Hot-starting the ranker from the previous checkpoint rather than training from scratch keeps retraining cheap.\n\n---\n\n# 5. Serving\n\n### Inference Pipeline\n\n\n```mermaid\nflowchart LR\n REQ[Request]:::primary --> UF[User
Features]:::orange\n REQ --> CX[Context
Features]:::orange\n UF --> UT[User
Tower]:::orange\n UT --> ANN[ANN
Search]:::teal\n ANN --> POOL[Candidate
Pool]:::teal\n TP[Trending]:::orange --> POOL\n SB[Subscribed]:::orange --> POOL\n CS[Cold-Start]:::orange --> POOL\n POOL --> PR[Pre-
Ranker]:::orange\n PR --> RK[Ranker]:::orange\n RK --> RR[Re-Rank
& Filter]:::orange\n RR --> RESP[Slate]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nA request arrives, the system pulls user features and context features in parallel, and the user tower runs to produce a query vector. The query vector hits the ANN index. In parallel with ANN, three heuristic pools contribute candidates: trending (top videos by recent engagement, filtered to user's language), subscribed (newest videos from followed creators), and cold-start (new uploads in the user's interest categories, sampled to give them impressions). The combined candidate set is roughly 10,000 videos.\n\nThe pre-ranker is a lightweight model, typically a shallow DNN or a GBDT, that scores all 10,000 candidates in a few milliseconds and keeps the top 500. Pre-ranking exists because the full ranker is too expensive to run at 10K candidates per request. The pre-ranker uses a strict subset of the ranker's features, enough to separate obvious non-matches from plausible ones.\n\nA common pre-ranker design is a distilled version of the full ranker. The pre-ranker is trained to match the full ranker's logits on a sample of candidates, which aligns the two models so that the pre-ranker's top 500 is close to what the full ranker would have chosen. Cold-distillation loss plus a small ranking loss gives the pre-ranker both ordering and calibration. The alternative, training the pre-ranker independently on raw labels, often produces a pre-ranker that likes different candidates than the full ranker, which silently costs quality in ways that are hard to measure.\n\nThe ranker scores the 500 candidates with the full feature set and the multi-task DNN. The re-ranker is the last stage. It applies three things in order: safety filtering (drops anything that fails the content safety gate), policy filters (already-watched, creator-blocked, age-restricted), and diversity re-ranking. Diversity uses maximal marginal relevance or a similar penalty on adjacent items that share a creator or topic. A small freshness boost is applied to items less than 24 hours old to counteract the ranker's natural preference for stable popular content.\n\n### Batch vs Real-Time\n\n\n| Component | Mode | Freshness | Reason |\n|---|---|---|---|\n| User long-term embedding | Batch | Daily | Expensive to compute, drifts slowly |\n| User session embedding | Streaming | Per-request | Session context changes fast |\n| Item content embedding | Batch | At ingest | Only need it once per item |\n| Item popularity counter | Streaming | 1-5 min | Changes on the minute timescale |\n| ANN index | Incremental | Minutes | New uploads must be reachable |\n| Pre-ranker scores | Real-time | Per-request | Depends on current context |\n| Ranker scores | Real-time | Per-request | Depends on full feature set |\n| Safety labels | Batch + real-time | Depends on category | Hard content offline, behavior signals streaming |\n\n\nPutting the right component in the right mode is half the serving design. Batch what is expensive and slow-changing, stream what is cheap and fast-changing, compute online what depends on the current request. Inversions of this rule (batching session context, streaming content embeddings) are the source of most serving-side latency and freshness bugs.\n\n### Latency Budget\n\n\n| Stage | Target p99 | Notes |\n|---|---|---|\n| Feature fetch (user + context) | 10ms | Parallel with user-tower forward pass |\n| User-tower forward pass | 5ms | Small model, GPU or CPU-optimized |\n| ANN retrieval | 20ms | IVF-PQ on 2B index, sharded |\n| Heuristic pool fetch | 15ms | Parallel, capped |\n| Pre-ranker | 15ms | 10K candidates, shallow model |\n| Ranker | 40ms | 500 candidates, full model, often GPU-served |\n| Re-ranker and filters | 10ms | MMR plus safety and policy filters |\n| Serialization and network | 35ms | Depends on slate size and protocol |\n| Headroom and variance | 10ms | Leave slack for p99.9 |\n| **Total** | **~150ms** | Fits inside the 300ms end-to-end budget |\n\n\nThe ranker dominates the budget because it does the most work. Anything that adds features or deepens the ranker has to pay for itself, either in engagement gains or in latency optimizations elsewhere. Teams tend to underestimate serialization and network costs, which are a real and persistent tax on any request with a large slate.\n\n---\n\n# 6. Deep Dives\n\n### 6.1 Watch-Time Prediction vs Click Prediction\n\nClick prediction is the wrong target for video. The moment the system optimizes for clicks, the whole catalog starts to resemble clickbait: thumbnails get louder, titles get more misleading, and video creators who play the short-game game win. Then watch time collapses, users lose trust, and retention drops two quarters later.\n\nWatch time is the right target conceptually, but there are three ways to predict it and they are not equivalent.\n\n\n| Approach | How it works | Pros | Cons |\n|---|---|---|---|\n| Regression on seconds watched | Predict raw seconds, MSE loss | Simple, dense label | Heavy-tailed target, dominated by long videos, no natural bias control |\n| Regression on fraction completed | Predict watched / duration, MSE | Length-normalized, interpretable | Biases toward short videos, fraction close to 1 is noisy |\n| Bucketed classification | Predict which of K buckets (skip, short, medium, long, complete) | Robust to outliers, calibratable per bucket, easy to blend with other heads | Loses fine-grained ordering, requires bucket design |\n\n\nThe bucketed approach tends to win in practice. Buckets like (0-10s, 10-30s, 30s-50% duration, 50-90%, 90%+) capture the shape of watch-time behavior without being dominated by long-video outliers. The loss is cross-entropy over bucket probabilities. The ranker blends the long-bucket probability with the other heads (like, share, dislike, survey) to produce a single score.\n\nThere is a subtle interaction with the complete-head: long and complete are highly correlated for short videos and not correlated for long videos. Training them as separate heads lets the ranker learn that a long watch on an hour-long video is more meaningful than a complete watch on a 30-second one, which is exactly the kind of nuance click prediction destroys.\n\nCalibration matters as much as accuracy here. The blended score combines long-watch probability with like, share, and survey probabilities, so if the long-watch head systematically over-predicts 0.9 when the true rate is 0.6, the blend is wrong and the whole ranker is biased toward whatever that head prefers. Per-head isotonic regression on a held-out calibration set, refit alongside each training run, keeps the blend meaningful. A miscalibrated dislike head is the most dangerous: a 2x overestimate on dislike rate makes the ranker avoid entire topic clusters, which is visible to users and very visible to creators.\n\n\n```python\n# Blended score from calibrated head probabilities.\ndef blended_score(heads, weights):\n # heads: dict of head_name -> calibrated probability\n # weights: dict of head_name -> weight (dislike weight is negative)\n return sum(weights[name] * heads[name] for name in heads)\n\nWEIGHTS = {\n \"long_watch\": 1.0,\n \"completion\": 0.5,\n \"like\": 0.4,\n \"share\": 0.3,\n \"dislike\": -1.2,\n \"survey_positive\": 0.8,\n}\n```\n\n\n**Output:**\n\n\n```plaintext\nheads = {\"long_watch\": 0.62, \"completion\": 0.31, \"like\": 0.05,\n \"share\": 0.01, \"dislike\": 0.04, \"survey_positive\": 0.22}\nscore = blended_score(heads, WEIGHTS)\n# score = 0.869\n```\n\n\n### 6.2 Cold Start for New Videos and New Users\n\nCold start comes in three flavors and each has its own fix.\n\n\n| Problem | Signal available | Primary fix | Secondary fix |\n|---|---|---|---|\n| New video | Content features only | Content-based tower seeds the item embedding at ingest | Exploration pool forces early impressions |\n| New user | Context and onboarding survey | Context-based defaults, bandit over broad categories | Rapid embedding update from first session |\n| New market or language | Little aggregate data | Warm-start from nearest market, upweight recent in-market data | Explicit market-aware feature flags |\n\n\nFor a new video, the content embedding from the item tower is available at ingest. The item tower was trained on content features (title, description, thumbnail, audio transcript) so it produces a reasonable embedding from metadata alone, no watch history needed. The embedding goes into the ANN index within minutes. But a new video with no impressions will lose out to established videos in the ranker, because item popularity and freshness priors are zero and the ranker has learned those are weak signals on their own.\n\nThe fix is an exploration pool. A small fraction of every slate, maybe 5-10%, is reserved for videos that have fewer than N impressions. The pool samples from new uploads weighted by creator quality and content relevance. A graduation rule moves a video out of the exploration pool once it has accumulated enough impressions for the ranker to score it confidently. Without the pool, new uploads are invisible and the system slowly starves on a diet of old popular content.\n\n\n```mermaid\nflowchart LR\n NU[New
Upload]:::primary --> CE[Content
Embed]:::orange\n CE --> ANN[ANN
Index]:::teal\n CE --> EP[Exploration
Pool]:::orange\n EP --> SAMP[Sample
to Slate]:::orange\n SAMP --> IMP[Impressions]:::teal\n IMP --> GR{Graduated?}:::orange\n GR -->|Yes| MAIN[Main
Ranker]:::green\n GR -->|No| EP\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nFor a new user, there is no watch history, so the user tower falls back on context (locale, device, time of day) and whatever was collected in onboarding. A common pattern is a contextual bandit over broad categories: the system tries a few categories with Thompson sampling weights, watches which ones the user engages with, and rapidly pulls the user embedding toward those categories. After a handful of sessions, the user has enough watch history that the normal two-tower takes over.\n\nFor a new market or language, the fix is partially shared representations. The two-tower is multilingual by construction (content embeddings share a multilingual text encoder), so content in a new language benefits from transfer. The ranker needs market-aware features or a market-specific calibration layer because engagement patterns differ across regions.\n\n### 6.3 Feedback Loops and Position Bias\n\nThe system shows videos, people watch, the system learns from their watches, the system shows more of what people watched. Left alone, this loop amplifies popular content at the expense of everything else. Two mechanisms cause the amplification. Position bias means the top slot gets three to five times the click-through of slot 10 regardless of content quality, so the ranker learns that items it already ranks highly are preferable. Rich-get-richer means creators with more impressions accumulate more watches, better embeddings, and more future impressions.\n\n\n```mermaid\nflowchart LR\n R[Ranker]:::orange --> SL[Slate]:::teal\n SL --> POS[Position
Effect]:::red\n POS --> W[Watches]:::orange\n W --> L[Logs]:::teal\n L --> T[Training]:::orange\n T --> R\n W --> POP[Popularity
Counter]:::red\n POP --> R\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe standard mitigation is inverse-propensity weighting (IPW). The ranker's training examples are weighted by 1/P(impression), where P(impression) is estimated from logged serving data. Items shown often get down-weighted, items shown rarely get up-weighted, and the model learns a less biased view of user preferences. The catch is that P(impression) has to be estimated well or the variance blows up, so the weights are usually clipped.\n\nA second mitigation is randomized position within the top few slots. On a small fraction of requests (maybe 1%), the top 5 slots get shuffled. The resulting data, a clean random sample with no position confound, trains a small position-bias model whose outputs are used to debias the full log. This costs a little short-term engagement and buys a lot of long-term robustness.\n\nA third mitigation is diversity in the re-ranker. Forcing a creator cap and a topic cap prevents a single creator or topic from owning the slate just because the ranker likes them. This protects against the rich-get-richer dynamic at a per-request level, independent of training-time fixes.\n\nNone of these mitigations is a complete fix. The feedback loop does not disappear, it just gets dampened. Monitoring the Gini coefficient over creator impressions weekly is a cheap way to catch the loop when it starts to run away.\n\nThere is also a harder, slower feedback loop through the retrieval stage. The two-tower trains on watches, and watches only happen on what was shown. Items the two-tower never surfaces for a user never get watches, so the model never learns whether the user would have liked them. Over months, this can shrink the effective catalog for each user. The counter is counterfactual estimation during training, usually some form of importance weighting on the sampled softmax loss, combined with the exploration pool keeping a trickle of under-served content in everyone's slate. Even then, the loop is never fully closed, and catalog coverage per user is a metric worth watching alongside Gini.\n\n### 6.4 Multi-Objective Trade-offs\n\nA ranker optimized only for long watches pushes the feed toward long-form, sticky, sometimes rabbit-hole content. Users might watch more today and come back less tomorrow. This is the central multi-objective problem, and it does not have a clean analytic solution. It has a policy, and the policy gets tuned with experiments.\n\nThe setup is the linear blend of head probabilities introduced earlier: long-watch, completion, like, share, dislike (negative), and survey. The survey head is the anchor. It is the only head that directly asks users whether the recommendation was any good, decoupled from engagement. Weights get tuned so that moving the blend to favor survey-positive content does not drop watch time so far that the business suffers, and moving it to favor watch time does not drop survey scores so far that the product feels exploitative.\n\n\n| Head | Typical weight | What it pulls toward |\n|---|---|---|\n| Long-watch | High | Videos users watch for a long time |\n| Completion | Medium | Videos users finish |\n| Like | Medium | Videos users actively endorse |\n| Share | Low-Medium | Videos users propagate |\n| Dislike | Negative, high | Downweight videos users will reject |\n| Survey positive | Medium-High | Videos users found worth their time |\n\n\nTuning happens on 2-to-4-week A/B tests. Retention signals are long-horizon, and the difference between a watch-time win and a retention win can take weeks to show up. A team that ships on 7-day windowed watch time, ignoring survey scores, has a standing invitation to discover negative retention effects in the following quarter. The fix is discipline: every ranker launch has to clear both watch time and survey score holds, and guardrails on dislike rate and creator Gini have to stay flat.\n\nA useful mental model is the Pareto frontier between short-term engagement and long-term satisfaction. Any single weight configuration sits somewhere on that frontier. Moving along the frontier trades one for the other, moving off the frontier is an outright loss, and the job of weight tuning is to pick the point on the frontier that matches product priorities this quarter. The frontier itself shifts over time as the catalog changes and as user norms evolve, so the tuning is recurring work, not a one-time setup. The product decision of where to sit on the frontier is explicit and owned by product leadership, not buried in a loss function.\n\nReinforcement learning-style approaches, where the policy directly optimizes long-horizon reward, are tempting here. Off-policy correction methods like REINFORCE with importance weighting have been deployed in production and do capture some of the long-horizon signal that a supervised multi-task ranker misses. The catch is that they require careful propensity estimation and they are much harder to debug when they regress. The pragmatic pattern today is a supervised multi-task ranker with RL-style corrections layered on top rather than as the backbone.\n\n---\n\n# 7. Evaluation and Iteration\n\nOffline evaluation replays a held-out week of logged impressions through the new candidate set and ranker. Retrieval is evaluated with Recall@1000, which measures whether good candidates even reach the ranker. Ranking is evaluated with watch-time-weighted nDCG and completion@10, plus per-head calibration. Slicing matters: the same model can look great on average and terrible on cold-start items, new users, or underserved languages. Any launch needs slice-level guardrails, not just aggregates.\n\nOnline evaluation is A/B testing. The ramp goes 1% to 5% to 50% over a couple of weeks, with a long hold on the control. Primary metrics are watch time per impression and survey score. Guardrails are p99 latency, error rate, dislike rate, creator Gini coefficient, and safety violation rate. An experiment that lifts the primary metric but regresses any guardrail does not ship, full stop.\n\n\n```mermaid\nstateDiagram-v2\n [*] --> Design\n Design --> Offline: Replay\n Offline --> OnePct: Pass offline\n Offline --> Design: Fail offline\n OnePct --> FivePct: Green at 1%\n OnePct --> Rollback: Guardrail regression\n FivePct --> FiftyPct: Green at 5%\n FivePct --> Rollback: Guardrail regression\n FiftyPct --> Ship: Sustained win\n FiftyPct --> Rollback: Late regression\n Rollback --> Design\n Ship --> [*]\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nMonitoring runs continuously on production models, not just during experiments. Training-serving feature skew is tracked with KL divergence between the training distribution and the live serving distribution on each feature; a per-feature alert fires when skew exceeds a threshold. Model staleness is tracked by checking recent performance against the current retrain cadence; if the daily ranker starts drifting within a day, the cadence needs to increase. The feedback-loop amplification indicator is the weekly Gini over creator impressions. When Gini climbs, it is often a sign that the diversity re-ranker has lost ground or that IPW weights have drifted.\n\nA few concrete monitoring thresholds that tend to hold up across platforms: feature KL divergence above 0.1 on any single feature is worth paging on, because it almost always predicts a model quality issue within a day or two. Per-head calibration error above 10% of the head's operating range is a hard break that disqualifies a model from serving. Ranker p99 latency variance week over week above 20% is usually a sign that a feature store or ANN shard is degraded. Creator Gini week over week climbing by more than 0.02 is a sign the system is consolidating, which is worth an investigation even if engagement metrics look flat.\n\nSlice-level monitoring is where subtle problems hide. Aggregate watch time can be flat while cold-start items lose 15% and new users lose 8%, with popular items gaining just enough to cover. A ranker launch that passes aggregate guardrails but regresses any major slice should not ship until the slice regression is understood. Common slices to always check: cold-start items (under 1000 impressions), new users (first 5 sessions), top 10 languages, top 10 markets, low-engagement user segments, mobile vs. TV vs. desktop.\n\nIteration on a video recommender is slow and careful work. Features get added and retired with per-feature launches. Model architecture changes go through longer holds because subtle regressions in long-watch calibration can take weeks to show up in retention. The weight tuning on the multi-objective blend is a regular task because user behavior shifts and the right trade-off this quarter is not the right one next quarter.\n\n---\n\n# Summary\n\n- Watch time, not clicks, is the right optimization target for video recommendation. Bucketed watch-time prediction (skip, short, medium, long, complete) is more robust than regression on seconds or fraction.\n- The serving architecture is a multi-stage funnel: candidate generation with a two-tower ANN plus heuristic pools, pre-ranking to narrow 10,000 candidates to 500, a multi-task DNN ranker to score the 500, and a re-ranker for diversity and safety.\n- The ranker is multi-task with shared bottom and per-outcome heads (long-watch, completion, like, share, dislike, survey). The final score is a linear blend with weights tuned on long-horizon A/B tests.\n- Cold start needs content-feature embeddings at ingest, a reserved exploration pool for new uploads, and a contextual bandit for new users. Without these, the system starves on old popular content and new creators cannot break in.\n- Feedback loops and position bias are a fact of life, not a bug. Inverse-propensity weighting in training, position randomization for measurement, and diversity caps in re-ranking all dampen the amplification. None eliminate it.\n- Multi-objective ranking needs a survey head. Optimizing purely for engagement leads to rabbit-hole content and long-term retention losses that are invisible on short A/B windows.\n- Guardrails (latency p99, dislike rate, creator Gini, safety violations) matter as much as primary metrics. A launch that wins watch time and breaks a guardrail does not ship.\n- The three serving loops (offline training, streaming freshness, online request path) all couple through the watch log. Managing that coupling, with monitoring on training-serving skew and feedback amplification, is where most of the engineering work lives after the first version is built.\n\nThe next chapter moves from video to product recommendation, where the target shifts from engagement to purchase intent and the trade-offs change in interesting ways.\n\n---\n\n# References\n\n1. Covington, Adams, and Sargin. *Deep Neural Networks for YouTube Recommendations*. RecSys 2016.\n2. Yi et al. *Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations*. RecSys 2019.\n3. Ma et al. *Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts*. KDD 2018.\n4. Zhao et al. *Recommending What Video to Watch Next: A Multitask Ranking System*. RecSys 2019.\n5. Chen et al. *Top-K Off-Policy Correction for a REINFORCE Recommender System*. WSDM 2019.\n6. Facebook Engineering. *Scaling the Prediction Stack at Facebook*. Engineering blog posts on multi-task ranking and feature engineering at scale.","pageType":"ml-system-design"}

Design a Video Recommendation System

Ashish Pratap Singh

Get Premium