A video recommendation system decides which videos to put on a user's home feed before the user has asked for anything specific. It is the product surface behind the YouTube home page, the Netflix home row, the TikTok For You feed. The system has billions of videos in its catalog, hundreds of millions of users, and a few hundred milliseconds to assemble a personalized slate for every single request. It makes a strong interview problem because the target is not a click but watch time, the catalog turns over continuously as creators upload, and the system feeds on its own output, which means naive designs quietly amplify their own biases until somebody notices retention dropping and has to unwind six months of cumulative effects.

Video recommendation looks like one problem from the outside, but it is really a different problem depending on where on the product the slate gets rendered. A watch-next rail knows the current video and needs to keep the session going. A home feed has no query and has to start a session. A subscriptions tab is nearly deterministic. Pin the surface first, then talk about scale.

Clarifying Questions

Candidate: "Which surface are we designing for: home feed, watch-next, search results, or something else?"

Interviewer: "Home feed. The user has just opened the app. We need to fill a slate of roughly 20 videos with more loading as they scroll."

Candidate: "What's the catalog size and the user scale?"

Interviewer: "About 2 billion videos in the catalog with roughly 500,000 new uploads per day. 500 million daily active users. Peak traffic is around 200,000 requests per second globally."

Candidate: "What's the end-to-end latency budget for a home feed load?"

Interviewer: "300 milliseconds p99 from request hit to first byte. You own the ranking pipeline. Assume feature fetch and serving infrastructure give you about 150 milliseconds to work with."

Candidate: "What signals do we have for training? Watch events, explicit feedback, anything else?"

Interviewer: "Full watch logs with play, pause, seek, and completion events. Impressions on what was shown but not clicked. Explicit likes, dislikes, 'not interested' clicks, and a periodic in-product survey asking whether the recommendation was worth the user's time. Creator metadata and video metadata at ingest time."

Candidate: "What's the freshness requirement for new uploads? When a creator posts, how fast do we have to be able to recommend it?"

Interviewer: "Within minutes for the creator's own subscribers. Within an hour for broader discovery. New uploads that go viral should be reachable through the system before they hit the trending page."

Candidate: "Are there constraints I should treat as hard limits beyond latency? Safety, diversity, creator fairness?"

Interviewer: "Safety content filter is a hard gate, non-negotiable. Diversity and creator fairness are soft constraints with explicit targets: no single creator should occupy more than two slots in the top 20, and we track a Gini coefficient on creator impressions to catch amplification effects."

Candidate: "Do we need to support logged-out users, or can we assume a user ID on every request?"

Interviewer: "Logged-in only for this design. Logged-out gets a simpler popularity-plus-geo heuristic that you don't need to cover."

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

The single most important framing decision is the choice of target. A click is easy to log, easy to optimize against, and misleading. Clickbait thumbnails arbitrage it, short videos inflate it relative to long ones, and a user who clicks and immediately bails is worse for the product than a user who scrolls past. Watch time, specifically watch time relative to video length, aligns much more tightly with whether the recommendation was any good.

But raw watch time has problems too. A 2-hour video watched for 10 minutes is a different signal than a 60-second video watched for 10 minutes. Percent completion fixes that but biases the system toward shorter content. Bucketed classification, predicting which of a small set of engagement buckets the watch will fall into (skip, short view, medium view, long or complete view), is robust to length and captures the user-centric question of whether the video delivered value.

Ranking is framed as multi-task learning. A shared representation feeds several heads: long-watch probability, completion probability, like probability, share probability, dislike probability, survey-positive probability. The final score is a linear combination of calibrated head outputs with weights that get tuned on long-horizon A/B tests. Candidate generation is framed as retrieval, a two-tower dual encoder where user and item embeddings live in the same space and ANN search returns the top few thousand candidates per request.

Metrics

Offline Metrics:

Online Metrics:

Guardrails:

The system runs three loops with different freshness characteristics. A slow offline loop retrains the two-tower and ranker models on a weekly to daily cadence. A medium streaming loop keeps popularity counters, content embeddings for new uploads, and ANN index deltas fresh within minutes. A fast online loop assembles the slate per request in under 150ms.

The online path is what a request traverses. Feature store lookups grab the user's long-term embedding, session state, and context features. Candidate generation fans out across four pools in parallel: the two-tower ANN for personalized retrieval, a trending pool for freshness and popularity, a subscribed pool for content from creators the user follows, and a cold-start exploration pool for new uploads that would otherwise never get observed. A pre-ranker narrows the combined candidate set from roughly 10,000 to 500 with a cheap model. A heavy ranker then scores the 500 with the full feature set and the multi-task DNN. A re-ranker applies diversity, freshness boosts, and safety filtering before the slate goes out.

The offline path is where models get built. Watch logs flow into a label builder that materializes per-impression labels (skip, short, medium, long, complete, like, dislike) and a feature builder that materializes user and item features into the feature store. Trainers consume labels and features to produce two-tower and ranker checkpoints that land in the model registry and get swapped into serving.

The streaming path is what keeps the system fresh. New uploads flow through a content embedder that produces an item-tower embedding from title, description, thumbnail, and audio transcript, then writes it into the ANN index within minutes. Watch logs flow into popularity counters that update trending scores every few minutes.

Data Flow

The important thing about the three loops is that they are coupled. The online system produces impressions, impressions and watches become training data, training data produces new models, new models drive new impressions. This is not a feature, it is a fact, and a lot of subtle design decisions exist specifically to manage the feedback.

Scale Numbers

Getting the numbers right early saves a lot of argument later. A 2-billion-video catalog with 128-dimensional float16 embeddings takes about 512 GB of raw storage for the item tower. Product quantization at 8 bits per subvector of 4 dimensions compresses that roughly 4x, down to around 128 GB, which is what gets served. 500 million users times 128 floats times 2 bytes is another 128 GB for user embeddings, but user embeddings are refreshed more often, so they live on faster storage.

At 200,000 QPS, each request pulls 10,000 candidates and scores 500 of them through the ranker. The ranker feature vector per candidate is roughly 50 features, so the pipeline sustains 200K 500 50 = 5 billion feature lookups per second. That number is the reason the pre-ranker exists. Skipping the pre-ranker and scoring 10,000 candidates directly would cost 20x more serving infrastructure.

Training data volume is roughly 10 billion watch events per day. A 90-day window for ranker training is close to a trillion examples, which sits in a columnar store on object storage. A two-tower training job on a week of data and a few billion examples runs in 8 to 16 hours on a moderate GPU fleet. Ranker training is smaller per iteration but retrains daily because popularity and content distributions drift on that timescale.

Data Sources

Watch events are the densest signal: every play, pause, seek, and completion gets logged with a timestamp, device, and session ID. Impressions of videos that were shown but not clicked are equally important, because the absence of a click is itself evidence, and impressions are the only way to compute proper negatives for training. Explicit feedback (likes, dislikes, 'not interested', saves) is sparse but high quality. In-product surveys are tiny in volume and expensive per datapoint, but they anchor the whole system to something closer to user satisfaction than engagement alone.

On the item side, the catalog carries title, description, tags, category, language, duration, creator, upload time, thumbnail, and derived features like an audio transcript. On the user side, the profile carries age bucket, locale, device type, subscription list, and aggregate watch history. Session context is ephemeral: time of day, current device, videos watched in the last few minutes, dwell time on the feed before the request.

Feature Engineering

Feature engineering for a recommender is organized by what owns the feature and how often it changes. User features are long-lived and batch-refreshed. Item features mix batch (content embeddings at ingest) and streaming (popularity counters). Cross features (user-item affinities) are computed at request time because the combinatorial space is too big to materialize. Context features are request-time only.

The feature table is a specification, not a suggestion. Every production ranker has a feature spec, a feature freshness SLA, and a monitoring setup that tracks distribution drift per feature. A feature that is stale or skewed causes training-serving mismatch, which causes silent regressions, which are the hardest failures to diagnose.

Baseline Approach

The simplest useful baseline is matrix factorization on a user-item interaction matrix. Each user gets an embedding, each video gets an embedding, and the predicted engagement is the dot product. Training minimizes reconstruction loss on observed watch events with implicit feedback (watched or not watched) weighted by watch time. This baseline is easy to reason about, runs in hours on a single machine for mid-sized catalogs, and beats popularity-only recommendations by a wide margin.

It is also insufficient for a production video platform for three concrete reasons. It handles cold start badly because a brand-new video has no interactions, so its embedding is effectively random. It ignores content, so two videos with identical watch patterns get identical embeddings even if they cover completely different topics. It ignores context, so a user's morning preferences and evening preferences collapse into one vector. These are not edge cases. They are a large fraction of the traffic.

Advanced Approach: Two-Tower Retrieval

The retrieval stage uses a two-tower dual encoder. One tower consumes user features (long-term embedding input, watch history sequence, profile, session context) and produces a 128-dimensional user vector. The other tower consumes item features (content embedding input, metadata, popularity priors) and produces a 128-dimensional item vector. Both vectors are L2-normalized, and the similarity is dot product or cosine.

Training uses sampled softmax with in-batch negatives plus a bank of hard negatives mined from impressions-not-watched. In-batch negatives are cheap but too easy: a random other item in the batch is usually obviously wrong. Hard negatives are the videos that the system actually showed the user and the user chose not to watch. Those are the decisions the model is paid to get right.

At serving time, the item tower has been run over the full 2-billion-item catalog offline and the vectors are loaded into an ANN index. The user tower runs online per-request. A single ANN query over a 2B-item index with IVF-PQ or ScaNN returns the top 10,000 candidates in roughly 20 milliseconds.

The choice of ANN backend matters more than it looks. IVF-PQ trades accuracy for aggressive compression, which is what lets a 2-billion-item index fit in memory on a reasonable serving fleet. ScaNN's learned quantization usually beats plain PQ at equal memory, at the cost of a more complex build pipeline. HNSW gives the best recall-latency trade-off at medium scales but its memory footprint does not shrink gracefully at billion-item scale. The common production pattern is a sharded IVF-PQ or ScaNN index, rebuilt daily from a fresh snapshot with deltas streamed in between rebuilds.

One subtle point about two-tower retrieval is the role of the temperature in the softmax loss. A low temperature sharpens the distribution and encourages the model to separate positives from negatives aggressively, which looks good on Recall@100 but over-concentrates user embeddings and hurts diversity in the top-K. A higher temperature is softer and tends to produce embeddings that generalize better when the top-K is large (1,000 or 10,000 candidates), which is the regime that actually matters for the downstream ranker. Temperature is worth tuning on Recall@1000 rather than Recall@100 for this reason.

Output:

Advanced Approach: Multi-Task Ranker

The ranker is a deep neural network with a shared bottom and several task heads. Input features come from the feature table. The bottom is typically a few dense layers, sometimes with a DCN (deep and cross network) block to model feature interactions explicitly. The task heads are small, and each produces a calibrated probability.

A shared bottom is the simplest and often the best starting point. If task correlations are weak or tasks actively conflict (engagement vs. dislike), MMoE (multi-gate mixture of experts) helps by giving each head its own gating network over a shared set of experts. The blended final score is score = w_long * P(long_watch) + w_comp * P(completion) + w_like * P(like) + w_share * P(share) - w_dis * P(dislike) + w_surv * P(survey_positive). The weights are not learned end-to-end. They are treated as product knobs that get tuned on long-horizon A/B tests.

The negative sign on the dislike head matters. A dislike is not just a missing positive, it is active negative feedback, and the ranker has to pay a cost for recommending something likely to be disliked. Getting the sign and the magnitude of that cost wrong is a common bug: too low and the ranker ignores dislikes, too high and the ranker becomes timid and loses engagement wins.

Training

Training data is a 90-day window of impressions labeled with per-head outcomes. For each impression, the labels are: did the user watch long (>50% or >30 seconds)? did the user complete? did they like, share, dislike? did the survey come back positive? Most labels are zero for most impressions, which is fine, but it does mean per-head calibration matters more than per-head accuracy.

Negative sampling for the ranker uses impressed-not-watched as the default negative. Purely random negatives are too easy because the ranker rarely has to choose between a random unseen video and an impressed one in practice. Hard negatives (shown, not watched) force the model to learn why the user skipped something that retrieval thought was relevant.

Label engineering is where a lot of model quality lives. A watch of 31 seconds on a 30-second video is a complete, but a watch of 31 seconds on a 5-minute video is a skip. Defining the long-watch label as "watched past the point where the video-specific completion curve shows the typical user drops off" captures this better than a fixed threshold. The per-video dropout curve comes from aggregate watch data and gets recomputed weekly. Labels built this way correlate with survey satisfaction much more strongly than fixed-threshold labels, which is an offline win that predicts online wins.

Refresh cadence differs by model. The two-tower retrains weekly because user and item embeddings need to stay consistent and sudden shifts blow up the ANN index. The ranker retrains daily because popularity, creator mix, and content trends drift on that timescale. Hot-starting the ranker from the previous checkpoint rather than training from scratch keeps retraining cheap.

Inference Pipeline

A request arrives, the system pulls user features and context features in parallel, and the user tower runs to produce a query vector. The query vector hits the ANN index. In parallel with ANN, three heuristic pools contribute candidates: trending (top videos by recent engagement, filtered to user's language), subscribed (newest videos from followed creators), and cold-start (new uploads in the user's interest categories, sampled to give them impressions). The combined candidate set is roughly 10,000 videos.

The pre-ranker is a lightweight model, typically a shallow DNN or a GBDT, that scores all 10,000 candidates in a few milliseconds and keeps the top 500. Pre-ranking exists because the full ranker is too expensive to run at 10K candidates per request. The pre-ranker uses a strict subset of the ranker's features, enough to separate obvious non-matches from plausible ones.

A common pre-ranker design is a distilled version of the full ranker. The pre-ranker is trained to match the full ranker's logits on a sample of candidates, which aligns the two models so that the pre-ranker's top 500 is close to what the full ranker would have chosen. Cold-distillation loss plus a small ranking loss gives the pre-ranker both ordering and calibration. The alternative, training the pre-ranker independently on raw labels, often produces a pre-ranker that likes different candidates than the full ranker, which silently costs quality in ways that are hard to measure.

The ranker scores the 500 candidates with the full feature set and the multi-task DNN. The re-ranker is the last stage. It applies three things in order: safety filtering (drops anything that fails the content safety gate), policy filters (already-watched, creator-blocked, age-restricted), and diversity re-ranking. Diversity uses maximal marginal relevance or a similar penalty on adjacent items that share a creator or topic. A small freshness boost is applied to items less than 24 hours old to counteract the ranker's natural preference for stable popular content.

Batch vs Real-Time

Putting the right component in the right mode is half the serving design. Batch what is expensive and slow-changing, stream what is cheap and fast-changing, compute online what depends on the current request. Inversions of this rule (batching session context, streaming content embeddings) are the source of most serving-side latency and freshness bugs.

Latency Budget

The ranker dominates the budget because it does the most work. Anything that adds features or deepens the ranker has to pay for itself, either in engagement gains or in latency optimizations elsewhere. Teams tend to underestimate serialization and network costs, which are a real and persistent tax on any request with a large slate.

6.1 Watch-Time Prediction vs Click Prediction

Click prediction is the wrong target for video. The moment the system optimizes for clicks, the whole catalog starts to resemble clickbait: thumbnails get louder, titles get more misleading, and video creators who play the short-game game win. Then watch time collapses, users lose trust, and retention drops two quarters later.

Watch time is the right target conceptually, but there are three ways to predict it and they are not equivalent.

The bucketed approach tends to win in practice. Buckets like (0-10s, 10-30s, 30s-50% duration, 50-90%, 90%+) capture the shape of watch-time behavior without being dominated by long-video outliers. The loss is cross-entropy over bucket probabilities. The ranker blends the long-bucket probability with the other heads (like, share, dislike, survey) to produce a single score.

There is a subtle interaction with the complete-head: long and complete are highly correlated for short videos and not correlated for long videos. Training them as separate heads lets the ranker learn that a long watch on an hour-long video is more meaningful than a complete watch on a 30-second one, which is exactly the kind of nuance click prediction destroys.

Calibration matters as much as accuracy here. The blended score combines long-watch probability with like, share, and survey probabilities, so if the long-watch head systematically over-predicts 0.9 when the true rate is 0.6, the blend is wrong and the whole ranker is biased toward whatever that head prefers. Per-head isotonic regression on a held-out calibration set, refit alongside each training run, keeps the blend meaningful. A miscalibrated dislike head is the most dangerous: a 2x overestimate on dislike rate makes the ranker avoid entire topic clusters, which is visible to users and very visible to creators.

Output:

6.2 Cold Start for New Videos and New Users

Cold start comes in three flavors and each has its own fix.

For a new video, the content embedding from the item tower is available at ingest. The item tower was trained on content features (title, description, thumbnail, audio transcript) so it produces a reasonable embedding from metadata alone, no watch history needed. The embedding goes into the ANN index within minutes. But a new video with no impressions will lose out to established videos in the ranker, because item popularity and freshness priors are zero and the ranker has learned those are weak signals on their own.

The fix is an exploration pool. A small fraction of every slate, maybe 5-10%, is reserved for videos that have fewer than N impressions. The pool samples from new uploads weighted by creator quality and content relevance. A graduation rule moves a video out of the exploration pool once it has accumulated enough impressions for the ranker to score it confidently. Without the pool, new uploads are invisible and the system slowly starves on a diet of old popular content.

For a new user, there is no watch history, so the user tower falls back on context (locale, device, time of day) and whatever was collected in onboarding. A common pattern is a contextual bandit over broad categories: the system tries a few categories with Thompson sampling weights, watches which ones the user engages with, and rapidly pulls the user embedding toward those categories. After a handful of sessions, the user has enough watch history that the normal two-tower takes over.

For a new market or language, the fix is partially shared representations. The two-tower is multilingual by construction (content embeddings share a multilingual text encoder), so content in a new language benefits from transfer. The ranker needs market-aware features or a market-specific calibration layer because engagement patterns differ across regions.

6.3 Feedback Loops and Position Bias

The system shows videos, people watch, the system learns from their watches, the system shows more of what people watched. Left alone, this loop amplifies popular content at the expense of everything else. Two mechanisms cause the amplification. Position bias means the top slot gets three to five times the click-through of slot 10 regardless of content quality, so the ranker learns that items it already ranks highly are preferable. Rich-get-richer means creators with more impressions accumulate more watches, better embeddings, and more future impressions.

The standard mitigation is inverse-propensity weighting (IPW). The ranker's training examples are weighted by 1/P(impression), where P(impression) is estimated from logged serving data. Items shown often get down-weighted, items shown rarely get up-weighted, and the model learns a less biased view of user preferences. The catch is that P(impression) has to be estimated well or the variance blows up, so the weights are usually clipped.

A second mitigation is randomized position within the top few slots. On a small fraction of requests (maybe 1%), the top 5 slots get shuffled. The resulting data, a clean random sample with no position confound, trains a small position-bias model whose outputs are used to debias the full log. This costs a little short-term engagement and buys a lot of long-term robustness.

A third mitigation is diversity in the re-ranker. Forcing a creator cap and a topic cap prevents a single creator or topic from owning the slate just because the ranker likes them. This protects against the rich-get-richer dynamic at a per-request level, independent of training-time fixes.

None of these mitigations is a complete fix. The feedback loop does not disappear, it just gets dampened. Monitoring the Gini coefficient over creator impressions weekly is a cheap way to catch the loop when it starts to run away.

There is also a harder, slower feedback loop through the retrieval stage. The two-tower trains on watches, and watches only happen on what was shown. Items the two-tower never surfaces for a user never get watches, so the model never learns whether the user would have liked them. Over months, this can shrink the effective catalog for each user. The counter is counterfactual estimation during training, usually some form of importance weighting on the sampled softmax loss, combined with the exploration pool keeping a trickle of under-served content in everyone's slate. Even then, the loop is never fully closed, and catalog coverage per user is a metric worth watching alongside Gini.

6.4 Multi-Objective Trade-offs

A ranker optimized only for long watches pushes the feed toward long-form, sticky, sometimes rabbit-hole content. Users might watch more today and come back less tomorrow. This is the central multi-objective problem, and it does not have a clean analytic solution. It has a policy, and the policy gets tuned with experiments.

The setup is the linear blend of head probabilities introduced earlier: long-watch, completion, like, share, dislike (negative), and survey. The survey head is the anchor. It is the only head that directly asks users whether the recommendation was any good, decoupled from engagement. Weights get tuned so that moving the blend to favor survey-positive content does not drop watch time so far that the business suffers, and moving it to favor watch time does not drop survey scores so far that the product feels exploitative.

Tuning happens on 2-to-4-week A/B tests. Retention signals are long-horizon, and the difference between a watch-time win and a retention win can take weeks to show up. A team that ships on 7-day windowed watch time, ignoring survey scores, has a standing invitation to discover negative retention effects in the following quarter. The fix is discipline: every ranker launch has to clear both watch time and survey score holds, and guardrails on dislike rate and creator Gini have to stay flat.

A useful mental model is the Pareto frontier between short-term engagement and long-term satisfaction. Any single weight configuration sits somewhere on that frontier. Moving along the frontier trades one for the other, moving off the frontier is an outright loss, and the job of weight tuning is to pick the point on the frontier that matches product priorities this quarter. The frontier itself shifts over time as the catalog changes and as user norms evolve, so the tuning is recurring work, not a one-time setup. The product decision of where to sit on the frontier is explicit and owned by product leadership, not buried in a loss function.

Reinforcement learning-style approaches, where the policy directly optimizes long-horizon reward, are tempting here. Off-policy correction methods like REINFORCE with importance weighting have been deployed in production and do capture some of the long-horizon signal that a supervised multi-task ranker misses. The catch is that they require careful propensity estimation and they are much harder to debug when they regress. The pragmatic pattern today is a supervised multi-task ranker with RL-style corrections layered on top rather than as the backbone.

Offline evaluation replays a held-out week of logged impressions through the new candidate set and ranker. Retrieval is evaluated with Recall@1000, which measures whether good candidates even reach the ranker. Ranking is evaluated with watch-time-weighted nDCG and completion@10, plus per-head calibration. Slicing matters: the same model can look great on average and terrible on cold-start items, new users, or underserved languages. Any launch needs slice-level guardrails, not just aggregates.

Online evaluation is A/B testing. The ramp goes 1% to 5% to 50% over a couple of weeks, with a long hold on the control. Primary metrics are watch time per impression and survey score. Guardrails are p99 latency, error rate, dislike rate, creator Gini coefficient, and safety violation rate. An experiment that lifts the primary metric but regresses any guardrail does not ship, full stop.

Monitoring runs continuously on production models, not just during experiments. Training-serving feature skew is tracked with KL divergence between the training distribution and the live serving distribution on each feature; a per-feature alert fires when skew exceeds a threshold. Model staleness is tracked by checking recent performance against the current retrain cadence; if the daily ranker starts drifting within a day, the cadence needs to increase. The feedback-loop amplification indicator is the weekly Gini over creator impressions. When Gini climbs, it is often a sign that the diversity re-ranker has lost ground or that IPW weights have drifted.

A few concrete monitoring thresholds that tend to hold up across platforms: feature KL divergence above 0.1 on any single feature is worth paging on, because it almost always predicts a model quality issue within a day or two. Per-head calibration error above 10% of the head's operating range is a hard break that disqualifies a model from serving. Ranker p99 latency variance week over week above 20% is usually a sign that a feature store or ANN shard is degraded. Creator Gini week over week climbing by more than 0.02 is a sign the system is consolidating, which is worth an investigation even if engagement metrics look flat.

Slice-level monitoring is where subtle problems hide. Aggregate watch time can be flat while cold-start items lose 15% and new users lose 8%, with popular items gaining just enough to cover. A ranker launch that passes aggregate guardrails but regresses any major slice should not ship until the slice regression is understood. Common slices to always check: cold-start items (under 1000 impressions), new users (first 5 sessions), top 10 languages, top 10 markets, low-engagement user segments, mobile vs. TV vs. desktop.

Iteration on a video recommender is slow and careful work. Features get added and retired with per-feature launches. Model architecture changes go through longer holds because subtle regressions in long-watch calibration can take weeks to show up in retention. The weight tuning on the multi-objective blend is a regular task because user behavior shifts and the right trade-off this quarter is not the right one next quarter.

The next chapter moves from video to product recommendation, where the target shifts from engagement to purchase intent and the trade-offs change in interesting ways.