{"title":"Design an Ad Click Prediction System","description":"","content":"Ad click prediction estimates the probability that a user clicks a specific ad in a specific context, and it's the model that sits at the heart of every major ads business. On the surface it looks like another ranking problem, but two constraints make it unique. The output has to be a calibrated probability because it feeds into an auction where pCTR is multiplied by an advertiser's bid to compute expected value, and the full decision has to happen inside a single-digit millisecond budget because an ad request is blocking a page render somewhere. Get the calibration wrong and the auction allocates revenue to the wrong ads; get the latency wrong and the page ships without an ad at all.\n\n---\n\n# 1. Problem Formulation\n\n### Clarifying Questions\n\nThe first step isn't drawing boxes. It's pinning down what \"ad click prediction\" means on this specific platform, because the answer is different on a search results page, a social feed, and a display network.\n\n**Candidate:** \"What surface are we predicting clicks on? Search ads, feed-embedded ads, display network banners, or all three?\"\n\n**Interviewer:** \"Feed-embedded sponsored content, similar to the promoted posts you see on Facebook or Instagram. Assume a single ad slot inserted at positions 4, 10, and 16 in the organic feed.\"\n\n**Candidate:** \"How is the ad selected? Is there a retrieval stage, or am I ranking a pre-filtered shortlist?\"\n\n**Interviewer:** \"Retrieval is separate. A candidate generator returns between 500 and 2000 eligible ads based on targeting rules, budget pacing, and frequency caps. Your model scores that shortlist.\"\n\n**Candidate:** \"What does an advertiser bid on? A click, an impression, or a downstream conversion?\"\n\n**Interviewer:** \"Most advertisers bid on clicks, some bid on conversions with a target cost per action. For this design, assume cost-per-click billing. The auction multiplies your predicted CTR by the bid to get the effective cost, and the highest effective cost wins the slot.\"\n\n**Candidate:** \"What's the traffic scale and latency budget?\"\n\n**Interviewer:** \"Around 300 billion ad opportunities per day, peak traffic near 8 million QPS. The full ad decision, including candidate retrieval, CTR scoring, auction, and bid calculation, has to fit inside 30 milliseconds. Your piece, the CTR model from feature fetch to score, has a 10 millisecond p99 budget.\"\n\n**Candidate:** \"What defines a click label? Any tap, or a qualified click that gets past the landing page?\"\n\n**Interviewer:** \"A click is any user-initiated tap on the ad that redirects to the advertiser's destination. Qualified click, where the landing page loads and the user stays for at least two seconds, is a separate label we'll use for calibration and fraud filtering, not the primary training target.\"\n\n**Candidate:** \"Are we optimizing for CTR alone, or do we have a multi-objective setup like feed ranking?\"\n\n**Interviewer:** \"CTR is the primary prediction because it goes into the auction. You'll also want a conversion-rate head for advertisers on CPA bidding, but that's a secondary output and you can describe it briefly.\"\n\n### Requirements\n\n**Functional Requirements:**\n\n- Score a shortlist of 500 to 2000 candidate ads per request and return a calibrated click probability for each.\n- Handle sub-second frequency caps so the same user doesn't see the same ad twice in one session.\n- Support cold-start for new ads with zero history by routing them through an exploration policy.\n- Feed predictions to a downstream auction service that multiplies pCTR by the advertiser's bid to compute expected value.\n- Log every impression, click, post-click signal, and the features used at scoring time so the training pipeline can replay them faithfully.\n- Refresh the model on hourly or daily cadence so distribution drift doesn't degrade calibration.\n\n**Non-Functional Requirements:**\n\n- P99 end-to-end scoring latency under 10 milliseconds for a batch of up to 2000 candidates.\n- Calibration error (mean predicted CTR versus observed CTR, per decile) under 5% on rolling weekly windows.\n- Feature freshness under 60 seconds for user state and under 30 minutes for ad-level aggregates.\n- Revenue-neutral fallback: if the model service fails, a cached historical CTR table must serve degraded predictions so the auction still runs.\n- Fairness across advertisers: a small-budget advertiser with a genuinely relevant ad shouldn't be crowded out by a large-budget advertiser whose model predictions are slightly better simply because they have more training data.\n\n### ML Problem Framing\n\nThe core model is a binary classifier. For a user `u`, a candidate ad `a`, and a context `c` (device, time, page content, preceding organic items), it predicts P(click | u, a, c). Training data is impression logs joined to click logs, where a row is positive if the impression produced a click within the attribution window and negative otherwise.\n\nTwo things separate this from a generic click classifier. The prediction has to be a well-calibrated probability, not a ranking score, because the downstream auction turns it into money. If the model predicts 0.10 but the real rate is 0.05, every auction is paying twice what it should for that ad. And the label distribution is brutally skewed. Real CTR on feed ads is typically between 0.5% and 3%, which means 97% or more of training rows are negatives. The training pipeline has to subsample negatives to keep training tractable, and the model has to correct for that subsampling at serving time or every prediction will be biased high.\n\nFor advertisers on conversion-optimized campaigns, a second head predicts P(conversion | click, u, a, c), and the effective value to the advertiser is pCTR × pConv × bid. The two heads share most features and embeddings but have separate output layers and different training data, since conversions arrive days later and are much rarer than clicks.\n\n### Metrics\n\n\n| Type | Metric | Why It Matters |\n|---|---|---|\n| Offline | Log-loss | The loss the model is trained on; the metric that moves first when something regresses |\n| Offline | AUC-ROC | Measures ranking quality independent of calibration; useful when comparing model architectures |\n| Offline | PR-AUC | More informative than ROC on skewed data because it focuses on the positive class |\n| Offline | Normalized Entropy (NE) | Log-loss divided by the log-loss of a model that predicts the baseline CTR; isolates lift from picking the right number |\n| Offline | Calibration plot per decile | Predicted CTR vs observed CTR in each probability bin; the single most important diagnostic for ads |\n| Online | CTR lift | The whole point; measured on a holdback slice |\n| Online | Revenue per mille (RPM) | CTR × bid × fill rate; the dollar-level outcome |\n| Online | Advertiser ROAS | Return on ad spend for conversion-bid advertisers; their willingness to keep spending |\n| Guardrail | Qualified-click rate | Fraction of clicks that survive the two-second dwell filter; detects clickbait creative drift |\n| Guardrail | Serving p99 latency | A 12ms tail means the front-end is shipping pages with missing ads; straight revenue loss |\n| Guardrail | Prediction distribution KL-divergence | Compares live pCTR histogram to training histogram; catches feature-store outages before they hit revenue |\n\n\nNormalized Entropy is the one metric specific to ads that deserves a note. It sits between 0 and 1 (lower is better), and it answers \"how much better than 'always predict the baseline CTR' is my model?\" Because CTR baselines drift seasonally, raw log-loss changes even when the model is stable. NE divides it out.\n\n---\n\n# 2. System Architecture\n\nA single ad request flows through four services, all sitting inside a 30ms decision budget. Candidate generation fans out to ad-index shards, the CTR scorer batches predictions, the auction runs second-price pricing, and a logging service captures everything the model saw so the training loop can reconstruct it.\n\n\n```mermaid\nflowchart LR\n REQ[Ad
Request]:::primary --> TGT[Targeting
+ Retrieval]:::teal\n TGT --> FILT[Budget Pacing
+ Frequency Caps]:::red\n FILT --> FS[Feature
Fetch]:::orange\n FS --> CTR[CTR
Scorer]:::primary\n CTR --> AUC[Auction
+ Bidding]:::orange\n AUC --> SLOT[Ad Slot
Response]:::green\n CTR -.log.-> LOG[Impression
+ Feature Log]:::teal\n AUC -.log.-> LOG\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe retrieval layer is not the interesting part of this chapter. It's a targeting-constrained lookup against an inverted index keyed on user segments, with budget pacing and frequency caps applied as post-filters. The output is a shortlist of 500 to 2000 ads that the CTR model is responsible for scoring.\n\nWhat sits inside the CTR scorer is the focus of this design. The scorer owns three responsibilities that downstream services depend on: it produces a calibrated click probability for each candidate, it records the exact feature values used at scoring time so the training pipeline can reconstruct them, and it publishes prediction-level telemetry (distribution histograms, calibration buckets, latency percentiles) that the monitoring stack consumes in near real time. The auction service reads the probabilities, multiplies by advertiser bids, and picks winners. The logging service archives features and predictions for training. The monitoring service watches for distribution drift. Three very different consumers, all fed by the same request.\n\nThree separate loops keep the model working correctly, and they run on different cadences.\n\n\n```mermaid\nflowchart TD\n subgraph Offline[Offline Loop - Daily]\n IMPL[Impression
Logs]:::primary --> JN[Feature +
Label Join]:::teal\n JN --> SUB[Negative
Subsampling]:::orange\n SUB --> BATCH[Batch
Training]:::primary\n BATCH --> CKPT[Model
Checkpoint]:::green\n end\n\n subgraph Streaming[Streaming Loop - Seconds]\n EVT[Click +
Impression
Events]:::primary --> RT[Real-Time
Aggregator]:::teal\n RT --> AGG[Hourly
Ad CTR
Aggregates]:::orange\n AGG --> FS2[Online
Feature Store]:::orange\n UACT[User
Actions]:::primary --> USR[User State
Builder]:::teal\n USR --> FS2\n end\n\n subgraph Incremental[Incremental Loop - Hourly]\n STREAM[Recent
Click Logs]:::primary --> FTRL[FTRL Online
Update]:::orange\n FTRL --> DELTA[Wide-Side
Weight Delta]:::green\n end\n\n CKPT -.served by.-> SRV[Serving
Model]:::primary\n FS2 -.read by.-> SRV\n DELTA -.merged into.-> SRV\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe daily loop retrains the full model from a 14-to-30 day window of logs. The streaming loop keeps user-level and ad-level features fresh so a campaign launched at noon can have accurate CTR estimates by 1 PM. The hourly loop is a lighter online update: it uses FTRL-Proximal to adjust the linear part of the model on just-seen click data without touching the deep-network weights. The split matters because the deep network's embeddings are expensive to update online, but the linear \"wide\" side is where freshness has the biggest payoff.\n\n---\n\n# 3. Data\n\n### Sources\n\nTraining data is the Cartesian product of impression logs and click logs joined on `(request_id, ad_id)`. Anything the model wants to condition on has to be captured at impression time because the features at click time would leak label information.\n\n\n| Source | What's In It | Update Cadence |\n|---|---|---|\n| Impression log | Every ad shown: user_id, ad_id, slot, context, feature snapshot | Streaming, ~1B events/hour at peak |\n| Click log | Click events keyed to impression_id, with client timestamps | Streaming, ~20M events/hour |\n| Conversion log | Post-click events from advertiser pixels or SDK callbacks | Streaming, delayed by minutes to days |\n| User profile store | Demographics, interests, historical engagement | Updated by nightly batch + streaming writes |\n| Ad catalog | Creative attributes, advertiser, category, landing page | Updated on ad creation and edit |\n| Context feed | Organic items above and below the ad slot | Captured at request time |\n\n\nThe key design choice is what the pipeline calls a \"feature snapshot.\" At impression time, the serving stack records the exact feature values it used to score the ad. This lets training data match serving data exactly, eliminating training-serving skew from rolling aggregates that change between when the impression was logged and when the trainer reads it.\n\n\n```mermaid\nflowchart LR\n IMP[Impression
Events]:::primary --> KAF[Kafka
Ingest]:::teal\n CLK[Click
Events]:::primary --> KAF\n CONV[Conversion
Events]:::primary --> KAF\n KAF --> JOIN[Streaming
Join on
impression_id]:::orange\n JOIN --> RAW[Raw Training
Rows]:::primary\n JOIN --> AGG[Rolling
Aggregator]:::orange\n AGG --> ONLINE[Online Feature
Store]:::green\n RAW --> LAKE[Data Lake
Parquet]:::teal\n LAKE --> TRAIN[Training
Jobs]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe join on `impression_id` is the piece that glues everything together. Impressions arrive first, clicks arrive seconds to minutes later, and conversions arrive hours to days later. The streaming join holds impressions in a windowed state store, matches clicks as they arrive, and emits labeled rows once the attribution window closes. Late clicks past the window get dropped rather than attributed, because relabeling a historical impression row invalidates any training job that already consumed it.\n\n### Labels and Delayed Feedback\n\nA click is a positive label if it arrives within a 30-minute attribution window of the impression. Outside that window, the event is counted as a new click tied to a later impression if there is one. This window is a practical compromise: too short and you lose legitimate late clicks; too long and training data becomes stale and you confuse overlapping impressions.\n\nConversions are harder because they can arrive hours or days after the click, and the training pipeline can't wait days for fresh data. Two patterns coexist:\n\n- **Short-window training with correction:** train every hour on conversions seen in the last 6 hours, then apply a conversion-delay correction learned from historical data to un-bias predictions for long-tail conversions.\n- **Delayed-feedback modeling:** explicitly model P(conversion by day t | click at day 0) and let the model predict at any time offset; this is more principled but much harder to debug.\n\nMost large platforms use the first pattern for production training and keep the second as a research track.\n\n### Feature Engineering\n\nFeatures fall into five groups. The table below isn't exhaustive, but it hits the categories that matter for this problem.\n\n\n| Group | Example Features | Type | Typical Freshness |\n|---|---|---|---|\n| User | user_id, age bucket, gender, country, device OS, app version | Sparse categorical + dense | Hourly |\n| User history | 7-day clicked-ad-categories, 30-day advertiser interactions, recent search queries | Sparse bag-of-tokens | 5 minutes |\n| Ad | ad_id, advertiser_id, category, creative format, landing domain | Sparse categorical | On create |\n| Ad stats | ad CTR in last hour, last day, last 7 days, advertiser CTR | Dense | 5-15 minutes |\n| Context | hour_of_week, position in feed, preceding organic item category, connection type | Sparse + dense | Request-time |\n| Cross | user_country × ad_category, user_interest × advertiser, time × device_os | Hashed sparse | Request-time |\n\n\nCross features are where ads CTR models earn their reputation. A feature like `user_interest=fitness` on its own has modest signal, and `ad_category=running_shoes` on its own has modest signal, but `user_interest=fitness × ad_category=running_shoes` is where the real lift lives. Hand-engineered crosses were the dominant technique in ads modeling for a decade before the deep networks caught up, and they're still essential today.\n\n\n```mermaid\nflowchart LR\n A[user_country
= US]:::primary --> CAT[Concatenate
+ Name]:::teal\n B[ad_category
= running_shoes]:::primary --> CAT\n C[hour
= 20]:::primary --> CAT\n CAT --> STR[cross:
US:running_shoes:20]:::orange\n STR --> H[MurmurHash3
mod 2^26]:::orange\n H --> BUK[Bucket
14,382,199]:::green\n BUK --> WGT[Wide-Side
Weight Lookup]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nMost sparse features go through the hashing trick. Each unique token gets mapped to one of `2^K` buckets, where `K` is typically between 24 and 28. Collisions do happen but they're rare enough that the loss in signal is smaller than the memory saved.\n\n\n```python\nimport mmh3\n\ndef hash_feature(name: str, value: str, n_buckets: int = 2**26) -> int:\n # Mix the feature name into the key so a user_id \"1234\"\n # doesn't collide with an ad_id \"1234\"\n key = f\"{name}={value}\"\n return mmh3.hash(key, seed=42) % n_buckets\n\n# Example: generate a cross feature\nuser_country = \"US\"\nad_category = \"running_shoes\"\ncross_bucket = hash_feature(\n \"user_country_x_ad_category\",\n f\"{user_country}:{ad_category}\"\n)\n```\n\n\nThe same hashing function runs in both the training pipeline and the serving stack, and it gets versioned together with the model checkpoint. A hash-function change forces a full retrain.\n\n### Sample Bias\n\nThe impression log is produced by the current model's predictions. If the live model avoids showing an ad-user pair, that pair never shows up as a training row, and the next model can't learn anything new about it. This is a classic exploration problem, and it's why every ads platform runs a small always-on exploration slot that picks randomly from the candidate set without regard for predicted CTR. The exploration traffic is logged with an `explore=true` flag so training can weight it correctly when counteracting selection bias.\n\nPosition bias is a related issue. An ad shown at feed position 4 has a higher click rate than the same ad shown at position 16, not because users like it more but because position 4 is more visible. The training pipeline handles this by including position as a feature during training but setting it to a fixed value (often position 0 or the median position) at serving time. This lets the model learn the position effect without letting position contaminate the score used in the auction. A more sophisticated variant uses inverse-propensity weighting, where each training row is reweighted by the inverse of its position's expected CTR share.\n\n---\n\n# 4. Model\n\n### Baseline: Logistic Regression on Hashed Features\n\nBefore any neural architecture, the industry baseline is logistic regression on hashed sparse features plus a handful of dense features, trained with FTRL-Proximal. It sounds old-fashioned, but it's still a deployed workhorse at several large platforms for tail traffic, fallback serving, and as a canary to validate that the deep model is worth its extra complexity.\n\nThe appeal is practical. Training is embarrassingly parallel and can ingest a full day of logs in an hour. Prediction is a sparse dot product, which serves in under a millisecond even for a shortlist of 2000 ads. The weights are interpretable: if a cross feature's coefficient drops month over month, an analyst can see it and ask why. And FTRL produces sparse solutions, which means most hash buckets end up with zero weight and don't cost serving memory.\n\nThe limits show up when the feature engineering burden grows. Every useful cross feature has to be declared by hand, and the combinatorial explosion is painful. By the time the feature list hits tens of thousands of hand-designed crosses, teams start looking at models that learn crosses automatically.\n\n### Advanced: Wide and Deep\n\nThe Wide and Deep architecture is the canonical upgrade. It stacks two branches with complementary strengths and sums their logits.\n\n\n```mermaid\nflowchart LR\n IN[Input
Features]:::primary --> WC[Hashed Cross
Features]:::orange\n IN --> EMB[Categorical
Embeddings]:::teal\n IN --> DENSE[Dense
Features]:::teal\n\n WC --> WIDE[Wide Side
Linear]:::orange\n EMB --> CONCAT[Concat +
BatchNorm]:::teal\n DENSE --> CONCAT\n CONCAT --> H1[Hidden
1024]:::primary\n H1 --> H2[Hidden
512]:::primary\n H2 --> H3[Hidden
256]:::primary\n H3 --> DEEP[Deep
Logit]:::primary\n\n WIDE --> SUM[Sum
Logits]:::green\n DEEP --> SUM\n SUM --> SIG[Sigmoid]:::green\n SIG --> P[pCTR]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe **wide side** is the logistic regression from the baseline, and it memorizes. Its input is a long vector of hashed cross features, and it learns the exact coefficient for `user_country=IN × ad_category=mobile_plans × hour=19`, which is a specific combination a deep network would struggle to represent without many examples.\n\nThe **deep side** is a multi-layer perceptron over dense features and embedded categoricals, and it generalizes. Each sparse feature goes through an embedding table that maps its hash bucket to a dense vector of 16 to 64 dimensions. The embeddings are concatenated with the numeric features and passed through three hidden layers, each with ReLU activation and batch normalization.\n\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass WideAndDeep(nn.Module):\n def __init__(self, wide_dim, sparse_vocabs, emb_dim=32,\n dense_dim=64, hidden=(1024, 512, 256)):\n super().__init__()\n # Wide side: a single linear layer over hashed cross features\n self.wide = nn.Linear(wide_dim, 1, bias=False)\n\n # Deep side: one embedding table per sparse feature group\n self.embeddings = nn.ModuleList([\n nn.EmbeddingBag(v, emb_dim, mode=\"mean\")\n for v in sparse_vocabs\n ])\n\n deep_in = emb_dim * len(sparse_vocabs) + dense_dim\n layers = []\n prev = deep_in\n for h in hidden:\n layers += [nn.Linear(prev, h), nn.BatchNorm1d(h), nn.ReLU()]\n prev = h\n layers.append(nn.Linear(prev, 1))\n self.deep = nn.Sequential(*layers)\n\n def forward(self, wide_x, sparse_x, dense_x):\n deep_emb = [e(s) for e, s in zip(self.embeddings, sparse_x)]\n deep_in = torch.cat(deep_emb + [dense_x], dim=1)\n logit = self.wide(wide_x) + self.deep(deep_in)\n return torch.sigmoid(logit).squeeze(-1)\n```\n\n\nThe two sides are trained jointly on a single log-loss objective, and their gradients flow independently through their own parameter sets. In production, the deep side typically has 50M to 200M parameters, most of which live in the embedding tables. The wide side is sparser, maybe 10M active weights after FTRL's L1 regularization has zeroed out the rest.\n\nLater architectures like DCN, DeepFM, and DLRM replace or augment the wide side with models that learn crosses automatically. DCN adds explicit cross layers that compute feature interactions to arbitrary order, and DeepFM replaces the hashed crosses with a factorization machine. All of these outperform the hand-crafted Wide and Deep on benchmarks, but the production cost-benefit is less clear. Many large platforms still ship Wide and Deep variants because interpretability and FTRL-based online updates are operationally valuable.\n\n### Training\n\nTraining data is 14 to 30 days of logs, joined and subsampled. Negative subsampling is essential: at a 1.5% positive rate, keeping all negatives means the data is 98.5% junk from a gradient perspective. A common rate is to keep 1 in 10 negatives, which balances out to roughly 15% positives in the training set.\n\n\n```python\ndef subsample(rows, keep_positive=1.0, keep_negative=0.1):\n for r in rows:\n if r.label == 1 and random.random() < keep_positive:\n yield r, 1.0 / keep_positive\n elif r.label == 0 and random.random() < keep_negative:\n yield r, 1.0 / keep_negative\n```\n\n\nEach surviving row carries an inverse-probability weight, and the log-loss is weighted so the expected gradient matches the full, unsampled dataset. Without that weighting, the model predicts 10× the true CTR because it's trained on a distribution with 10× more positives than reality.\n\nThe full training loop runs daily on a multi-GPU cluster using data-parallel training. Synchronous SGD with large batch sizes (8K to 32K) works well because the gradient signal is strong per example and batch size scales up nicely. Learning rates are tuned with a linear warmup followed by cosine decay, and a second-moment optimizer like Adam or Adagrad handles the sparse embedding updates.\n\nThe hourly incremental update only touches the wide side. It loads the freshest hour of logs, runs FTRL-Proximal to get a weight delta, and merges the delta into the serving model. The deep side is updated daily from the full retrain. This split matters because FTRL on sparse linear models converges in minutes on a single CPU, while retraining the deep embeddings would take hours.\n\n\n```mermaid\nflowchart LR\n LOGS[30-Day
Log Window]:::primary --> SUB[Subsample
+ Feature Build]:::teal\n SUB --> TRN[Daily GPU
Training]:::orange\n TRN --> VAL[Holdout
Eval]:::teal\n VAL --> REG[Model
Registry]:::green\n\n HR[Last Hour
Logs]:::primary --> FTRL2[FTRL-Proximal
CPU Update]:::orange\n FTRL2 --> WGT[Wide-Side
Weight Delta]:::green\n WGT -.merge.-> REG\n\n REG --> DEPLOY[Canary
Rollout]:::primary\n DEPLOY --> PROD[Serving
Fleet]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nCanary rollout deserves its own note. A fresh model starts serving 1% of traffic, and the serving fleet watches five guardrail metrics for the first hour: p99 latency, CTR on the canary slice, calibration error, prediction-distribution KL vs the previous model, and qualified-click rate. If any of the five regresses beyond a preset threshold, the rollout halts and rolls back automatically. Full rollout, if all checks pass, completes over the next four to six hours.\n\n---\n\n# 5. Serving\n\n### Inference Pipeline\n\nThe serving path is tight. A request arrives, the service fetches features for the user, the ad candidates, and the context, runs a single forward pass over all candidates in one batch, applies calibration, and returns a vector of pCTRs to the auction.\n\n\n```mermaid\nflowchart LR\n REQ[Ad Req
t=0ms]:::primary --> CAND[Candidate
Shortlist
+2ms]:::orange\n CAND --> UF[User
Features
+2ms]:::teal\n CAND --> AF[Ad Feature
Batch
+2ms]:::teal\n UF --> FWD[Batched
Forward Pass
+3ms]:::primary\n AF --> FWD\n FWD --> CAL[Calibration
+0.5ms]:::orange\n CAL --> OUT[pCTR Vector
+0.5ms]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe trick that makes this work is batching. Instead of scoring one ad at a time, the serving stack runs a single forward pass over the full shortlist. The user features are repeated across the batch dimension, ad features vary per row, and the model computes 500 to 2000 scores in roughly the same wall-clock time it would take to compute 50. Modern accelerators (A100s, H100s, or in-house TPU-equivalents) are bandwidth-bound, and the marginal cost of additional batch rows is close to zero until the batch saturates memory.\n\n### Latency Budget\n\nThe 10ms p99 budget breaks down roughly like this.\n\n\n| Stage | Budget | Notes |\n|---|---|---|\n| Feature fetch (user) | 2ms | Single key lookup in Redis or RocksDB; user features are pre-aggregated |\n| Feature fetch (ads) | 2ms | Batched multi-get; ad features keyed on ad_id |\n| Feature assembly | 0.5ms | Hash, cross, and tensor-pack the inputs |\n| Model forward pass | 3ms | GPU-batched inference; dominated by embedding lookups, not matmul |\n| Calibration | 0.5ms | Isotonic regression lookup; vectorized |\n| Network + serialization | 2ms | Ad service to scorer RPC, response wire format |\n| Buffer | 0ms | Intentionally none; every ms counts |\n\n\nFeature fetch is typically the biggest source of tail latency, not the model itself. A p50 feature fetch might take 1ms while the p99 spikes to 8ms because a cache miss hits cold storage. The mitigation is aggressive read-through caching at the serving host, plus replica reads from the closest feature-store shard. Some platforms push user features all the way to the serving host's local memory using a shared-memory daemon that subscribes to the feature-store update stream.\n\n### Feature Store\n\nThe online feature store has two tiers. A hot tier (Redis or an in-memory store like Memcached) holds user state and ad-level aggregates with sub-millisecond reads. A warm tier (RocksDB or Aerospike) holds less frequently accessed features and falls back to the hot tier on miss. Feature writes go to both tiers from the streaming pipeline, with the hot tier treated as an LRU cache.\n\nUser embeddings, which are too expensive to compute per request, are precomputed nightly or hourly and stored alongside user features. The advantage is that the serving model doesn't need to run the user-tower forward pass; it just looks up the embedding and feeds it into the last few layers of the model. The trade-off is that user embeddings are a few hours stale, so recent behavior doesn't make it into the embedding directly. Streaming features on top of the embedding cover the gap.\n\n### Degraded Serving\n\nWhen the model service fails, or when the feature store is slow enough to blow the latency budget, the serving stack falls back to a cached CTR lookup table keyed on `(ad_id, user_country, device_os)`. The table is trivially fast to serve and gives a reasonable-quality pCTR for the auction to work with. Revenue in degraded mode is typically 85 to 95% of full mode, which is dramatically better than the revenue from silently dropping ad responses past the budget.\n\nThe fallback is built as a strict tier, not a graceful one. The scorer runs with a hard deadline of 8ms, and any request that exceeds it is aborted and routed to the lookup table. Partial results aren't merged, because mixing a full model's output for some candidates with cached values for others produces inconsistent rankings. The lookup table is refreshed every 15 minutes from the same log stream that feeds training, so it stays close enough to current reality that auctions running on fallback predictions don't catastrophically misprice ads. When incidents do happen, the post-mortem focuses on what caused the scorer's timeout more than what the fallback served, because the fallback doing its job is the success case.\n\n---\n\n# 6. Deep Dives\n\n### 6.1 Feature Crosses and Sparse Representations\n\nCrosses are what separates an ad CTR model from a generic classifier. The reason is that CTR signals are fundamentally interactional: a user's likelihood of clicking an ad depends on the specific combination of user properties, ad properties, and context, not on each in isolation. An example that comes up in almost every discussion: `user_country=IN × ad_advertiser=flipkart × hour=20` might have a CTR of 4%, while `user_country=IN × ad_advertiser=flipkart` averaged across all hours is 1.5%, and `ad_advertiser=flipkart × hour=20` averaged across all countries is 0.8%. Without the three-way cross, the model has no way to represent the evening-shopping peak in India for that advertiser.\n\nThree approaches are live in industry today.\n\n**Hand-crafted crosses + hashing** is the original approach and still widely deployed. An engineer inspects feature importance plots and log analyses, identifies promising cross candidates, and hard-codes them into the feature pipeline. The wide side of the model learns a coefficient per hashed cross. This works well when the team has strong analysts who can find the important crosses, but it caps out around tens of thousands of hand-designed features.\n\n**Factorization Machines (FM)** and their deep variants (DeepFM, FiBiNET) learn pairwise feature interactions automatically as dot products of feature embeddings. The wide side becomes a sum over all pairs `⟨v_i, v_j⟩ * x_i * x_j`, which has the same representational power as a full pairwise cross but uses `O(k * n)` parameters instead of `O(n^2)`. The downside is that FMs only model pairwise interactions cleanly, and modeling higher-order crosses requires additional machinery.\n\n**Deep and Cross Networks (DCN)** add explicit cross layers that compute `x_{l+1} = x_0 * (w * x_l) + x_l`, which models polynomial interactions of arbitrary order without the parameter blow-up. DCN-v2 is the current state-of-the-art for learning crosses on pure sparse data, and it consistently beats hand-crafted crosses on public benchmarks.\n\n\n| Approach | Strength | Weakness | Best For |\n|---|---|---|---|\n| Hand-crafted + hashing | Interpretable, fast, FTRL-friendly | Analyst-bound, caps at 10K-100K crosses | Teams with deep domain expertise |\n| Factorization Machines | Learns all pairwise crosses automatically | Pairwise only, limited higher-order | Mid-scale teams without cross analysts |\n| Deep and Cross (DCN) | Learns higher-order crosses, state-of-the-art | Harder to interpret, no cheap online update | Large teams with ML infra budget |\n\n\nHashing collisions deserve a note. At `2^26` buckets, a feature with 10 million unique values has a collision probability around 15%, which sounds bad but rarely hurts in practice. Most collisions pair a frequent value with a rare one, and the model effectively ignores the rare one. The real risk is a collision between two frequent values, which corrupts both of their learned weights. The mitigation is using a larger hash space (2^28) for high-cardinality features and monitoring collision rates in the training pipeline.\n\n### 6.2 Calibration and Real-Time Bidding\n\nCalibration matters in ads more than almost anywhere else in ML because the prediction is multiplied by money. The auction computes expected value as `pCTR × bid_amount`, and the highest expected value wins the slot. If the model reports 0.04 when the true rate is 0.02, every auction is evaluating that ad at twice its real worth. Revenue accounting uses the same probabilities to charge advertisers (under CPC billing, the charged cost is `second_price × 1 / pCTR`), so miscalibration corrupts billing too.\n\nTwo sources of miscalibration dominate in practice.\n\n**Negative subsampling bias** is deterministic. If training keeps all positives and 1 in 10 negatives, the model learns a distribution with 10× too many positives. At serving time, every prediction is biased high. The fix is a simple closed-form correction.\n\n\n```python\ndef correct_subsampling(p_trained: float, neg_keep_rate: float) -> float:\n # Undo the effect of downsampling negatives at rate r.\n # Derivation: the model learned odds that are 1/r too high.\n p = p_trained\n return p / (p + (1 - p) / neg_keep_rate)\n```\n\n\nFor `neg_keep_rate=0.1` and `p_trained=0.3`, the corrected prediction is `0.3 / (0.3 + 7) ≈ 0.041`, which is much closer to real-world CTR.\n\n**Distribution drift** is harder. A model trained last night expects a certain feature distribution. Overnight, an advertiser launches a viral creative, a sports event changes user behavior, or an app update shifts device fingerprints. The model's predictions remain internally consistent but systematically biased up or down by 5 to 15%.\n\nThe production fix is a two-stage calibration layer.\n\n\n```mermaid\nflowchart LR\n RAW[Raw Model
pCTR]:::primary --> SUB[Subsample
Correction]:::orange\n SUB --> ISO[Isotonic
Regression]:::orange\n ISO --> MON[Drift
Monitor]:::teal\n MON --> PLAT[Platt
Recalibration]:::orange\n PLAT --> FINAL[Served
pCTR]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nIsotonic regression is fit daily on held-out logs. It's a non-parametric monotonic mapping from raw predictions to calibrated probabilities, and it fixes local distortions the model has in specific probability bands. Platt scaling, which is a simple logistic regression on a single input, runs as a lightweight recalibration on a rolling one-hour window to correct fast drift. The two in sequence handle both persistent miscalibration and short-horizon drift.\n\nMonitoring the calibration is done with reliability diagrams. The system bucks predictions into deciles and plots predicted vs observed CTR for each. In a well-calibrated model, the points fall on the diagonal. A systematic offset above or below the diagonal means the model is biased; a curved shape means it's miscalibrated at the extremes. Alarms fire when the calibration error exceeds 5% in any decile over a four-hour rolling window, because that's often the earliest signal that a feature-store outage or a data pipeline bug is corrupting predictions.\n\n### 6.3 Explore-Exploit and Cold-Start Ads\n\nFresh ads are a systemic blind spot. A newly launched campaign has zero impressions and zero clicks, which means the model's embeddings for its ad_id are untrained. The model falls back on advertiser-level and category-level features, but those are coarse enough that the predicted CTR has a wide confidence interval. Under pure greedy ranking, the auction almost never picks the new ad because its expected value is lower than that of established ads with tighter estimates. The new campaign starves, and the advertiser churns.\n\nThree approaches to the exploration problem work in practice.\n\n**Epsilon-greedy exploration** reserves a fraction of ad opportunities for random selection among new ads. Simple to implement and easy to reason about, but it's inefficient because the exploration rate is constant regardless of how uncertain the model is about specific ads. New ads that already have 10,000 impressions don't need exploration; they need exploitation.\n\n**Upper Confidence Bound (UCB)** scores an ad by `pCTR + α * uncertainty`, where the uncertainty term shrinks as the ad accumulates impressions. Ads with high uncertainty get a boost, so they're more likely to be shown early, and the boost decays naturally as data arrives. Production implementation requires maintaining per-ad impression counts and a closed-form uncertainty estimate.\n\n**Thompson sampling** maintains a posterior distribution over each ad's true CTR, and at serving time it samples a CTR value from that posterior and treats the sample as the point estimate. Ads with uncertain posteriors occasionally draw high samples and win auctions they'd otherwise lose, and the posterior tightens as more data arrives. Thompson sampling is the industry favorite because it's principled, it degrades gracefully as ads mature, and the sampling itself is almost free at serving time.\n\n\n```python\nimport numpy as np\n\nclass ThompsonSampler:\n def __init__(self, prior_alpha=1.0, prior_beta=100.0):\n # Beta prior centered around 1% CTR; tune to the platform\n self.prior_alpha = prior_alpha\n self.prior_beta = prior_beta\n\n def sample(self, impressions: int, clicks: int) -> float:\n # Posterior after observing k clicks in n impressions\n alpha = self.prior_alpha + clicks\n beta = self.prior_beta + impressions - clicks\n return np.random.beta(alpha, beta)\n\n# At serving time, sample once per (ad, request) tuple\nsampler = ThompsonSampler()\nsampled_ctr = sampler.sample(impressions=50, clicks=1)\n# Feed sampled_ctr (not the point estimate) into the auction\n```\n\n\nIn a production system, the posterior is conditioned on features. Raw beta-binomial posteriors work for ad-level exploration, and a neural network outputs a distribution over possible CTRs for feature-conditioned exploration. Meta's DLRM-variant ads model and Google's various CTR models all use some form of posterior sampling for cold-start handling.\n\n\n| Policy | Exploration Efficiency | Implementation Cost | Works When |\n|---|---|---|---|\n| Epsilon-greedy | Low; flat rate wastes budget on mature ads | Trivial | Small catalogs, early-stage products |\n| UCB | Medium; shrinks with data | Moderate; needs per-ad counts | Stable inventory, well-understood feature |\n| Thompson sampling | High; naturally adaptive | Moderate; needs posterior | Most production ads systems |\n\n\nCold-start mitigation doesn't stop at the policy. Ads systems also seed new campaigns with per-category priors (a new restaurant ad borrows the category's historical CTR as a starting point), give advertisers a free first-week impression allocation so they accumulate data fast enough to escape exploration, and cap the penalty applied to new ads so they don't get crushed by a well-calibrated but conservative model.\n\n---\n\n# 7. Evaluation and Iteration\n\n### Offline Evaluation\n\nOffline evaluation uses a held-out day from logs. The model is trained on days 1-13 and evaluated on day 14, and the primary metrics are log-loss, AUC, PR-AUC, normalized entropy, and calibration error per decile. A new model ships to production only if NE improves by at least 1% (the noise floor on large offline sets is surprisingly low) and calibration error doesn't regress.\n\nSlicing matters as much as the headline number. A model that improves NE by 2% globally can still regress on a high-value slice like conversion-bid advertisers or on a sensitive slice like new-user traffic. Evaluation dashboards break out metrics by country, device type, ad format, advertiser tier, and user tenure, and the ship bar is that no slice regresses by more than 3% on NE or 2% on calibration error. Chasing aggregate wins while ignoring slice regressions is one of the classic ways ads model launches hurt revenue even when offline numbers look great.\n\nThe subtlety is that offline data reflects the previous model's selection bias. If the old model never showed ad X to user Y, there's no training row for that pair, and the new model can't be fairly evaluated on pairs that didn't appear. Counterfactual evaluation using inverse-propensity weighting fixes part of this: each training row is weighted by `1 / P(shown | features)`, where the propensity is estimated from a separate model. This reweights the data toward what a policy-free distribution would look like. It's noisy for rare pairs but substantially better than naive evaluation when the two models under comparison have very different exposure policies.\n\nCalibration plots are checked at every offline evaluation. A regression in AUC with stable calibration is a ranking problem; a regression in calibration with stable AUC is a serving-side bug (subsampling correction, feature pipeline, version mismatch). Separating the two dimensions at offline time saves hours of debugging post-launch.\n\n### Online Evaluation\n\nOnline evaluation is an A/B test with a rate-holdback or traffic split design. A 1% to 5% holdback runs for one to two weeks, long enough to capture weekly cycles and advertiser budget rebalancing. The primary success metrics are RPM (the revenue composite), qualified-click rate, and advertiser ROAS, and the guardrail metrics are serving p99 latency, prediction-distribution KL-divergence, and organic feed metrics (time spent, negative feedback rate).\n\nAds A/B tests are harder than organic A/B tests because the auction creates interference. An ad that wins in the treatment bucket doesn't get shown in control, so the two buckets see different mixes of ads. The standard mitigation is budget-equalized experiments: each bucket is given its own copy of the advertiser budgets, and budget pacing runs independently within each bucket. This is expensive in engineering effort but it's the only way to measure revenue lift without budget effects contaminating the comparison.\n\n### Monitoring\n\nOnce a model is in production, the monitoring split mirrors the training pipeline's structure. Three classes of signal get dashboarded.\n\n\n| Signal | What It Catches | Alarm Threshold |\n|---|---|---|\n| Prediction distribution | Feature store outages, pipeline drift | KL > 0.1 vs rolling baseline |\n| Calibration error per decile | Model drift, creative mix shift | >5% error in any decile over 4h |\n| Feature freshness lag | Streaming pipeline backlog | User features older than 5 min |\n| Training-serving skew | Feature pipeline divergence | Top-100 features by importance, any >2% divergence |\n| Ad-level CTR distribution | Creative drift, advertiser manipulation | Per-ad CTR shift >3σ over 7 days |\n| Conversion attribution lag | Delayed feedback outage | Hourly conversion rate <50% of baseline |\n\n\nThe single most important monitor is training-serving skew on high-importance features. A feature that's mean-zero in training but mean-0.1 at serving time silently corrupts every prediction, and the calibration layer often masks it until the error compounds. Running a shadow job that recomputes features at serving time from logged raw inputs, then compares against what the live pipeline produced, catches these bugs in hours instead of weeks.\n\n\n```mermaid\nflowchart LR\n LIVE[Live Serving
Predictions]:::primary --> KL[KL-Divergence
Detector]:::orange\n LIVE --> CAL[Calibration
Per Decile]:::orange\n LOG[Impression
Log Stream]:::primary --> SHADOW[Shadow Feature
Recompute]:::teal\n SHADOW --> SKEW[Skew
Detector]:::orange\n LOG --> CTRMON[Per-Ad CTR
Distribution]:::orange\n\n KL --> ALERT[Alert
Router]:::red\n CAL --> ALERT\n SKEW --> ALERT\n CTRMON --> ALERT\n ALERT --> PAGE[On-Call
Page]:::red\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nRetraining cadence is usually daily for the full model and hourly for the wide-side FTRL update. Some platforms run continuous training that ingests logs minute-by-minute, but the serving infrastructure required to atomically swap a model every few minutes without latency spikes is non-trivial, and the marginal accuracy gain over hourly is modest for most ads surfaces.\n\n---\n\n# Interview Questions\n\n**Q1: Why is calibration so much more important in ad CTR prediction than in most other ranking problems?**\n\nThe downstream auction multiplies predicted CTR by the advertiser's bid to compute expected value, and the winning ad is the one with the highest product. If the model's predictions are off by a constant factor, the winner stays the same (so ranking quality is preserved) but the prices charged to advertisers, paid to publishers, and used for budget pacing are all wrong. A model that's 2× miscalibrated means advertisers are overspending or underspending by 2×, which shows up as churn and unhappy sales teams. Calibration also affects the choice between CPC-bidded and CPA-bidded ads: if one bid type's predictions drift while the other's stay stable, the auction systematically favors one over the other even when their true value is equal. In feed ranking or search ranking, where predictions just determine order, a constant miscalibration is invisible. In ads it's a direct revenue error.\n\n**Q2: Walk me through the math of negative subsampling and why you need to correct for it.**\n\nTraining data has a natural positive rate of around 1-2%, which makes gradient updates inefficient because 98-99% of the loss is concentrated on a tiny subset of rows. Subsampling keeps all positives and a fraction `r` of negatives (say `r = 0.1`), which balances the training set to something like 10-15% positives. The model learns odds that are `1/r` times too high. If the corrected probability is `p` and the trained probability is `p'`, the closed-form relationship is `p = p' / (p' + (1-p')/r)`. Without this correction, every serving prediction is biased up by a factor of roughly `1/r`, which destroys the downstream auction because the absolute numbers drive bid calculations. The correction is a single scalar operation per prediction, so it's cheap at serving time, and the correction factor itself is a function of the known subsampling rate, not a learned parameter.\n\n**Q3: When would you choose to drop either the wide side or the deep side of a Wide and Deep model?**\n\nDrop the deep side when interpretability matters more than accuracy, when the team has strong cross-feature engineering talent, or when the serving fleet is CPU-only and can't afford embedding lookups. A pure FTRL-trained logistic regression is a workhorse for CTR prediction and still runs tail traffic at several large platforms. Drop the wide side when the team doesn't have bandwidth for hand-crafting crosses and the ad catalog is stable enough for deep embeddings to learn them automatically. In that case a DCN-v2 or DeepFM variant recovers most of the wide side's value. The architectures coexist in most production systems because they cover different failure modes: the wide side memorizes rare specific patterns that the deep side smooths over, and the deep side generalizes to pairs that the wide side never saw. Killing either removes a distinct kind of signal, and the decision should be driven by concrete A/B tests, not by intellectual preference for one architecture.\n\n**Q4: How do you choose the attribution window, and what are the trade-offs of a shorter versus a longer window?**\n\nThe attribution window is the time between an impression and a click beyond which the click no longer counts as caused by that impression. A short window (say 5 minutes) keeps training data fresh and reduces the chance that a click is attributed to a stale impression when the user actually clicked an intervening ad. The downside is losing legitimate late clicks: users who saw an ad, left the app, and came back 20 minutes later to tap it. A long window (say 24 hours) captures those late clicks but introduces ambiguity when a user saw the same ad several times, and it delays how quickly the training pipeline can generate labeled rows. Most platforms settle somewhere in the middle, around 30 minutes to a few hours, and the choice is validated by measuring what fraction of clicks arrive at each time offset and where the diminishing returns flatten out. Longer windows also bias the model toward broad-awareness campaigns and away from direct-response campaigns, which is an advertiser-mix effect worth understanding before changing the window.\n\n**Q5: A new advertiser launches a campaign with five ads, all with zero history. Your model predicts low pCTR for all of them and they never win auctions. How do you diagnose and fix this?**\n\nThe first check is whether the predictions are low because of missing ad embeddings (expected, and the fix is exploration) or because of a feature pipeline bug (unexpected and severe). Look at what features the new ads are populated with: advertiser-level CTR, category-level CTR, creative-type dummy, and landing-domain history should all be populated from day one. If any of those are zero or null, the training pipeline has a gap. Once features are verified, the fix is a combination of exploration policy and impression seeding. Thompson sampling on the posterior gives new ads occasional wins even with wide confidence intervals, and a seeded first-week budget lets them accumulate enough impressions to tighten the posterior fast. Some platforms also apply an explicit new-ad boost (a multiplicative adjustment to pCTR) that decays over the first 10,000 impressions. The boost isn't principled but it's practical, and it prevents advertiser churn from ads that the model would otherwise never evaluate. Over time, as the ad accumulates real CTR data, the boost and the posterior naturally converge on the true rate.\n\n---\n\n# Summary\n\n- Ad click prediction is a binary classifier whose output is a calibrated probability, because the downstream auction multiplies it by a bid to allocate revenue, and miscalibration corrupts pricing even when it preserves ranking.\n- The feature space is dominated by sparse categoricals and their crosses; Wide and Deep, DCN, and DeepFM are the mainstream architectures, with the wide side memorizing specific high-value combinations and the deep side generalizing across them.\n- Training data is massively imbalanced, so negative subsampling is used to keep training tractable and a closed-form probability correction at serving time undoes the induced bias.\n- Serving has a sub-10ms p99 latency budget, and the trick that makes it work is batched forward-pass scoring over the full candidate shortlist in a single model call, with feature fetch (not the model itself) typically being the biggest source of tail latency.\n- Calibration is maintained with a two-stage layer: isotonic regression fit on held-out logs handles persistent distortions, and a lightweight Platt scaling layer on a rolling one-hour window handles fast drift.\n- Cold-start is a first-class concern; Thompson sampling over a per-ad posterior is the dominant policy because it's principled, naturally adaptive, and cheap at serving time.\n- Offline evaluation uses log-loss, AUC, normalized entropy, and calibration plots; online evaluation is a budget-equalized A/B test that measures RPM, qualified-click rate, advertiser ROAS, and guardrails like serving latency and organic feed quality.\n- Retraining runs on three cadences: daily full retrains for the deep network, hourly FTRL updates for the wide side, and streaming feature updates for user state and ad-level aggregates.\n- The most dangerous production failure isn't a bad model, it's training-serving skew that silently corrupts feature values; shadow jobs comparing live features to logs are the best defense.","pageType":"ml-system-design"}

Design an Ad Click Prediction System

Ashish Pratap Singh

Get Premium