Ad click prediction estimates the probability that a user clicks a specific ad in a specific context, and it's the model that sits at the heart of every major ads business. On the surface it looks like another ranking problem, but two constraints make it unique. The output has to be a calibrated probability because it feeds into an auction where pCTR is multiplied by an advertiser's bid to compute expected value, and the full decision has to happen inside a single-digit millisecond budget because an ad request is blocking a page render somewhere. Get the calibration wrong and the auction allocates revenue to the wrong ads; get the latency wrong and the page ships without an ad at all.

Clarifying Questions

The first step isn't drawing boxes. It's pinning down what "ad click prediction" means on this specific platform, because the answer is different on a search results page, a social feed, and a display network.

Candidate: "What surface are we predicting clicks on? Search ads, feed-embedded ads, display network banners, or all three?"

Interviewer: "Feed-embedded sponsored content, similar to the promoted posts you see on Facebook or Instagram. Assume a single ad slot inserted at positions 4, 10, and 16 in the organic feed."

Candidate: "How is the ad selected? Is there a retrieval stage, or am I ranking a pre-filtered shortlist?"

Interviewer: "Retrieval is separate. A candidate generator returns between 500 and 2000 eligible ads based on targeting rules, budget pacing, and frequency caps. Your model scores that shortlist."

Candidate: "What does an advertiser bid on? A click, an impression, or a downstream conversion?"

Interviewer: "Most advertisers bid on clicks, some bid on conversions with a target cost per action. For this design, assume cost-per-click billing. The auction multiplies your predicted CTR by the bid to get the effective cost, and the highest effective cost wins the slot."

Candidate: "What's the traffic scale and latency budget?"

Interviewer: "Around 300 billion ad opportunities per day, peak traffic near 8 million QPS. The full ad decision, including candidate retrieval, CTR scoring, auction, and bid calculation, has to fit inside 30 milliseconds. Your piece, the CTR model from feature fetch to score, has a 10 millisecond p99 budget."

Candidate: "What defines a click label? Any tap, or a qualified click that gets past the landing page?"

Interviewer: "A click is any user-initiated tap on the ad that redirects to the advertiser's destination. Qualified click, where the landing page loads and the user stays for at least two seconds, is a separate label we'll use for calibration and fraud filtering, not the primary training target."

Candidate: "Are we optimizing for CTR alone, or do we have a multi-objective setup like feed ranking?"

Interviewer: "CTR is the primary prediction because it goes into the auction. You'll also want a conversion-rate head for advertisers on CPA bidding, but that's a secondary output and you can describe it briefly."

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

The core model is a binary classifier. For a user u, a candidate ad a, and a context c (device, time, page content, preceding organic items), it predicts P(click | u, a, c). Training data is impression logs joined to click logs, where a row is positive if the impression produced a click within the attribution window and negative otherwise.

Two things separate this from a generic click classifier. The prediction has to be a well-calibrated probability, not a ranking score, because the downstream auction turns it into money. If the model predicts 0.10 but the real rate is 0.05, every auction is paying twice what it should for that ad. And the label distribution is brutally skewed. Real CTR on feed ads is typically between 0.5% and 3%, which means 97% or more of training rows are negatives. The training pipeline has to subsample negatives to keep training tractable, and the model has to correct for that subsampling at serving time or every prediction will be biased high.

For advertisers on conversion-optimized campaigns, a second head predicts P(conversion | click, u, a, c), and the effective value to the advertiser is pCTR × pConv × bid. The two heads share most features and embeddings but have separate output layers and different training data, since conversions arrive days later and are much rarer than clicks.

Metrics

Normalized Entropy is the one metric specific to ads that deserves a note. It sits between 0 and 1 (lower is better), and it answers "how much better than 'always predict the baseline CTR' is my model?" Because CTR baselines drift seasonally, raw log-loss changes even when the model is stable. NE divides it out.

A single ad request flows through four services, all sitting inside a 30ms decision budget. Candidate generation fans out to ad-index shards, the CTR scorer batches predictions, the auction runs second-price pricing, and a logging service captures everything the model saw so the training loop can reconstruct it.

The retrieval layer is not the interesting part of this chapter. It's a targeting-constrained lookup against an inverted index keyed on user segments, with budget pacing and frequency caps applied as post-filters. The output is a shortlist of 500 to 2000 ads that the CTR model is responsible for scoring.

What sits inside the CTR scorer is the focus of this design. The scorer owns three responsibilities that downstream services depend on: it produces a calibrated click probability for each candidate, it records the exact feature values used at scoring time so the training pipeline can reconstruct them, and it publishes prediction-level telemetry (distribution histograms, calibration buckets, latency percentiles) that the monitoring stack consumes in near real time. The auction service reads the probabilities, multiplies by advertiser bids, and picks winners. The logging service archives features and predictions for training. The monitoring service watches for distribution drift. Three very different consumers, all fed by the same request.

Three separate loops keep the model working correctly, and they run on different cadences.

The daily loop retrains the full model from a 14-to-30 day window of logs. The streaming loop keeps user-level and ad-level features fresh so a campaign launched at noon can have accurate CTR estimates by 1 PM. The hourly loop is a lighter online update: it uses FTRL-Proximal to adjust the linear part of the model on just-seen click data without touching the deep-network weights. The split matters because the deep network's embeddings are expensive to update online, but the linear "wide" side is where freshness has the biggest payoff.

Sources

Training data is the Cartesian product of impression logs and click logs joined on (request_id, ad_id). Anything the model wants to condition on has to be captured at impression time because the features at click time would leak label information.

The key design choice is what the pipeline calls a "feature snapshot." At impression time, the serving stack records the exact feature values it used to score the ad. This lets training data match serving data exactly, eliminating training-serving skew from rolling aggregates that change between when the impression was logged and when the trainer reads it.

The join on impression_id is the piece that glues everything together. Impressions arrive first, clicks arrive seconds to minutes later, and conversions arrive hours to days later. The streaming join holds impressions in a windowed state store, matches clicks as they arrive, and emits labeled rows once the attribution window closes. Late clicks past the window get dropped rather than attributed, because relabeling a historical impression row invalidates any training job that already consumed it.

Labels and Delayed Feedback

A click is a positive label if it arrives within a 30-minute attribution window of the impression. Outside that window, the event is counted as a new click tied to a later impression if there is one. This window is a practical compromise: too short and you lose legitimate late clicks; too long and training data becomes stale and you confuse overlapping impressions.

Conversions are harder because they can arrive hours or days after the click, and the training pipeline can't wait days for fresh data. Two patterns coexist:

Most large platforms use the first pattern for production training and keep the second as a research track.

Feature Engineering

Features fall into five groups. The table below isn't exhaustive, but it hits the categories that matter for this problem.

Cross features are where ads CTR models earn their reputation. A feature like user_interest=fitness on its own has modest signal, and ad_category=running_shoes on its own has modest signal, but user_interest=fitness × ad_category=running_shoes is where the real lift lives. Hand-engineered crosses were the dominant technique in ads modeling for a decade before the deep networks caught up, and they're still essential today.

Most sparse features go through the hashing trick. Each unique token gets mapped to one of 2^K buckets, where K is typically between 24 and 28. Collisions do happen but they're rare enough that the loss in signal is smaller than the memory saved.

The same hashing function runs in both the training pipeline and the serving stack, and it gets versioned together with the model checkpoint. A hash-function change forces a full retrain.

Sample Bias

The impression log is produced by the current model's predictions. If the live model avoids showing an ad-user pair, that pair never shows up as a training row, and the next model can't learn anything new about it. This is a classic exploration problem, and it's why every ads platform runs a small always-on exploration slot that picks randomly from the candidate set without regard for predicted CTR. The exploration traffic is logged with an explore=true flag so training can weight it correctly when counteracting selection bias.

Position bias is a related issue. An ad shown at feed position 4 has a higher click rate than the same ad shown at position 16, not because users like it more but because position 4 is more visible. The training pipeline handles this by including position as a feature during training but setting it to a fixed value (often position 0 or the median position) at serving time. This lets the model learn the position effect without letting position contaminate the score used in the auction. A more sophisticated variant uses inverse-propensity weighting, where each training row is reweighted by the inverse of its position's expected CTR share.

Baseline: Logistic Regression on Hashed Features

Before any neural architecture, the industry baseline is logistic regression on hashed sparse features plus a handful of dense features, trained with FTRL-Proximal. It sounds old-fashioned, but it's still a deployed workhorse at several large platforms for tail traffic, fallback serving, and as a canary to validate that the deep model is worth its extra complexity.

The appeal is practical. Training is embarrassingly parallel and can ingest a full day of logs in an hour. Prediction is a sparse dot product, which serves in under a millisecond even for a shortlist of 2000 ads. The weights are interpretable: if a cross feature's coefficient drops month over month, an analyst can see it and ask why. And FTRL produces sparse solutions, which means most hash buckets end up with zero weight and don't cost serving memory.

The limits show up when the feature engineering burden grows. Every useful cross feature has to be declared by hand, and the combinatorial explosion is painful. By the time the feature list hits tens of thousands of hand-designed crosses, teams start looking at models that learn crosses automatically.

Advanced: Wide and Deep

The Wide and Deep architecture is the canonical upgrade. It stacks two branches with complementary strengths and sums their logits.

The wide side is the logistic regression from the baseline, and it memorizes. Its input is a long vector of hashed cross features, and it learns the exact coefficient for user_country=IN × ad_category=mobile_plans × hour=19, which is a specific combination a deep network would struggle to represent without many examples.

The deep side is a multi-layer perceptron over dense features and embedded categoricals, and it generalizes. Each sparse feature goes through an embedding table that maps its hash bucket to a dense vector of 16 to 64 dimensions. The embeddings are concatenated with the numeric features and passed through three hidden layers, each with ReLU activation and batch normalization.

The two sides are trained jointly on a single log-loss objective, and their gradients flow independently through their own parameter sets. In production, the deep side typically has 50M to 200M parameters, most of which live in the embedding tables. The wide side is sparser, maybe 10M active weights after FTRL's L1 regularization has zeroed out the rest.

Later architectures like DCN, DeepFM, and DLRM replace or augment the wide side with models that learn crosses automatically. DCN adds explicit cross layers that compute feature interactions to arbitrary order, and DeepFM replaces the hashed crosses with a factorization machine. All of these outperform the hand-crafted Wide and Deep on benchmarks, but the production cost-benefit is less clear. Many large platforms still ship Wide and Deep variants because interpretability and FTRL-based online updates are operationally valuable.

Training

Training data is 14 to 30 days of logs, joined and subsampled. Negative subsampling is essential: at a 1.5% positive rate, keeping all negatives means the data is 98.5% junk from a gradient perspective. A common rate is to keep 1 in 10 negatives, which balances out to roughly 15% positives in the training set.

Each surviving row carries an inverse-probability weight, and the log-loss is weighted so the expected gradient matches the full, unsampled dataset. Without that weighting, the model predicts 10× the true CTR because it's trained on a distribution with 10× more positives than reality.

The full training loop runs daily on a multi-GPU cluster using data-parallel training. Synchronous SGD with large batch sizes (8K to 32K) works well because the gradient signal is strong per example and batch size scales up nicely. Learning rates are tuned with a linear warmup followed by cosine decay, and a second-moment optimizer like Adam or Adagrad handles the sparse embedding updates.

The hourly incremental update only touches the wide side. It loads the freshest hour of logs, runs FTRL-Proximal to get a weight delta, and merges the delta into the serving model. The deep side is updated daily from the full retrain. This split matters because FTRL on sparse linear models converges in minutes on a single CPU, while retraining the deep embeddings would take hours.

Canary rollout deserves its own note. A fresh model starts serving 1% of traffic, and the serving fleet watches five guardrail metrics for the first hour: p99 latency, CTR on the canary slice, calibration error, prediction-distribution KL vs the previous model, and qualified-click rate. If any of the five regresses beyond a preset threshold, the rollout halts and rolls back automatically. Full rollout, if all checks pass, completes over the next four to six hours.

Inference Pipeline

The serving path is tight. A request arrives, the service fetches features for the user, the ad candidates, and the context, runs a single forward pass over all candidates in one batch, applies calibration, and returns a vector of pCTRs to the auction.

The trick that makes this work is batching. Instead of scoring one ad at a time, the serving stack runs a single forward pass over the full shortlist. The user features are repeated across the batch dimension, ad features vary per row, and the model computes 500 to 2000 scores in roughly the same wall-clock time it would take to compute 50. Modern accelerators (A100s, H100s, or in-house TPU-equivalents) are bandwidth-bound, and the marginal cost of additional batch rows is close to zero until the batch saturates memory.

Latency Budget

The 10ms p99 budget breaks down roughly like this.

Feature fetch is typically the biggest source of tail latency, not the model itself. A p50 feature fetch might take 1ms while the p99 spikes to 8ms because a cache miss hits cold storage. The mitigation is aggressive read-through caching at the serving host, plus replica reads from the closest feature-store shard. Some platforms push user features all the way to the serving host's local memory using a shared-memory daemon that subscribes to the feature-store update stream.

Feature Store

The online feature store has two tiers. A hot tier (Redis or an in-memory store like Memcached) holds user state and ad-level aggregates with sub-millisecond reads. A warm tier (RocksDB or Aerospike) holds less frequently accessed features and falls back to the hot tier on miss. Feature writes go to both tiers from the streaming pipeline, with the hot tier treated as an LRU cache.

User embeddings, which are too expensive to compute per request, are precomputed nightly or hourly and stored alongside user features. The advantage is that the serving model doesn't need to run the user-tower forward pass; it just looks up the embedding and feeds it into the last few layers of the model. The trade-off is that user embeddings are a few hours stale, so recent behavior doesn't make it into the embedding directly. Streaming features on top of the embedding cover the gap.

Degraded Serving

When the model service fails, or when the feature store is slow enough to blow the latency budget, the serving stack falls back to a cached CTR lookup table keyed on (ad_id, user_country, device_os). The table is trivially fast to serve and gives a reasonable-quality pCTR for the auction to work with. Revenue in degraded mode is typically 85 to 95% of full mode, which is dramatically better than the revenue from silently dropping ad responses past the budget.

The fallback is built as a strict tier, not a graceful one. The scorer runs with a hard deadline of 8ms, and any request that exceeds it is aborted and routed to the lookup table. Partial results aren't merged, because mixing a full model's output for some candidates with cached values for others produces inconsistent rankings. The lookup table is refreshed every 15 minutes from the same log stream that feeds training, so it stays close enough to current reality that auctions running on fallback predictions don't catastrophically misprice ads. When incidents do happen, the post-mortem focuses on what caused the scorer's timeout more than what the fallback served, because the fallback doing its job is the success case.

6.1 Feature Crosses and Sparse Representations

Crosses are what separates an ad CTR model from a generic classifier. The reason is that CTR signals are fundamentally interactional: a user's likelihood of clicking an ad depends on the specific combination of user properties, ad properties, and context, not on each in isolation. An example that comes up in almost every discussion: user_country=IN × ad_advertiser=flipkart × hour=20 might have a CTR of 4%, while user_country=IN × ad_advertiser=flipkart averaged across all hours is 1.5%, and ad_advertiser=flipkart × hour=20 averaged across all countries is 0.8%. Without the three-way cross, the model has no way to represent the evening-shopping peak in India for that advertiser.

Three approaches are live in industry today.

Hand-crafted crosses + hashing is the original approach and still widely deployed. An engineer inspects feature importance plots and log analyses, identifies promising cross candidates, and hard-codes them into the feature pipeline. The wide side of the model learns a coefficient per hashed cross. This works well when the team has strong analysts who can find the important crosses, but it caps out around tens of thousands of hand-designed features.

Factorization Machines (FM) and their deep variants (DeepFM, FiBiNET) learn pairwise feature interactions automatically as dot products of feature embeddings. The wide side becomes a sum over all pairs ⟨v_i, v_j⟩ * x_i * x_j, which has the same representational power as a full pairwise cross but uses O(k * n) parameters instead of O(n^2). The downside is that FMs only model pairwise interactions cleanly, and modeling higher-order crosses requires additional machinery.

Deep and Cross Networks (DCN) add explicit cross layers that compute x_{l+1} = x_0 * (w * x_l) + x_l, which models polynomial interactions of arbitrary order without the parameter blow-up. DCN-v2 is the current state-of-the-art for learning crosses on pure sparse data, and it consistently beats hand-crafted crosses on public benchmarks.

Hashing collisions deserve a note. At 2^26 buckets, a feature with 10 million unique values has a collision probability around 15%, which sounds bad but rarely hurts in practice. Most collisions pair a frequent value with a rare one, and the model effectively ignores the rare one. The real risk is a collision between two frequent values, which corrupts both of their learned weights. The mitigation is using a larger hash space (2^28) for high-cardinality features and monitoring collision rates in the training pipeline.

6.2 Calibration and Real-Time Bidding

Calibration matters in ads more than almost anywhere else in ML because the prediction is multiplied by money. The auction computes expected value as pCTR × bid_amount, and the highest expected value wins the slot. If the model reports 0.04 when the true rate is 0.02, every auction is evaluating that ad at twice its real worth. Revenue accounting uses the same probabilities to charge advertisers (under CPC billing, the charged cost is second_price × 1 / pCTR), so miscalibration corrupts billing too.

Two sources of miscalibration dominate in practice.

Negative subsampling bias is deterministic. If training keeps all positives and 1 in 10 negatives, the model learns a distribution with 10× too many positives. At serving time, every prediction is biased high. The fix is a simple closed-form correction.

For neg_keep_rate=0.1 and p_trained=0.3, the corrected prediction is 0.3 / (0.3 + 7) ≈ 0.041, which is much closer to real-world CTR.

Distribution drift is harder. A model trained last night expects a certain feature distribution. Overnight, an advertiser launches a viral creative, a sports event changes user behavior, or an app update shifts device fingerprints. The model's predictions remain internally consistent but systematically biased up or down by 5 to 15%.

The production fix is a two-stage calibration layer.

Isotonic regression is fit daily on held-out logs. It's a non-parametric monotonic mapping from raw predictions to calibrated probabilities, and it fixes local distortions the model has in specific probability bands. Platt scaling, which is a simple logistic regression on a single input, runs as a lightweight recalibration on a rolling one-hour window to correct fast drift. The two in sequence handle both persistent miscalibration and short-horizon drift.

Monitoring the calibration is done with reliability diagrams. The system bucks predictions into deciles and plots predicted vs observed CTR for each. In a well-calibrated model, the points fall on the diagonal. A systematic offset above or below the diagonal means the model is biased; a curved shape means it's miscalibrated at the extremes. Alarms fire when the calibration error exceeds 5% in any decile over a four-hour rolling window, because that's often the earliest signal that a feature-store outage or a data pipeline bug is corrupting predictions.

6.3 Explore-Exploit and Cold-Start Ads

Fresh ads are a systemic blind spot. A newly launched campaign has zero impressions and zero clicks, which means the model's embeddings for its ad_id are untrained. The model falls back on advertiser-level and category-level features, but those are coarse enough that the predicted CTR has a wide confidence interval. Under pure greedy ranking, the auction almost never picks the new ad because its expected value is lower than that of established ads with tighter estimates. The new campaign starves, and the advertiser churns.

Three approaches to the exploration problem work in practice.

Epsilon-greedy exploration reserves a fraction of ad opportunities for random selection among new ads. Simple to implement and easy to reason about, but it's inefficient because the exploration rate is constant regardless of how uncertain the model is about specific ads. New ads that already have 10,000 impressions don't need exploration; they need exploitation.

Upper Confidence Bound (UCB) scores an ad by pCTR + α * uncertainty, where the uncertainty term shrinks as the ad accumulates impressions. Ads with high uncertainty get a boost, so they're more likely to be shown early, and the boost decays naturally as data arrives. Production implementation requires maintaining per-ad impression counts and a closed-form uncertainty estimate.

Thompson sampling maintains a posterior distribution over each ad's true CTR, and at serving time it samples a CTR value from that posterior and treats the sample as the point estimate. Ads with uncertain posteriors occasionally draw high samples and win auctions they'd otherwise lose, and the posterior tightens as more data arrives. Thompson sampling is the industry favorite because it's principled, it degrades gracefully as ads mature, and the sampling itself is almost free at serving time.

In a production system, the posterior is conditioned on features. Raw beta-binomial posteriors work for ad-level exploration, and a neural network outputs a distribution over possible CTRs for feature-conditioned exploration. Meta's DLRM-variant ads model and Google's various CTR models all use some form of posterior sampling for cold-start handling.

Cold-start mitigation doesn't stop at the policy. Ads systems also seed new campaigns with per-category priors (a new restaurant ad borrows the category's historical CTR as a starting point), give advertisers a free first-week impression allocation so they accumulate data fast enough to escape exploration, and cap the penalty applied to new ads so they don't get crushed by a well-calibrated but conservative model.

Offline Evaluation

Offline evaluation uses a held-out day from logs. The model is trained on days 1-13 and evaluated on day 14, and the primary metrics are log-loss, AUC, PR-AUC, normalized entropy, and calibration error per decile. A new model ships to production only if NE improves by at least 1% (the noise floor on large offline sets is surprisingly low) and calibration error doesn't regress.

Slicing matters as much as the headline number. A model that improves NE by 2% globally can still regress on a high-value slice like conversion-bid advertisers or on a sensitive slice like new-user traffic. Evaluation dashboards break out metrics by country, device type, ad format, advertiser tier, and user tenure, and the ship bar is that no slice regresses by more than 3% on NE or 2% on calibration error. Chasing aggregate wins while ignoring slice regressions is one of the classic ways ads model launches hurt revenue even when offline numbers look great.

The subtlety is that offline data reflects the previous model's selection bias. If the old model never showed ad X to user Y, there's no training row for that pair, and the new model can't be fairly evaluated on pairs that didn't appear. Counterfactual evaluation using inverse-propensity weighting fixes part of this: each training row is weighted by 1 / P(shown | features), where the propensity is estimated from a separate model. This reweights the data toward what a policy-free distribution would look like. It's noisy for rare pairs but substantially better than naive evaluation when the two models under comparison have very different exposure policies.

Calibration plots are checked at every offline evaluation. A regression in AUC with stable calibration is a ranking problem; a regression in calibration with stable AUC is a serving-side bug (subsampling correction, feature pipeline, version mismatch). Separating the two dimensions at offline time saves hours of debugging post-launch.

Online Evaluation

Online evaluation is an A/B test with a rate-holdback or traffic split design. A 1% to 5% holdback runs for one to two weeks, long enough to capture weekly cycles and advertiser budget rebalancing. The primary success metrics are RPM (the revenue composite), qualified-click rate, and advertiser ROAS, and the guardrail metrics are serving p99 latency, prediction-distribution KL-divergence, and organic feed metrics (time spent, negative feedback rate).

Ads A/B tests are harder than organic A/B tests because the auction creates interference. An ad that wins in the treatment bucket doesn't get shown in control, so the two buckets see different mixes of ads. The standard mitigation is budget-equalized experiments: each bucket is given its own copy of the advertiser budgets, and budget pacing runs independently within each bucket. This is expensive in engineering effort but it's the only way to measure revenue lift without budget effects contaminating the comparison.

Monitoring

Once a model is in production, the monitoring split mirrors the training pipeline's structure. Three classes of signal get dashboarded.

The single most important monitor is training-serving skew on high-importance features. A feature that's mean-zero in training but mean-0.1 at serving time silently corrupts every prediction, and the calibration layer often masks it until the error compounds. Running a shadow job that recomputes features at serving time from logged raw inputs, then compares against what the live pipeline produced, catches these bugs in hours instead of weeks.

Retraining cadence is usually daily for the full model and hourly for the wide-side FTRL update. Some platforms run continuous training that ingests logs minute-by-minute, but the serving infrastructure required to atomically swap a model every few minutes without latency spikes is non-trivial, and the marginal accuracy gain over hourly is modest for most ads surfaces.

Q1: Why is calibration so much more important in ad CTR prediction than in most other ranking problems?

The downstream auction multiplies predicted CTR by the advertiser's bid to compute expected value, and the winning ad is the one with the highest product. If the model's predictions are off by a constant factor, the winner stays the same (so ranking quality is preserved) but the prices charged to advertisers, paid to publishers, and used for budget pacing are all wrong. A model that's 2× miscalibrated means advertisers are overspending or underspending by 2×, which shows up as churn and unhappy sales teams. Calibration also affects the choice between CPC-bidded and CPA-bidded ads: if one bid type's predictions drift while the other's stay stable, the auction systematically favors one over the other even when their true value is equal. In feed ranking or search ranking, where predictions just determine order, a constant miscalibration is invisible. In ads it's a direct revenue error.

Q2: Walk me through the math of negative subsampling and why you need to correct for it.

Training data has a natural positive rate of around 1-2%, which makes gradient updates inefficient because 98-99% of the loss is concentrated on a tiny subset of rows. Subsampling keeps all positives and a fraction r of negatives (say r = 0.1), which balances the training set to something like 10-15% positives. The model learns odds that are 1/r times too high. If the corrected probability is p and the trained probability is p', the closed-form relationship is p = p' / (p' + (1-p')/r). Without this correction, every serving prediction is biased up by a factor of roughly 1/r, which destroys the downstream auction because the absolute numbers drive bid calculations. The correction is a single scalar operation per prediction, so it's cheap at serving time, and the correction factor itself is a function of the known subsampling rate, not a learned parameter.

Q3: When would you choose to drop either the wide side or the deep side of a Wide and Deep model?

Drop the deep side when interpretability matters more than accuracy, when the team has strong cross-feature engineering talent, or when the serving fleet is CPU-only and can't afford embedding lookups. A pure FTRL-trained logistic regression is a workhorse for CTR prediction and still runs tail traffic at several large platforms. Drop the wide side when the team doesn't have bandwidth for hand-crafting crosses and the ad catalog is stable enough for deep embeddings to learn them automatically. In that case a DCN-v2 or DeepFM variant recovers most of the wide side's value. The architectures coexist in most production systems because they cover different failure modes: the wide side memorizes rare specific patterns that the deep side smooths over, and the deep side generalizes to pairs that the wide side never saw. Killing either removes a distinct kind of signal, and the decision should be driven by concrete A/B tests, not by intellectual preference for one architecture.

Q4: How do you choose the attribution window, and what are the trade-offs of a shorter versus a longer window?

The attribution window is the time between an impression and a click beyond which the click no longer counts as caused by that impression. A short window (say 5 minutes) keeps training data fresh and reduces the chance that a click is attributed to a stale impression when the user actually clicked an intervening ad. The downside is losing legitimate late clicks: users who saw an ad, left the app, and came back 20 minutes later to tap it. A long window (say 24 hours) captures those late clicks but introduces ambiguity when a user saw the same ad several times, and it delays how quickly the training pipeline can generate labeled rows. Most platforms settle somewhere in the middle, around 30 minutes to a few hours, and the choice is validated by measuring what fraction of clicks arrive at each time offset and where the diminishing returns flatten out. Longer windows also bias the model toward broad-awareness campaigns and away from direct-response campaigns, which is an advertiser-mix effect worth understanding before changing the window.

Q5: A new advertiser launches a campaign with five ads, all with zero history. Your model predicts low pCTR for all of them and they never win auctions. How do you diagnose and fix this?

The first check is whether the predictions are low because of missing ad embeddings (expected, and the fix is exploration) or because of a feature pipeline bug (unexpected and severe). Look at what features the new ads are populated with: advertiser-level CTR, category-level CTR, creative-type dummy, and landing-domain history should all be populated from day one. If any of those are zero or null, the training pipeline has a gap. Once features are verified, the fix is a combination of exploration policy and impression seeding. Thompson sampling on the posterior gives new ads occasional wins even with wide confidence intervals, and a seeded first-week budget lets them accumulate enough impressions to tighten the posterior fast. Some platforms also apply an explicit new-ad boost (a multiplicative adjustment to pCTR) that decays over the first 10,000 impressions. The boost isn't principled but it's practical, and it prevents advertiser churn from ads that the model would otherwise never evaluate. Over time, as the ad accumulates real CTR data, the boost and the posterior naturally converge on the true rate.