Clarifying Questions

Notification systems at Meta Platforms, LinkedIn, DoorDash, and Duolingo all operate at massive scale, but they optimize for different goals.

Defining the scope, user surfaces, and north-star metric upfront shapes almost every design decision that follows.

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

The natural framing is two-stage. Stage one is a multi-task scorer that predicts, for a candidate notification, the probability of each downstream action. Stage two is a decision policy that combines those predictions into a single utility score, subtracts a fatigue cost, and compares the result to a threshold.

The naive framing is a single-head classifier predicting P(open | sent). It looks reasonable. Every delivered notification yields a clean binary label. The problem is that optimizing open rate rewards shock-value notifications, and the model has no way to trade an open today against an unsubscribe next week. A user who opens ten in a row and then disables notifications entirely is a negative event the single-head model cannot see.

A better framing uses four heads:

The utility for a candidate is:

The fatigue term is not a prediction, it's a deterministic function of how many notifications the user has already received and engaged with in a recent window. The weights w1-w4 and the penalty lambda are calibrated against retention in A/B tests, not against any single offline metric.

Stage two turns V into a decision. If V exceeds a dynamic threshold, the notification is sent. The threshold moves with population-level signals: if platform-wide opt-out rate ticks up, a control loop raises the threshold and sends fewer marginal notifications the next day.

Metrics

Retention and unsubscribe rate are the two numbers that matter most. Every deep-dive design choice downstream gets evaluated against both, and a gain on one without a regression on the other is the bar for shipping a change.

A notification's journey starts when some service publishes a candidate event onto the event bus. From there it flows through eligibility filtering, feature enrichment, scoring, a send-decision step, a channel router, and a send-time optimizer before reaching the delivery gateway. Real-time and batched paths share the scorer and the feature store but diverge in how they handle timing.

The eligibility filter is the first and most important stage. It applies hard constraints: opt-outs, quiet hours, daily cap, blocked senders, regulatory rules.

A notification that fails any of these is dropped without being scored, which saves compute and guarantees the model can never override a user's explicit preference.

The filter runs on cheap lookups: a bitset for opt-outs per notification type, a counter for today's sends per user, a timezone table for quiet hours.

Surviving candidates enter the feature fetch stage, which reads user features, sender features, content features, and fatigue state from an online feature store. The multi-task scorer runs the DNN and produces the four head probabilities.

The send-decision step computes V, subtracts fatigue, compares to the current threshold, and emits a boolean plus a priority tier. The channel router picks push, email, or inbox based on per-channel predicted engagement. The send-time optimizer picks immediate versus queued-for-later based on the user's active-hours profile and the priority tier.

Scale pressure shapes a few of these choices. With 10 billion candidates per day and an average of 20 candidates per DAU, the system needs to average around 120K scoring QPS with peaks around 200K during high-traffic hours. Feature fetches dominate the latency budget, so the online store is colocated with the scorer and cached aggressively for hot user features.

Send-decision state (today's cap counter, fatigue) sits in a fast key-value store with per-user sharding and a sub-millisecond read path. Storage for decision logs runs at roughly two terabytes a day at one kilobyte per log record, which is tractable with commodity object storage plus daily partitioning.

Data Flow

Three loops operate on very different cadences and together keep the system calibrated.

The offline loop trains the model, corrects for selection bias with inverse-propensity weighting, and recalibrates the four heads. It runs daily with a full retrain weekly. The streaming loop keeps short-horizon user state fresh: recent app opens, recent notification engagements, today's fatigue counter. The online loop handles the per-event scoring path and completes in tens of milliseconds.

Data Sources

The system pulls from seven primary sources. The event bus carries candidate notification events. A user activity stream carries app opens, background pings, and engagement events on the feed and messaging. The user profile store holds stable attributes and declared preferences. Channel-state tables track device tokens, email bounce history, and per-channel subscription status.

Historical engagement logs, partitioned by day, drive offline training. A notification-content service provides sender metadata and content features. A delivery gateway returns success or failure signals from APNs, FCM, and the email provider, which close the loop on whether a send actually reached the device.

Feature Engineering

Features fall into four natural families. User features describe who the recipient is and how engaged they are. Candidate features describe the specific notification. Context features describe the time and device. Cross features describe the interaction between user and candidate, and they tend to be the highest-signal inputs the model has.

User activity features decay over three time horizons. Past-hour, past-day, and past-week counts of app opens capture current intent, recent habit, and baseline engagement.

A user who just opened the app five seconds ago does not need a re-engagement push, and a simple recency feature lets the model learn that shape without having to reason about it from scratch. Notification receipt counts over the same windows feed into fatigue-related features.

Candidate features capture what the notification is about. Type (comment, like, friend request, re-engagement) is a categorical. Content features include length, whether the content contains media, and the content's global engagement rate aggregated over the past hour.

Sender features include how often this sender's notifications have been opened across the platform in the past week, which acts as a quality prior.

Cross features are the workhorse. User-sender affinity, computed as the smoothed click-through rate of past notifications from this sender to this user, is the single strongest predictor of open. User-type engagement rate captures whether this user responds to this category at all. Channel affinity captures the user's historical engagement rate on push versus email versus inbox for this notification type, which the channel router later consumes directly.

Feature Table

Baseline Approach

A reasonable baseline is a gradient-boosted tree trained on P(open | sent). Candidates are scored, the top N per user per day subject to the hard cap are sent, the rest are dropped. It is simple to build and ships useful relevance against a random-policy baseline. It fails on three fronts.

First, a single head cannot represent the trade-off between short-term opens and long-term unsubscribes. Second, it has no notion of fatigue, so a burst of high-probability candidates around a popular event all fire together and drive same-day opt-outs. Third, it treats every delivery slot as interchangeable, so it has no model of when to send or which channel to use. The advanced model exists to solve all three.

Advanced Approach

The production model is a multi-task deep network with a shared trunk and four task heads. The trunk is a feed-forward stack with embedding lookups for categoricals (type, sender_id bucketed, device, hour_of_week) and dense inputs for numeric features. The four heads each produce a probability: open, action-given-open, unsubscribe, and dismiss.

Multi-task sharing matters because unsubscribe and dismiss labels are rare. Unsubscribe rates sit around 0.1% of sends, dismiss around 20%. Training those heads in isolation starves them of signal; sharing the trunk lets them borrow representations learned from the much denser open head. The loss is a weighted sum of per-head binary cross-entropy.

Head loss weights reflect label frequency and business cost. The unsubscribe head gets a larger weight than its 0.1% rate would suggest, because a false negative there is a long-term retention loss the other heads will not catch.

Fatigue Model

Fatigue is not a learned head; it is an explicit state variable applied after scoring. The reason is policy. A learned fatigue signal might decide that a particular heavy-engagement user can tolerate more sends, and over time it would push that user to the edge of opt-out. An explicit fatigue term gives the system a safety property: volume is bounded by a formula the team can reason about.

The fatigue score for a user is an exponentially decayed count of recent sends, with type-specific decay constants. Social notifications decay faster than marketing, because the system wants to allow bursts of genuinely relevant social activity without being throttled by a slow marketing decay.

The penalty lambda is tuned against retention and opt-out guardrails, not against an offline metric. A common pattern is a simple control loop that monitors weekly opt-out rate: if opt-outs rise above a target, lambda increases, fewer marginal notifications pass the threshold, and the next week's data shows whether opt-outs recovered.

Training

Training runs nightly on the previous 28 days of joined decision and engagement logs. A full retrain runs weekly to rebake embeddings as new senders and content types enter the platform.

The hardest training problem is selection bias. The model only has labels for notifications that were sent. Anything filtered or dropped has no outcome label, so naive training teaches the model to predict well on the distribution of notifications the old policy liked, not on the distribution of all candidates. Left uncorrected, the new model just reinforces the old policy.

Two mitigations run in production. Inverse-propensity scoring reweights training examples by one over the probability the old policy would have sent them; candidates that the old policy sent often get smaller weights, candidates it sent rarely get larger ones.

The propensity estimate comes from a separate logged-propensity model. The second mitigation is epsilon-greedy exploration: a small fraction of candidates, say 1%, are sent or dropped randomly regardless of score. These exploration logs provide an unbiased sample on which offline counterfactual evaluation can be trusted.

Inference Pipeline

Real-time path: an event arrives, the eligibility filter applies hard constraints, surviving candidates enter feature fetch, the scorer runs, the decision policy blends heads and fatigue, and surviving candidates flow into channel and send-time routing. The entire path sits behind a request-response interface with strict latency SLAs.

Batch vs Real-Time

Not every candidate needs to be scored the moment it is produced. The system distinguishes three priority tiers. Immediate-send notifications like a direct message or a close-friend live-stream start score through the real-time path and fire within seconds.

Near-real-time notifications like comments on the user's own post score through the real-time path but get held in a per-user buffer for up to a few minutes so adjacent events can be coalesced into one. Digest-eligible notifications, like friend activity or algorithmic re-engagement nudges, get aggregated into a once-or-twice-daily digest delivered at the user's optimal hour.

The batched digest path runs as a scheduled scan: the system collects all digest-eligible candidates for each user over the past 24 hours, scores them in bulk, picks the top handful, and emits a single composite notification. The batched path amortizes feature fetches and model calls across many candidates for the same user, which cuts cost per candidate by an order of magnitude versus the real-time path.

Latency Budget

The digest path runs at a different SLA: the scheduled scan must complete within a 10-minute window before the delivery target time, which translates to minutes of per-user compute rather than milliseconds.

6.1 Send-Time Optimization

A notification sent at the right moment can outperform the same notification sent at a random time by 3x or more. The problem is how to pick the moment for each user without hand-tuning 500 million schedules.

The core idea is to model each user's activity as a probability distribution over hour-of-week. A two-dimensional histogram with 168 buckets, smoothed and normalized, captures when the user typically opens the app. The system then picks the next hour whose intersection with the allowed window (within the next delivery deadline, outside quiet hours) has the highest expected engagement.

For users with sparse history, the histogram is shrunk toward a cohort prior. Cohorts are segments defined by timezone plus a coarse persona (morning commuter, evening scroller, weekend-only). Shrinkage strength depends on how much direct history the user has: a user with 90 days of activity gets almost no prior influence; a three-day-old user is mostly the cohort prior.

A subtle issue: optimizing for send-time based on past opens has a self-fulfilling bias. If the system has only ever sent notifications at 8pm to a user, the histogram will show 8pm opens because that's when notifications arrived. \

The fix is to separate the histogram into organic-opens (app opens not preceded by a notification) and notification-triggered opens. Only organic opens inform the send-time model. That single change materially improves lift, because it captures true active hours rather than hours the system manufactured.

6.2 Fatigue Modeling and Frequency Capping

Fatigue is the constraint that keeps the system honest. Without it, the scorer is rewarded for finding ever more plausible reasons to interrupt the user, and the long-term cost of those interruptions shows up only weeks later in retention.

Three layers work together. A hard cap enforces a ceiling; non-transactional notifications above it are dropped regardless of score. A soft penalty, the fatigue term in the utility, makes each additional send on the same day less likely to pass the threshold. A population-level control loop adjusts the penalty weight lambda when opt-outs drift.

Type-specific decay handles the common failure mode where a burst of social activity (user's post goes viral) generates ten high-value candidates in an hour. Without decay separation, the fatigue term would suppress later candidates just because marketing notifications earlier in the day inflated the count.

With social fatigue decaying in about three hours and marketing in about twelve, a viral burst of real social content can still get through while a marketing reminder earlier in the day does not count against it much.

Daily caps also need a per-type floor. An absolute daily cap of five can starve a user of important notifications like a direct-message summary if marketing notifications consumed the budget earlier. In practice, each high-priority type reserves its own budget slice, and only marketing and low-priority notifications compete for the shared residual.

6.3 Multi-Channel Coordination

The same candidate event can be delivered as a push, as an email, or as an in-app inbox notification. Each channel has different economics. Push is high-attention but expensive in fatigue because it lights up the device screen. Email is low-attention but cheap in fatigue because the user processes it lazily. In-app inbox has zero interruption cost but only reaches users who open the app anyway.

The channel router treats each option as a separate candidate with its own predicted engagement and its own fatigue charge. For a given notification, the router computes expected utility per channel and picks the max that is above a channel-specific threshold.

Content type constrains the choice. A direct-message notification is push or nothing because batching a DM into an email defeats the point. A re-engagement nudge is almost always email or inbox, rarely push, because push opt-in is too valuable to burn on speculative outreach. These constraints are encoded as hard rules in the router, and the model learns the per-channel score within the rule's allowed set.

Cross-channel deduplication matters too. If the system already sent a push for an event, sending the same event as an email later is a near-zero-value action and a positive fatigue cost. The router maintains a short-horizon dedupe table keyed on (user, event, content-hash) so that the same logical notification is never delivered twice across channels.

6.4 Cold-Start and Exploration

A user in their first week has essentially no engagement history. The scorer cannot rely on user-sender CTR because there is no CTR. The active-hours histogram is empty. Every feature that aggregates the user's past is uninformative.

If the system defaults to low scores and sends little, the user never learns the app has useful notifications and churns. If it defaults to high scores and sends a lot, the user drowns in noise and opts out. Both failure modes are worse than the steady state.

The approach is cohort-based backoff plus conservative volume plus explicit exploration. Cohort features replace missing user-level features: cohort CTR by type, cohort active-hours histogram, cohort opt-out rate. Cohorts are defined by signup source, declared interests at onboarding, device type, and timezone. New users start with a conservative daily cap, typically half the steady-state cap, which relaxes over the first two weeks as the model gains signal.

An exploration slice, around 10% of new-user notifications, is sent based on diversified content rather than cohort score, which both broadens the signal the system collects on the new user and prevents the model from over-fitting to the cohort prior.

A parallel cold-start problem exists for new notification types. When a new event type launches, the model has not seen training labels for it, so its predictions are unreliable. The system gives new types a small exploration budget with hand-calibrated priors, captures the first few days of labeled engagement, and includes the new type in the next training cycle. Budget is expanded as calibration improves.

This pattern also protects against adversarial type introductions, like a new marketing category that would flood the system if given immediate access to the full scoring path.

Offline Evaluation

The offline suite checks three things: per-head discrimination quality, calibration, and counterfactual uplift. AUC per head catches regressions on individual tasks, and per-segment AUC catches regressions hidden inside the average (a 2-point drop on the small cohort of heavy users can hide behind a flat overall number).

Calibration plots matter more here than in most ranking problems because the utility blend adds absolute probabilities across heads; if P(unsubscribe) is systematically under-predicted by even a few percentage points, the penalty term is too small and the policy over-sends.

Counterfactual uplift estimation uses the logged exploration traffic. On those randomly-sent examples, the system computes an inverse-propensity-weighted estimate of how the new policy would have performed, which gives a defensible signal before touching production traffic. The estimate is noisy on small exploration fractions, so results are reported with confidence intervals and compared across multiple weeks before a new model is promoted to A/B.

Online Evaluation

Online evaluation runs A/B tests on hashed cohorts of users. The holdout cell receives notifications under the current policy; the treatment cell receives them under the new one. Primary metrics are 7-day and 28-day active retention, sessions per DAU, and a composite engagement score. Guardrails include unsubscribe rate, push-enabled user share, and spam complaint rate.

A quirk of notification experiments is that the treatment effect takes weeks to saturate. Unsubscribe is a cumulative decision; a user who would have unsubscribed after the 50th annoying notification takes weeks to get there in the treatment cell.

Retention is a lagging metric by construction. So experiments run for three to four weeks minimum, with a pre-committed decision rule that weighs short-term engagement against the longer-term guardrails. Stopping early on a positive engagement signal without letting unsubscribes and retention shake out is the most common mistake teams make here.

One more subtlety: holdout cells that receive zero notifications for long periods are contaminated because users in them churn faster, which biases estimates of the treatment's effect. Most teams use a baseline policy as the control rather than a no-notifications control for this reason.

Monitoring

The monitoring dashboard tracks four families of signals. Delivery health: per-channel success rate from APNs, FCM, and the email provider; a spike in failures usually means a channel outage. Model health: online per-head AUC computed from streamed labels, calibration residuals, and prediction distribution drift. Policy health: per-user notification count distribution, per-type volume, opt-out rate by cohort, fatigue-term distribution. System health: latency percentiles, feature-fetch timeouts, eligibility-filter drop rates.

A particularly useful signal is the distribution of how close utility scores are to the threshold. A healthy system has a thick middle section of candidates near the boundary; if the distribution collapses toward the edges, the threshold has drifted or the model has lost calibration, both of which precede a measurable retention regression by days to weeks.