{"title":"Design Spam Detection System","description":"","content":"Spam detection is the system that sits between a sender and a recipient and decides whether a message is wanted. Gmail filters roughly 15 billion messages a day, Meta moderates billions of comments and DMs, and Twitter/X screens a steady stream of bot traffic. Rules catch the obvious stuff, but the long tail of phishing, coordinated campaigns, and evolving obfuscation is an ML problem.\n\n---\n\n# 1. Problem Formulation\n\n### Clarifying Questions\n\n**Candidate:** \"Which surface are we protecting? Email inbox, public comments, private DMs, and marketplace listings all have different latency and false-positive tolerances.\"\n\n**Interviewer:** \"Focus on email. The system has to decide before the message lands in the user's inbox.\"\n\n**Candidate:** \"What counts as spam here? Unsolicited promotional mail, phishing, scams, and automated bulk messaging all look different.\"\n\n**Interviewer:** \"All of those. The product team wants per-category labels so downstream teams can take different actions. Phishing gets blocked outright, promotional spam goes to a spam folder, scams get a warning banner.\"\n\n**Candidate:** \"What's the scale?\"\n\n**Interviewer:** \"Around 10 billion messages a day, with peaks near 120K QPS. About 300 million active users. Roughly half the raw inbound traffic is spam before any filtering.\"\n\n**Candidate:** \"What's the latency budget per message?\"\n\n**Interviewer:** \"The classifier has to return a decision under 100ms p99. SMTP servers can buffer, but delays above a few seconds start to cause timeouts for legitimate senders.\"\n\n**Candidate:** \"How do we feel about false positives versus false negatives?\"\n\n**Interviewer:** \"False positives are much worse. A missed spam message is annoying. A legitimate business email that disappears can cost someone a job offer or a signed contract. Target less than 0.1% FP on legitimate mail, even if it means giving up some recall.\"\n\n**Candidate:** \"What labels can we rely on? User reports? Human review?\"\n\n**Interviewer:** \"A mix. Users mark things as spam or 'not spam'. We run a honeypot of seed accounts that should never receive real mail. A small rater team adjudicates contested samples. Assume labels are noisy and delayed, sometimes by hours or days.\"\n\n### Requirements\n\n**Functional Requirements:**\n\n- Classify each inbound message into categories: ham, promotional, phishing, scam, bulk-automated, abuse.\n- Return a calibrated confidence score per category so downstream surfaces can pick their own thresholds.\n- Emit a reason code (suspicious URL, sender reputation, content similar to known campaign) for transparency and appeals.\n- Consume user-report feedback and update models on a short cycle.\n- Support per-tenant overrides for enterprise customers that allow or block specific senders.\n\n**Non-Functional Requirements:**\n\n- p99 end-to-end decision latency under 100ms.\n- 99.99% availability. A filter outage means either everything is blocked or everything is delivered, both bad.\n- False positive rate under 0.1% on legitimate mail, measured weekly.\n- The system should degrade gracefully when a model is unavailable, falling back to a simpler classifier rather than failing open or closed.\n- Adversarial robustness. Attackers actively probe the system.\n\n### ML Problem Framing\n\nThe core task is multi-label text classification with heavy auxiliary signals. A message can carry more than one label, for example a phishing email that also uses promotional language. The output is a vector of calibrated probabilities per category, plus a top-level \"is this spam\" score used for the block/deliver decision.\n\nThree properties make this different from a standard text classification problem.\n\nFirst, class imbalance is real and asymmetric across categories. Ham dominates the post-filter stream, phishing is rare but catastrophic when missed. Naive sampling wrecks calibration. Training uses class-weighted loss and under-samples ham, then recalibrates on a held-out set that reflects production prevalence.\n\nSecond, concept drift is continuous. A phishing campaign that didn't exist yesterday can account for 5% of inbound volume by lunch. The model has to pick up new patterns in hours, not weeks. That rules out a monthly retrain cadence and forces a feedback loop that pushes fresh labels into training fast.\n\nThird, the label signal is delayed and biased. A user reporting a message as spam hours after delivery gives you a clean positive. A user who never opens a message gives you nothing definitive, even though the message might be spam. This biases the training set toward messages that were salient enough to get a reaction.\n\n### Metrics\n\n\n| Metric | Type | What It Measures | Why It Matters |\n|---|---|---|---|\n| PR-AUC (per category) | Offline | Area under precision-recall curve | Better than ROC-AUC under heavy class imbalance |\n| Precision at FP=0.1% | Offline | How many real spams we catch while keeping FP at the operating point | Directly mirrors the product constraint |\n| Recall on fresh spam (<24h old) | Offline | Catch rate on campaigns that emerged in the last day | Measures responsiveness to drift |\n| Category recall breakdown | Offline | Recall split by phishing, scam, promotional, bulk | Phishing recall matters more than promotional recall |\n| Calibration error (ECE) | Offline | Gap between predicted probability and empirical frequency | Thresholds only work if scores are calibrated |\n| User spam-report rate | Online | \"Report spam\" clicks per 1000 delivered messages | Direct proxy for missed spam |\n| False-positive complaint rate | Online | \"Not spam\" clicks per 1000 quarantined messages | Direct proxy for over-blocking |\n| Appeal reversal rate | Online | Fraction of appealed decisions that get overturned | Catches systemic bias in thresholds |\n| Time-to-catch on new campaign | Online | Minutes between first appearance and effective blocking | Measures how fast the feedback loop closes |\n| Per-cohort FP rate | Online | FP rate for small businesses, non-English senders, new domains | Catches fairness regressions |\n\n\n---\n\n# 2. System Architecture\n\nThe system is a filter pipeline. Messages arrive at the ingress, flow through a series of progressively more expensive checks, and each stage can make a final decision or punt to the next one. Cheap checks catch the obvious stuff. Expensive checks handle the ambiguous cases.\n\n\n```mermaid\nflowchart LR\n A[Inbound
SMTP / API]:::primary --> B[Pre-filter
Denylist + Hash]:::teal\n B --> C[Stage 1
Linear Model]:::orange\n C --> D{Early
Exit?}:::orange\n D -->|High confidence| H[Decision
Service]:::green\n D -->|Uncertain| E[Stage 2
Content Model]:::primary\n E --> F[Stage 3
Graph / Behavior]:::primary\n F --> H\n H --> I[Action
Router]:::green\n I --> J[Deliver]:::green\n I --> K[Quarantine]:::orange\n I --> L[Block]:::red\n I --> M[Challenge
CAPTCHA / SPF]:::orange\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe pre-filter is a straight lookup: known bad IPs, known bad content hashes, SPF/DKIM/DMARC validation, senders on the user's personal denylist. This drops 60-70% of inbound volume before any ML touches it. The linear model sees the remainder and makes fast decisions on clear cases. The content model, a transformer over the message body, runs only on the residual. The graph stage consumes sender-behavior features that are expensive to compute, so it often runs asynchronously and updates the decision post-delivery if needed.\n\nTraining runs on three separate loops, each tuned to a different kind of signal.\n\n\n```mermaid\nflowchart TD\n A[Labeled
Message Pool]:::primary --> B[Daily Full
Retrain]:::orange\n C[User Reports
+ Honeypot Hits]:::primary --> D[Hourly
Incremental]:::orange\n E[Campaign
Detection]:::red --> F[Emergency
Retrain]:::red\n\n B --> G[Model
Registry]:::teal\n D --> G\n F --> G\n G --> H[Shadow
Deploy]:::teal\n H --> I[Staged
Rollout]:::green\n I --> J[Production
Serving]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe daily loop trains on a large labeled corpus and produces a high-quality base model. The hourly loop fine-tunes on fresh user reports and honeypot signal, updating a lightweight adapter rather than the full model. The emergency loop kicks in when a campaign detector flags a sudden burst of similar content, pushing a hot-patched model in under an hour.\n\n### Scale Numbers\n\nThe back-of-the-envelope shapes most downstream design choices.\n\n\n| Quantity | Value | Notes |\n|---|---|---|\n| Messages inbound per day | 10B | Roughly half dropped by pre-filter |\n| Peak QPS | 120K | Morning business hours in the US |\n| Avg message size | 10 KB | Multipart with HTML + attachments dominates |\n| Storage per day (messages + features) | ~120 TB raw, ~40 TB after compression | Short retention for compliance |\n| Active users | 300M | Roughly 30M DAU per region at peak |\n| Sender reputation table | ~500M rows | IPs, domains, ASNs, auth-verified senders |\n| Feature store hot set | ~150 GB | Sender stats plus user preference features |\n| Model inference cost budget | <$0.0001 per message | At 10B/day that's $1M/day for ML compute |\n\n\n### Data Flow\n\nInbound messages hit an edge SMTP cluster that strips envelope metadata, runs cryptographic validation, and forwards structured events to a streaming bus. From there, two paths diverge. The online path forks into the inference pipeline. The offline path fans the event out to a data warehouse, a feature pipeline, and a labeling pipeline that collects user reports and honeypot hits.\n\n\n```mermaid\nflowchart LR\n A[Edge
SMTP / API]:::primary --> B[Event
Bus]:::teal\n B --> C[Online Path
Inference]:::orange\n B --> D[Offline Path
Warehouse]:::teal\n D --> E[Feature
Pipeline]:::primary\n D --> F[Labeling
Pipeline]:::primary\n E --> G[Feature
Store]:::orange\n F --> H[Training
Pool]:::orange\n G --> C\n H --> I[Training
Jobs]:::primary\n I --> J[Model
Registry]:::green\n J --> C\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe online path has to make a decision in milliseconds. The offline path can take hours or days, because it feeds training rather than serving. The two paths share the event bus, which means training data reflects exactly what the serving system saw. That matters for debugging. When a specific message gets misclassified in production, the training pipeline has access to the same feature values that serving used, not a recomputed version that might differ subtly.\n\n---\n\n# 3. Data\n\n### Data Sources\n\n\n| Source | What It Provides | Freshness | Notes |\n|---|---|---|---|\n| Raw message | Subject, body, attachments, MIME structure | Real-time | Encrypted at rest, limited retention |\n| Sender metadata | From address, IP, domain, ASN, SPF/DKIM/DMARC | Real-time | Cryptographically signed where possible |\n| Sender reputation | 7/30/90-day send volume, complaint rate, honeypot hits | Hourly | Keyed by IP, domain, authenticated sender ID |\n| Recipient history | Prior contact with this sender, reply rate, folder moves | Daily batch | Personal to each user |\n| Send graph | Who sent to whom over the last 90 days | Daily batch | Used for coordinated-campaign detection |\n| User reports | \"Mark as spam\" and \"not spam\" clicks | Real-time, biased | Treated as noisy positive and negative labels |\n| Honeypot accounts | Seed accounts that should never receive legitimate mail | Real-time, clean | High-precision positive signal |\n| Human rater adjudication | Gold labels on contested samples | 24-48h delay | Small volume, high quality |\n| External threat feeds | Known malicious URLs, phishing kit hashes, compromised domains | Hourly | Commercial and open-source feeds |\n\n\n### Labels\n\nLabels come from four streams, each with a different precision-coverage trade-off. User \"report spam\" clicks are high-volume but noisy. Users sometimes report legitimate mail they don't want anymore, like a forgotten newsletter. Honeypot hits are rare but clean. If a message lands in a seed account, it's spam with very high confidence. Human raters handle the contested middle, labeling a sampled set for training and evaluation. Implicit signals like \"deleted without opening\" are soft negatives at best, since plenty of ham also gets ignored.\n\nLabel delay is a real problem. A phishing campaign can run for three hours before enough users report it to train an emergency model. That's three hours where the campaign is active and the detector is blind. Honeypot seeding shortens this window because attackers often scrape leaked address lists that contain seed accounts. The first honeypot hit of a new pattern is often the earliest signal available.\n\nThe other subtlety is that labels aren't symmetric. A \"report spam\" click is a strong positive. A \"not spam\" click on a quarantined message is a strong negative. But the absence of either click is ambiguous. A user who opens a message and does nothing might have decided it's fine, or might not have noticed it was spam. The training pipeline treats unreported messages as weak signal, weighted down compared to explicit actions.\n\n### Feature Engineering\n\nFeatures break into five buckets. Content features describe the message itself. Sender features describe the sending entity. Recipient-relation features describe the history between sender and recipient. Graph features describe the sender's place in the broader send network. Temporal features capture bursts and seasonality.\n\n\n| Bucket | Example Features | Source | Freshness |\n|---|---|---|---|\n| Content | Token n-grams, TF-IDF, BERT embeddings, URL features, attachment hashes, language, subject-body agreement | Real-time | Computed per-message |\n| Sender | Domain age, authentication pass-rate, send volume 1h/24h/7d, complaint rate, honeypot hit rate, country-of-origin mismatch | Feature store | Hourly |\n| Recipient-relation | Prior message count, reply rate, last-contact days, user's personal allow/deny list | Feature store | Daily |\n| Graph | Sender fanout 24h, shared-IP cluster size, shared-content-hash cluster size, community ID from graph partitioning | Batch | Daily |\n| Temporal | Sender send-rate z-score vs baseline, cross-recipient arrival burstiness, hour-of-day vs historical pattern | Streaming | Minutes |\n\n\nURL features deserve their own treatment because URLs carry most of the phishing signal. Attackers use shortened URLs, redirect chains, homoglyph domains, and freshly registered domains to hide destinations. The feature pipeline expands short links, resolves redirect chains up to a depth limit, and checks the final destination against threat feeds.\n\n\n```python\nfrom urllib.parse import urlparse\nimport tldextract\n\ndef extract_url_features(url: str, threat_feed, registrar_age_lookup) -> dict:\n expanded = expand_redirects(url, max_hops=5)\n parsed = urlparse(expanded)\n ext = tldextract.extract(expanded)\n registrable = f\"{ext.domain}.{ext.suffix}\"\n\n return {\n \"redirect_hops\": count_hops(url, expanded),\n \"domain_age_days\": registrar_age_lookup.get(registrable, -1),\n \"has_homoglyph\": has_unicode_homoglyph(ext.domain),\n \"tld_category\": categorize_tld(ext.suffix),\n \"on_threat_feed\": registrable in threat_feed,\n \"path_entropy\": shannon_entropy(parsed.path),\n \"query_param_count\": len(parsed.query.split(\"&\")) if parsed.query else 0,\n \"punycode_host\": parsed.hostname.startswith(\"xn--\") if parsed.hostname else False,\n }\n```\n\n\nSender reputation is aggregated rolling stats keyed on the sending entity. The aggregation pipeline runs every five minutes for hot keys and hourly for the long tail.\n\n\n```python\ndef sender_reputation(sender_id: str, feature_store) -> dict:\n stats = feature_store.get(sender_id, [\n \"send_count_1h\", \"send_count_24h\", \"send_count_7d\",\n \"spam_report_count_7d\", \"honeypot_hits_30d\",\n \"unique_recipients_24h\", \"auth_pass_rate_7d\",\n ])\n\n return {\n \"send_rate_z_score\": z_score(stats[\"send_count_1h\"], stats[\"send_count_7d\"] / (7 * 24)),\n \"complaint_rate\": stats[\"spam_report_count_7d\"] / max(stats[\"send_count_7d\"], 1),\n \"fanout_24h\": stats[\"unique_recipients_24h\"],\n \"honeypot_flag\": stats[\"honeypot_hits_30d\"] > 0,\n \"auth_pass_rate\": stats[\"auth_pass_rate_7d\"],\n }\n```\n\n\nSample bias shows up in two places. First, the pre-filter drops the easy spam before training even sees it, so the model trains on the harder residual. Second, honeypot addresses never receive ham, so features learned from the honeypot stream can be systematically different from features of real users. The training pipeline handles both by maintaining a separate \"pre-filter\" training stream and by stratified sampling of real users for ham examples.\n\nA common failure mode in feature engineering is leakage between training and serving. A feature like \"sender complaint rate 7d\" has to be computed as of the message arrival time, not using future complaints. If the training pipeline reads the feature store as it is today and joins it against messages from last week, the features contain information from the future. The model then looks great offline and collapses in production. Point-in-time correctness is enforced by the feature store: every feature lookup carries a timestamp, and the store returns the value that was live at that moment.\n\n---\n\n# 4. Model\n\n### Baseline Approach\n\nA logistic regression on token n-grams plus handcrafted sender features gets you further than you'd think. It's fast, interpretable, and its weights are easy to audit when an appeal comes in. For a decade, production spam filters were essentially this with better feature engineering.\n\nThe features are token unigrams and bigrams from subject and body, binary flags for a curated list of spam tokens, URL features, sender reputation features, and a small set of interaction terms like \"new domain AND contains login URL\". The model uses L1 regularization to keep the weight vector sparse, which makes scoring fast and lets a human read the top contributing features for any decision.\n\nThe ceiling on this approach is paraphrase. Spammers rotate wording, swap synonyms, and pad with benign text. A model that only sees bag-of-tokens misses that two messages with different surface forms are saying the same thing. It also misses cross-sender patterns. If the same campaign is sent from a hundred compromised accounts, each with its own sending domain, each message looks unusual-but-plausible in isolation. The model needs to see the pattern across senders.\n\n### Advanced Approach\n\nProduction systems layer a cascade. Each stage is tuned to a different budget.\n\n\n```mermaid\nflowchart LR\n A[Message]:::primary --> B[Stage 1
Linear
~1 ms]:::teal\n B --> C{Score}:::orange\n C -->|p < 0.05| D[Deliver]:::green\n C -->|p > 0.95| E[Block]:::red\n C -->|Uncertain| F[Stage 2
Transformer
~25 ms]:::primary\n F --> G{Score}:::orange\n G -->|Confident| H[Decision]:::green\n G -->|Uncertain| I[Stage 3
Graph GNN
~80 ms async]:::orange\n I --> H\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nStage 1 is the same logistic regression described above, running in under a millisecond. It produces early-exit decisions for the clear cases on both ends of the score distribution, which covers roughly 80% of post-pre-filter traffic.\n\nStage 2 is a transformer encoder fine-tuned on message bodies, producing a contextual representation that feeds a classification head. The model sees subject, body, and selected metadata as a single sequence. Distillation from a larger teacher model keeps inference under 25ms on GPU batches. The encoder is warm-cached per language, because a message in Portuguese and a message in English hit different expert heads.\n\nStage 3 runs a graph neural network over the sender's local subgraph: other senders that share IP blocks, domains, content hashes, or recent recipients. It outputs a \"coordinated campaign\" signal that the linear and content models can't see on their own. This stage is often asynchronous. The decision service returns a preliminary verdict from stages 1 and 2, and the graph signal can reverse the decision within minutes if the sender's subgraph lights up.\n\nThe tradeoff between this cascade and a single monolithic model is cost. A single transformer large enough to match cascade recall would have to run on every message, which at 120K QPS and 25ms per inference is a lot of GPU. The cascade runs the expensive model only on the 15-20% of messages that stage 1 couldn't confidently resolve, cutting serving cost by roughly 5x at similar recall.\n\n\n```python\ndef classify(message, stage1, stage2, stage3_async, low=0.05, high=0.95):\n features = build_features(message)\n\n p1 = stage1.predict_proba(features)\n if p1 < low:\n return Decision(label=\"ham\", score=p1, reason=\"stage1_low\")\n if p1 > high:\n return Decision(label=\"spam\", score=p1, reason=\"stage1_high\")\n\n p2 = stage2.predict_proba(message.body, features)\n if p2 < low or p2 > high:\n stage3_async.enqueue(message.id, features)\n return Decision(label=\"spam\" if p2 > 0.5 else \"ham\", score=p2, reason=\"stage2\")\n\n p3_placeholder = p2\n stage3_async.enqueue(message.id, features, reverse_if_needed=True)\n return Decision(label=\"spam\" if p3_placeholder > 0.5 else \"ham\", score=p3_placeholder, reason=\"stage2_pending_graph\")\n```\n\n\n### Training\n\nThe daily training job trains the full cascade end-to-end. Stage 1 trains on the full labeled pool with L1-regularized logistic regression. Stage 2 fine-tunes a base encoder on a balanced sample. Stage 3 trains on sender subgraphs extracted from the last 90 days of send activity.\n\nClass weighting handles imbalance. With ham outnumbering confirmed spam 20 to 1 in the labeled pool, the loss is weighted so the gradient from a spam example counts 10-20x more. Hard-negative mining adds another lift. The pipeline sweeps the false-positive set from the previous model's predictions on held-out ham and uses those as high-weight negative examples in the next round. This sharpens the decision boundary where it matters.\n\nThe hourly incremental loop fine-tunes a lightweight adapter, typically a LoRA-style module on the transformer, using the last few hours of user reports and honeypot hits. It doesn't touch the base model, so rollback is a matter of swapping the adapter. If the incremental update regresses calibration or FP rate, the adapter gets reverted in under a minute.\n\nEmergency retraining kicks in when the campaign detector flags a sudden cluster. A content-similarity job running over the last hour of inbound mail identifies bursts of near-duplicate messages. If the cluster is unlabeled and lands in honeypots, it triggers an emergency retrain focused on that cluster.\n\n\n```mermaid\nflowchart TD\n A[Campaign
Detector]:::red --> B{Cluster
Size}:::orange\n B -->|Small| C[Log Only]:::teal\n B -->|Large + Honeypot Hit| D[Extract
Cluster Samples]:::primary\n D --> E[Hard Negative
Mining from Ham]:::primary\n E --> F[Fine-tune
Adapter]:::orange\n F --> G[Eval on
Holdout]:::teal\n G -->|FP Regression| H[Reject]:::red\n G -->|Passes| I[Canary
Deploy]:::green\n I --> J[Full
Rollout]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe emergency path has guardrails. Even in a rush, a hot-patched model has to clear a short offline evaluation that checks for FP regression on a small ham holdout. Without that check, an attacker could craft a campaign that looks enough like legitimate mail to poison an emergency retrain. The whole point of the cascade architecture is that adversaries can't easily steer the system into a bad place, and the emergency loop is the part most vulnerable to that, so it gets the tightest review.\n\n---\n\n# 5. Serving\n\n### Inference Pipeline\n\nA message entering the pipeline hits a fan-out of feature lookups first, then the cascade, then a decision service that applies policy on top of the raw model scores.\n\n\n```mermaid\nflowchart LR\n A[Ingress
SMTP / API]:::primary --> B[Feature
Orchestrator]:::teal\n B --> C[Redis
Sender Stats]:::orange\n B --> D[Feature Store
User Prefs]:::orange\n B --> E[Threat
Feed Cache]:::orange\n C --> F[Model
Cascade]:::primary\n D --> F\n E --> F\n F --> G[Decision
Service]:::primary\n G --> H[Policy
Engine]:::teal\n H --> I[Action
Router]:::green\n I --> J[Event
Logger]:::teal\n J --> K[(Feedback
Store)]:::orange\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe feature orchestrator fires lookups in parallel. Redis hosts the hot set of sender statistics, keyed by authenticated sender ID, domain, and IP. The offline feature store serves user preference vectors and per-tenant rules. The threat feed cache is a small local cache refreshed every few minutes from the main feed service.\n\nThe decision service layers policy on top of the raw scores. Enterprise customers can set a per-tenant sensitivity level. Individual users can allow-list a sender. Regulatory requirements in some regions require logging before blocking. The policy engine applies these rules after the model, so policy changes don't require model retraining.\n\n### Batch vs Real-Time\n\n\n| Component | Mode | Why |\n|---|---|---|\n| Content features | Real-time | Computed per-message, can't be precomputed |\n| URL expansion | Real-time with cache | Fresh URLs need live resolution, popular URLs cached |\n| Sender reputation stats | Near-real-time (5 min) | Aggregated from streaming send events |\n| User preference features | Batch (daily) | Stable, updated once per day |\n| Graph features | Batch (daily) | Expensive to compute, updated nightly |\n| Threat feed matches | Cached (hourly refresh) | External feeds update hourly |\n| Model weights | Batch push | Daily base, hourly adapter, emergency as needed |\n\n\nThe key separation is between features that describe the message (real-time) and features that describe the sender's behavior over time (near-real-time or batch). A new sender that just started blasting spam has no reputation features yet, so the model has to rely on content signals for the first few minutes.\n\n### Latency Budget\n\n\n| Component | Budget | Notes |\n|---|---|---|\n| Pre-filter (denylist + hash) | 2ms | In-memory, no ML |\n| Feature fetch (parallel) | 10ms | Redis + feature store + threat cache |\n| Stage 1 linear model | 3ms | CPU, vectorized |\n| Stage 2 transformer (if invoked) | 25ms | GPU batch, ~15-20% of traffic |\n| Decision + policy engine | 5ms | Deterministic logic |\n| Network + serialization | 10ms | Internal RPCs |\n| **Total p99** | **~55ms (cascade early exit) / ~80ms (full)** | Well under the 100ms budget |\n\n\nStage 3 graph features are kept off the synchronous path on purpose. Waiting for graph traversal inside the request would blow the latency budget. Instead, the decision can be revised asynchronously once the graph signal arrives, which is fine for email because delivery happens into folders that can be moved after the fact.\n\nGraceful degradation matters more here than in most ML systems because the failure modes are both visible and bidirectional. If the feature store is down and the system falls back to \"deliver everything\", spam floods users. If the model service is down and the system falls back to \"quarantine everything\", legitimate mail disappears. The chosen fallback is to preserve the pre-filter and stage-1 linear model on cached weights, skip stages 2 and 3, and raise the operating threshold. This is conservative on FPs at the cost of lower recall, which matches the FP tolerance profile. An alert fires on any extended fallback so the service owner knows the system is running with diminished capability.\n\n---\n\n# 6. Deep Dives\n\n### Adversarial Robustness\n\nSpammers are not a stationary distribution. They probe the filter, learn what gets through, and adjust. This is the deep dive that shapes more of the system than any other.\n\nCommon obfuscation techniques include Unicode homoglyphs where a Latin 'a' gets replaced with a Cyrillic 'а', zero-width characters inserted to break token matching, image-only content to dodge text classifiers, URL shorteners and redirect chains to hide malicious destinations, base64 or HTML-entity encoded payloads, and content hidden in MIME parts that clients render but naive parsers skip.\n\nDefenses are layered. Text normalization maps homoglyphs to a canonical form before tokenization, strips zero-width characters, and decodes HTML entities. Image attachments flow through an OCR pipeline that extracts text, and the OCR output feeds the same content model. URL expansion follows redirect chains up to a depth of five hops and classifies the final destination. MIME parsers are tuned to match client rendering behavior so the model sees what the user actually sees.\n\nThe harder defense is against adversaries who tune their content specifically to the model. Attackers with enough volume can treat the filter as an oracle: send variations, see which land in spam versus inbox, iterate. The countermeasure is a rate limit on the signal that adversaries can extract. The decision service adds noise to feedback when it's returned to senders (for instance, in NDR messages) and randomizes decisions near the threshold boundary. A small fraction of stage-1 decisions get elevated to stage 2 even when stage 1 is confident, which makes the oracle noisy.\n\nCanary traps help catch adversaries early. A small pool of addresses in the honeypot receive different filtering treatments. If an attacker's volume drops sharply only on the treatment that blocked them, the filter's decisions are leaking. That's a signal to rotate thresholds and retrain.\n\n### False Positive Cost and Threshold Tuning\n\nFalse positives aren't uniform. Blocking a promotional email the user didn't really want is a small cost. Blocking an order confirmation from a customer's bank is a much bigger cost. Threshold tuning has to reflect that.\n\nThe system uses per-surface thresholds. Phishing gets the strictest threshold because even a small number of delivered phishing messages causes real harm. Promotional spam gets a looser threshold because the failure mode is a missed coupon. The decision service maps category scores to actions through a policy table that's separate from the model.\n\nPer-user thresholds also matter. A new account with no history is more likely to see FPs because the model has less personal context. A long-time user with a rich history of prior senders and clear folder preferences can tolerate a more aggressive threshold because the prior-contact signal is strong. The thresholding layer shifts based on user maturity and activity.\n\nQuarantine versus hard-block is the other axis. A hard block returns a 5xx to the sending server, which bounces. Quarantine places the message in a spam folder the user can check. Hard blocks should be reserved for cases where false positives are recoverable (because the sender sees the bounce and can retry or contact support) and where delivery would cause outsized harm (phishing with credential theft links). Everything else gets quarantined.\n\nThe appeals path closes the loop. A user who marks a quarantined message as \"not spam\" updates their personal allow-list and also emits a training signal. An enterprise admin can override system-wide decisions for their tenant. Appeal reversal rate is a first-class metric, and a rising reversal rate is the fastest signal that a new model has drifted out of calibration.\n\n\n```mermaid\nflowchart LR\n A[Model
Score]:::primary --> B[Category
Router]:::teal\n B --> C{Phishing?}:::orange\n C -->|Yes, p>0.9| D[Hard Block]:::red\n C -->|Yes, 0.7-0.9| E[Warning
Banner]:::orange\n B --> F{Scam?}:::orange\n F -->|Yes, p>0.85| G[Quarantine]:::orange\n B --> H{Promotional?}:::orange\n H -->|Yes, p>0.7| I[Promotions
Tab]:::teal\n B --> J[Deliver
Inbox]:::green\n K[User
Action]:::teal --> L[Personal
Rules]:::orange\n L --> M[Future
Override]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe routing decision uses raw scores, per-category thresholds, and user-level overrides. A user who consistently rescues newsletters from the promotions tab teaches the personal rules engine to lower the threshold for that specific sender. This layered approach keeps policy changes cheap: adjusting a threshold or adding a per-tenant rule doesn't require retraining.\n\n### Graph Signals and Spam Rings\n\nContent classifiers can be fooled. Coordinated campaigns are harder to hide because the pattern of who sends to whom leaves a structural signature.\n\nA spam ring typically shows up as a cluster of senders that share one or more of: IP blocks, autonomous system numbers, sending infrastructure (same MTA fingerprint), content similarity (near-duplicate bodies with cosmetic variation), or recipient overlap (same list of addresses hit by many senders in a short window).\n\nThe graph pipeline builds a weighted sender-sender graph updated nightly. Edges are weighted by how much the two senders share. A connected-component analysis pulls out candidate rings. A graph neural network trained on labeled rings produces a \"ring-likelihood\" score per sender that feeds the cascade.\n\n\n```mermaid\nflowchart TD\n A[Sender A
IP 10.0.0.5]:::primary\n B[Sender B
IP 10.0.0.5]:::primary\n C[Sender C
IP 10.0.0.6]:::primary\n D[Sender D
IP 10.0.0.7]:::primary\n H1[Content
Hash X]:::orange\n H2[Content
Hash Y]:::orange\n R1[Recipient
Cluster 1]:::teal\n\n A --- H1\n B --- H1\n C --- H2\n D --- H2\n A --- R1\n B --- R1\n C --- R1\n D --- R1\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nThe diagram shows four senders linked through shared content hashes and a common recipient cluster. A single sender in that set looks plausible in isolation. The cluster, hitting the same recipients with near-identical content from a tight IP range, does not.\n\nContent hashing uses SimHash or MinHash rather than exact hashes because spammers perturb content with random padding. SimHash produces a locality-sensitive fingerprint where small edits produce small Hamming-distance differences. Two messages with a SimHash Hamming distance under 5 on a 64-bit fingerprint are near-duplicates for practical purposes.\n\nThe tradeoff with graph signals is freshness. A brand new sending infrastructure has no graph history, so graph features are uninformative for the first hours. Content and sender-behavior features have to carry the first burst of a new campaign until the graph fills in.\n\n### Label Feedback Loop and Concept Drift\n\nSpam evolves faster than most ML problems. A classifier that works today can miss 30% of the next campaign, and the next campaign might launch tomorrow. The feedback loop is the part of the system that has to keep up.\n\nThe loop has three streams feeding it. User reports are high-volume but noisy and biased. Users report things they find annoying, not only things that are spam in the strict sense. Honeypot hits are low-volume but clean. Human raters are lowest volume and highest quality, and handle cases where the user-report signal disagrees with the honeypot signal.\n\n\n```mermaid\nflowchart LR\n A[Delivered
Message]:::primary --> B[User
Action]:::teal\n B --> C{Action}:::orange\n C -->|Report spam| D[Noisy
Positive]:::red\n C -->|Not spam| E[Noisy
Negative]:::green\n C -->|Ignore| F[Soft
Signal]:::teal\n G[Honeypot
Hit]:::red --> H[Clean
Positive]:::red\n I[Rater
Review]:::orange --> J[Gold
Label]:::orange\n D --> K[Label
Fusion]:::primary\n E --> K\n F --> K\n H --> K\n J --> K\n K --> L[Training
Pool]:::teal\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nLabel fusion combines signals using a simple precedence. Honeypot and rater labels override user labels when they conflict. Multiple user reports for the same message carry more weight than one. Reports against a sender the user has previously corresponded with are treated with suspicion, since a legitimate sender getting marked as spam is often a user who just unsubscribed mentally.\n\nSelection bias is the hardest part. The training pool is biased toward messages users actually saw. Messages that were blocked outright never get reported, so the model trains on what the previous model couldn't confidently block. The pipeline corrects for this with a small holdout where a random sample of predicted-spam messages is delivered anyway, generating ground truth for the high-confidence region. This is a direct cost in FPs, but without it the model drifts into a bubble.\n\nShadow models address deploy risk. Any new model runs in shadow mode for a few hours before taking any traffic. Shadow mode scores every request but never changes the decision. The offline comparison between shadow and live predictions flags whether the new model is going to regress FP rate before users feel it. A staged rollout follows: 1%, 5%, 25%, 100% over 24-48 hours with automated rollback on any guardrail breach.\n\n---\n\n# 7. Evaluation and Iteration\n\n### Offline Evaluation\n\nOffline evaluation uses a temporal split. Random splits over time leak future information into training, which matters a lot here because attacker behavior changes week to week. The split holds out the last 7-14 days of labeled data and evaluates on that.\n\nKey offline metrics: PR-AUC on the held-out window for each category, precision at the operating FP rate (typically 0.1%), recall on messages first seen less than 24 hours before the label, and calibration error across score buckets. A model that improves PR-AUC but loses calibration is not a good trade, because downstream surfaces rely on the probabilistic interpretation of scores to set their own thresholds.\n\nBacktests against specific campaigns add qualitative signal. The evaluation pipeline keeps a library of known past campaigns and measures how fast the candidate model would have caught each one given the same label feedback loop. If the candidate is slower than the current production model on historical campaigns, something's wrong even if aggregate metrics look good.\n\n### Online Evaluation\n\nOnline evaluation runs A/B tests with a twist. Standard A/B splits users. For spam, splitting users biases the treatment's label feedback loop because half the user reports go to one model and half to the other. The system uses a dual-delivery design instead. A small fraction of the message stream gets scored by both models. When scores disagree, both decisions are logged but only one takes effect. User-report feedback on the delivered outcome updates labels for both models.\n\nPrimary online metrics: user spam-report rate per thousand delivered messages, FP complaint rate per thousand quarantined messages, time-to-catch on new campaigns. Guardrails: per-cohort FP rate (especially for small businesses, new domains, non-English senders), appeal reversal rate, and end-to-end latency p99.\n\nAn online test runs for at least 7 days to span weekly seasonality. Campaign cycles often last a few days, so a 3-day test can miss the regression that a new campaign would trigger.\n\nOne practical challenge with dual-delivery is that user reports on disagreement cases have to be attributed carefully. If model A said spam and model B said ham, and the message got delivered, a later \"report spam\" click credits both models with a label, but only model A got the decision right in real time. The evaluation pipeline tracks both the score agreement and the eventual label to separate \"the new model is better\" from \"the new model just got lucky on this cohort\".\n\n### Monitoring\n\nProduction monitoring runs at a faster cadence than evaluation. The system watches score-distribution drift minute-by-minute using population stability index (PSI) between the current hour and a rolling baseline. Feature-level drift runs per-feature and flags when a feature's distribution shifts significantly. A sudden jump in the \"domain age\" feature median, for example, often means an attacker registered a new batch of domains.\n\n\n| Signal | Threshold | Action |\n|---|---|---|\n| Score PSI vs 7-day baseline | >0.2 | Page on-call, investigate |\n| Any feature PSI vs 7-day baseline | >0.3 | Alert, review feature pipeline |\n| Per-category catch rate drop | >15% week-over-week | Trigger investigation, consider emergency retrain |\n| FP complaint rate | >2x baseline | Auto-revert to previous model version |\n| Appeal reversal rate | >10% of appeals | Raise operating threshold |\n| Latency p99 | >150ms | Page on-call |\n| Graph feature staleness | >48h | Alert data platform team |\n| Honeypot catch rate | <80% | Emergency retrain |\n\n\nHoneypot catch rate is the fastest early warning. Honeypot addresses should never receive ham, so anything less than near-100% catch rate on honeypot traffic means the model is missing real spam. A drop from 99% to 90% is a five-alarm fire.\n\nPer-cohort monitoring catches fairness regressions that aggregate metrics hide. A new model might improve overall precision while regressing on small-business domains or non-English senders. The dashboard breaks FP rate by sender country, sender domain age bucket, language, and authentication status. A regression on any bucket triggers a review even if top-line metrics look clean. Fairness isn't only an ethical concern here, it's a product concern: a filter that systematically junks legitimate mail from Brazilian small businesses will lose those users, and that loss won't show up in global metrics for weeks.\n\nThe iteration cadence combines the daily full retrain, the hourly incremental adapter, and the emergency retrain hook. Daily retrains ship behind shadow mode and a staged rollout. Hourly adapters ship with a fast-revert, since they're small deltas. Emergency retrains bypass the staged rollout only when the campaign detector fires on a cluster that's both large and landing in honeypots, which is rare but time-critical.\n\n---\n\n# Summary\n\n- Spam detection is multi-label text classification layered with behavioral and graph signals, operating under tight latency and FP budgets on billions of messages per day.\n- A cascade architecture (fast linear → transformer content model → async graph GNN) cuts serving cost roughly 5x versus a monolithic model at similar recall, because the expensive stages see only the ambiguous residual.\n- False positive cost drives threshold design. Targets under 0.1% FP on legitimate mail, with per-surface and per-user thresholds to reflect different failure costs across phishing, scam, and promotional categories.\n- Concept drift requires three training loops running at different cadences: daily base retrain, hourly incremental adapter, and emergency retrain when campaign detection fires on honeypot-hitting clusters.\n- Graph signals beat content alone for coordinated campaigns, because individual spammer accounts look plausible but sender-subgraph structure (shared IPs, SimHash clusters, recipient overlap) reveals the ring.\n- Label feedback is delayed and biased. Honeypot seeds give the cleanest early signal, user reports give volume with noise, and a small randomized delivery holdout corrects the selection bias in predicted-spam messages.\n- Adversarial robustness is continuous work, not a one-time fix. Text normalization, URL expansion, image OCR, randomized threshold jitter, and canary traps make the system expensive to probe.\n- Monitoring hinges on honeypot catch rate and score-distribution PSI. Those two signals catch most production regressions faster than any aggregate metric.","pageType":"ml-system-design"}

Design Spam Detection System

Ashish Pratap Singh

Get Premium