Every ML model is only as good as the data it’s trained on.

You can have a strong architecture, well-tuned hyperparameters, and plenty of compute, but if the training data doesn’t match real-world conditions, performance won’t hold up in production.

Data collection isn’t just pulling in logs. It’s a deliberate process that shapes everything downstream, from feature engineering to evaluation.

Before collecting anything, you need to understand what kinds of data exist and what each is good for. Training data for ML systems generally falls into five categories: implicit feedback, explicit feedback, system-generated data, third-party data, and synthetic data.

Implicit Feedback

Implicit feedback is what users do without being asked. Clicks, page views, scroll depth, dwell time, purchases, add-to-cart actions, search queries, video watch time. This is the bread and butter of most production ML systems because it's abundant and cheap to collect.

YouTube's recommendation system, for example, trains primarily on implicit signals. Watch time is the core signal, but the system also tracks whether a user clicked a video and left within seconds (a negative signal), whether they watched to completion, and whether they shared or added it to a playlist.

The volume advantage is massive. Netflix collects billions of implicit interaction events per day. Asking users to rate every show they watch would yield maybe a few million ratings per day. Implicit signals outnumber explicit ones by orders of magnitude.

The downside is noise. A user might click on a video by accident, leave a page open while getting coffee, or scroll past content they actually liked but didn't engage with. Implicit feedback tells you what users did, not what they wanted.

Explicit Feedback

Explicit feedback is what users tell you directly: star ratings, thumbs up/down, reviews, survey responses, report-as-spam actions. It's higher quality per data point because it captures user intent, not just behavior.

The problem is volume. Fewer than 5% of users typically leave explicit feedback. This creates a selection bias: users who rate content tend to be either very satisfied or very dissatisfied. The silent majority in the middle is underrepresented. Yelp, for instance, has long dealt with the "J-curve" in restaurant ratings, where most reviews cluster at 1 star or 5 stars.

In practice, production systems combine both. Spotify uses listening history (implicit) alongside thumbs up/down on Discover Weekly (explicit) to train its recommendation models. The implicit data provides coverage and volume, while the explicit data anchors ground truth for evaluation.

System-Generated Data

Not all training data comes from users. Server logs, latency measurements, error rates, CPU utilization, and network traffic patterns are valuable for models that monitor infrastructure or detect anomalies.

Fraud detection systems at companies like Stripe and PayPal rely heavily on system-generated data: transaction metadata, IP addresses, device fingerprints, velocity patterns. These signals are often more predictive than user-provided information because they're harder to fake.

Third-Party Data

Sometimes you need data you can't collect yourself. Weather data for a delivery time prediction model. Demographic data for an ad targeting system. Financial market data for a trading model. Third-party data fills gaps in your own collection.

The trade-offs are cost, freshness, and control. You're dependent on the provider's quality, update frequency, and continued availability. LinkedIn's economic graph combines internal professional data with third-party company information (funding rounds, headcount) to power features like job recommendations.

Synthetic Data

When real data is scarce, expensive, or privacy-constrained, synthetic data offers an alternative. This is artificially generated data that mimics the statistical properties of real data.

Waymo generates millions of synthetic driving scenarios to train autonomous vehicle models, covering edge cases (pedestrian jaywalking in rain at night) that would take years to encounter in real driving. Tesla uses a similar approach for rare crash scenarios.

Synthetic data works well for augmenting underrepresented classes, pre-training before fine-tuning on real data, and testing system robustness. It works poorly when the synthetic distribution diverges from real-world distribution, which is the fundamental risk.

The following table summarizes the trade-offs across data types:

Knowing what data types exist is one thing. Actually capturing them reliably at scale is a different engineering problem entirely. A poorly designed logging system creates gaps, inconsistencies, and schema drift that poison downstream models.

What to Log

The instinct is to log everything. Don't. Logging everything is expensive, creates storage headaches, and most of it becomes noise. Instead, start from your ML problem formulation and work backward. If you're building a search ranking model, you need queries, impressions, clicks, and click positions. You don't need every CSS rendering event.

A useful framework is to think about logging at three levels:

The system state level is easy to forget but critical. If you don't log which model version served a prediction and what candidates were ranked, you can't do counterfactual evaluation later. Pinterest logs the full ranked list (not just what the user saw) for every recommendation request, which enables offline experiments comparing new ranking models against the production model's logged decisions.

Event Schema Design

A well-designed event schema prevents months of data cleaning later. Every event should capture a consistent set of fields that make it joinable with other events and usable for ML.

A typical event schema looks like this:

A few principles that save pain down the road:

Logging Architecture

The path from user action to training data involves several components, each with its own reliability and latency requirements.

The event collector sits between the client and the rest of the pipeline. Its job is to accept events at high throughput, validate schemas, and forward them. Companies like Uber use a custom event collection service that handles 1M+ events per second.

The message queue (typically Kafka) decouples producers from consumers and provides durability. If the downstream processors go down temporarily, events are buffered rather than lost. This is important because lost training data can introduce selection bias: if your logging system drops events during peak traffic, your model underrepresents high-traffic scenarios.

From there, events flow into either a stream processor for real-time features or a batch pipeline that loads data into a warehouse for training. The training pipeline pulls from the warehouse, not from the raw event stream, because training needs stable, consistent snapshots of data.

Client-Side vs Server-Side Logging

Where you place the logging instrumentation matters more than you might expect.

In practice, you need both. Server-side logging gives you reliable records of what was served and predicted. Client-side logging tells you what actually happened in the user's experience: did they see the recommendation, how long did they look at it, did they scroll past it?

Airbnb logs search results server-side (which listings were ranked and in what order) and engagement client-side (which listings the user viewed, hovered over, or clicked). Joining these two streams creates the full picture needed for training a search ranking model.

Every ML system faces the same chicken-and-egg problem at launch: you need data to train a model, but you need a model (or at least a product) to collect data. This isn't just a startup problem. It resurfaces every time you launch a new feature, enter a new market, or onboard a new user segment.

Bootstrapping Strategies

There's no single solution to cold start. The right approach depends on how much you can afford to be wrong early on and how quickly you need the model to improve.

Transfer learning works when a related domain has abundant data. A new e-commerce site in Brazil can fine-tune a product recommendation model pretrained on a similar US marketplace. The model won't be perfect out of the gate, but it will outperform random recommendations on day one. GPT-style foundation models have made this even more accessible: fine-tuning a pretrained language model on a small domain-specific dataset is now a standard approach for NLP tasks.

Rule-based systems as data generators is an underrated strategy. Before Uber had enough ride data to train an ETA prediction model, they used simple heuristics (distance / average speed for the region) to provide initial estimates. These heuristic predictions, paired with actual trip durations, became the first training set for the ML model. The key insight is that the rules don't need to be accurate. They need to be good enough to keep users engaged while the system collects real data.

Exploration strategies like epsilon-greedy or Thompson sampling intentionally show users diverse content rather than always exploiting the current best guess. This sacrifices short-term quality for long-term data diversity. DoorDash uses exploration in new markets to collect data on restaurant preferences across different neighborhoods, even if it means occasionally recommending a restaurant that wouldn't be the top choice.

Public and open-source datasets provide a starting point for pre-training. ImageNet for vision tasks, Common Crawl for language tasks, MovieLens for recommendation experiments. The model won't work perfectly on your specific domain, but it gives you a reasonable initialization to fine-tune once your own data starts flowing.

How Long Does Cold Start Last?

It depends on traffic volume and the complexity of the task. A simple click-through rate model might need a few hundred thousand impressions to become useful, which a moderate-traffic site can collect in days. A personalized recommendation system that needs to learn individual user preferences might take weeks or months to reach acceptable quality for long-tail users.

The practical answer: plan for 2-4 weeks of degraded model quality after launch. Set explicit thresholds (e.g., "switch from rules to ML when we have 500K labeled examples") and monitor the transition closely.

Data collection doesn't happen in a vacuum. Regulations like GDPR (EU), CCPA (California), and LGPD (Brazil) impose strict rules on what you can collect, how long you can keep it, and what you must do when users request deletion.

This isn't a legal formality. These regulations have direct engineering consequences for ML systems.

Key Regulations at a Glance

Engineering Implications

Right to deletion is the hardest one for ML systems. When a user requests their data be deleted, you must remove their data from your databases, your data warehouse, your feature store, and potentially from models trained on their data. Retraining a model from scratch without one user's data is often impractical at scale.

The common engineering solution is to design for deletion from the start:

Consent management affects what data you can collect in the first place. Under GDPR, you need explicit consent for each category of data processing. A user who consents to their click data being used for "improving search results" hasn't consented to that same data being used for "targeted advertising." Your logging infrastructure needs to tag events with the consent categories they fall under.

Data retention policies set time limits on how long you can keep data. If your policy says 12 months, your training pipeline must exclude data older than 12 months. This is actually beneficial for model quality in many cases, since very old interaction data often reflects outdated user preferences and product states.

Anonymization vs pseudonymization. Anonymization removes all identifying information and is irreversible. Pseudonymization replaces identifiers with tokens that can be reversed with a key. GDPR treats these differently: truly anonymized data falls outside GDPR's scope entirely, while pseudonymized data is still personal data. For ML training, pseudonymization is usually sufficient and more practical, since you still need to join events by user for feature engineering.

The most powerful advantage an ML system can have isn't a better algorithm. It's a self-reinforcing cycle where the product generates the data that makes the model better, which makes the product better, which attracts more users and generates more data.

How Flywheels Work in Practice

Google Search is the canonical example. More users means more search queries. More queries means better understanding of what results are relevant (through click patterns). Better relevance means users prefer Google over alternatives, which brings more users. Google processes over 8.5 billion searches per day, and every single one generates training signal.

Spotify's Discover Weekly follows the same pattern. User listening data trains the recommendation model. Better recommendations increase listening time. More listening time generates richer preference signals. This creates a compounding advantage: the model gets better for each user as they listen more, and better across all users as the user base grows.

Accelerating the Flywheel

Not all data is equally valuable. A few strategies help the flywheel spin faster:

When Flywheels Stall

Flywheels can stall for several reasons:

Feedback loop bias

The model only learns from data it generates. If a recommendation model never shows certain types of content, it never learns whether users would like that content. Over time, the model becomes increasingly confident in a narrow slice of the content space and increasingly ignorant about everything else. This is the "filter bubble" problem.

Diminishing returns

After a certain scale, more data provides marginal improvement. Going from 1M to 10M training examples might increase accuracy by 5%. Going from 100M to 1B might increase it by 0.1%. The flywheel still turns, but the returns per revolution shrink.

Data quality degradation

As a platform grows, the proportion of spam, bot traffic, and adversarial behavior tends to increase. If you don't invest in data quality alongside data quantity, the flywheel can actually hurt model quality. This is why data validation (covered in a later chapter) is essential infrastructure.

The fix for stalling flywheels is usually deliberate exploration: introducing randomization or diversity into what the model serves, even at the cost of short-term engagement. Twitter/X's "For You" feed deliberately mixes in content from outside a user's typical engagement pattern to prevent the recommendation model from collapsing into a narrow echo chamber.