A model with 95% test accuracy can still fail in production.

Latency spikes, stale predictions, or broken data pipelines aren’t problems with the model itself. They’re failures in the system around it.

ML system design focuses on that broader picture. It’s about building reliable pipelines, serving infrastructure, and monitoring, so the model works correctly in real-world conditions, not just in offline evaluation.

Model selection, hyperparameter tuning, and improving accuracy on a held-out test set are important work. But they represent a surprisingly small fraction of what it takes to run ML in production.

A widely cited observation from Google's engineering team puts it bluntly: ML code, the part that trains and runs the model, accounts for roughly 5-10% of the total code in a production ML system. The other 90% is everything else. Data collection and validation, feature extraction and management, serving infrastructure, monitoring, configuration, testing, and the glue code that ties it all together.

Here's what a production ML system actually looks like:

The model lives inside the training box. Everything outside it, data pipelines, serving systems, monitoring, is what makes it useful in the real world.

A model can achieve state-of-the-art accuracy and still fail if it cannot serve predictions under 100 ms, handle high traffic, or adapt when user behavior changes. At that point, it is not a production system. It is just a research experiment.

This is the key mindset shift. ML system design is not about choosing the right algorithm. It is about designing the entire system that makes the model work reliably at scale.

Consider YouTube's recommendation engine. Yes, there's a deep learning model at its core.

But the system also includes data pipelines that process billions of user interactions daily, a feature store that serves hundreds of features per request in single-digit milliseconds, a multi-stage ranking pipeline that narrows millions of candidate videos down to a few dozen, A/B testing infrastructure that evaluates whether a new model actually improves watch time, and monitoring systems that detect when recommendations go off the rails.

The model is one component among many, and arguably not even the hardest one to get right.

Every production ML system goes through four major phases. Understanding this lifecycle matters because each phase comes with its own challenges, tools, and failure modes.

These phases form a continuous loop, not a one-time pipeline. In production, you don't build a model and walk away. You collect data, train, deploy, monitor, and then use what you learn from monitoring to improve the next iteration. This loop runs indefinitely for as long as the system is in production.

Data

Everything starts with data. You need to collect it, clean it, validate it, and transform it into features the model can use. This is usually the most time-consuming part. In practice, most of the work in ML is data work.

The challenge is not just getting data. It's getting the right data, in the right format, at the right time, and making sure it stays reliable. A recommendation system, for example, depends on user click logs, watch histories, search queries, and item metadata. Each comes from different sources, arrives at different times, and brings its own issues such as missing values, duplicates, delays, or schema changes.

Feature engineering turns raw data into signals the model can learn from. A raw timestamp is not useful on its own, but "hours since the user's last session" or "day of the week" might be highly predictive. Feature engineering is where domain knowledge meets data, and it's often where the biggest accuracy gains come from.

Training

Training is more than calling model.fit(). You need to split data correctly (avoiding data leakage), select appropriate evaluation metrics, track experiments so you know which configuration produced which results, and validate that the new model is actually better than what's currently in production.

Production training pipelines run on a schedule. A fraud detection model might retrain every few hours to catch emerging patterns. A recommendation model might retrain daily or weekly. The pipeline needs to be automated, reproducible, and resilient to failures like missing data or infrastructure issues.

Serving

This is where the model meets users. Serving is the process of taking a trained model and making its predictions available to applications, whether that's a search results page, a fraud alert, or a content feed. It needs to happen fast (often under 100ms), reliably (99.9%+ uptime), and at scale (thousands to millions of requests per second).

There are two main serving patterns. Batch inference precomputes predictions for all users or items ahead of time, stores the results, and serves them from a lookup table. It's simple and cheap, but predictions can be stale. Real-time inference runs the model on demand when a request comes in. It's fresh but more complex and expensive. Most production systems use a combination of both.

Monitoring

Monitoring is the phase most teams underinvest in, and it's where things silently break. Unlike traditional software, where a broken feature usually throws an error, a degrading ML model can keep serving predictions that are technically valid but increasingly wrong.

You need to monitor the model's prediction quality over time (are click-through rates dropping?), the distribution of input data (has user behavior shifted?), and operational health (latency, throughput, error rates). When monitoring detects a problem, it feeds back into the data phase, triggering a retraining cycle or a deeper investigation.

The following table summarizes each phase:

Model building starts with a clean dataset, tries different algorithms, optimizes a metric, and submits the best result. It's a valuable skill, but it solves a different problem than ML system design.

Model building asks: "Given this data, what's the best model?" ML system design asks: "How do I build a reliable, scalable system that uses ML to solve a business problem?"

The difference shows up in what you think about:

A concrete example: in a Kaggle competition for click-through rate prediction, you might spend days engineering features from the provided dataset and testing different gradient-boosted tree configurations. You never think about where the data comes from, how fast the model needs to run, what happens when the data distribution shifts, or how to roll out the model safely.

In an ML system design interview for the same problem, you'd need to address all of those questions. How do you collect click logs at scale? How do you compute features in real-time vs batch? How do you serve predictions in under 50ms? How do you detect when user behavior shifts and your model's accuracy degrades? How do you run A/B tests to validate that a new model actually improves revenue? The model architecture might take 5 minutes of the conversation. The system around it takes the other 40.

ML system design interviews evaluate a different set of skills than coding interviews or traditional system design interviews. Understanding what interviewers are looking for helps you focus your preparation in the right areas.

Interviewers evaluate three core competencies:

Systems thinking

Can you design a complete pipeline from data ingestion to model serving? Do you understand how components interact and where bottlenecks emerge? When you propose a candidate retrieval stage, do you think about what feeds it data and what consumes its output?

Trade-off analysis

There are no perfect solutions in ML system design. Every decision involves trade-offs. Batch inference is cheaper but produces stale predictions. Complex models are more accurate but harder to serve at low latency. Real-time features are powerful but expensive to compute. Interviewers want to see that you can articulate both sides and make a reasoned choice for the given context.

Practical awarenes

Do you know what actually matters in production? Can you anticipate operational challenges like data drift, training-serving skew, or cold-start problems? Have you thought about how to monitor the system after deployment?

One common misconception is that you need to know the latest model architectures or be able to derive backpropagation on a whiteboard. In practice, what matters far more is the ability to design a system that can reliably deliver ML predictions at scale.

A simple approach like logistic regression, combined with a well-thought-out data pipeline, serving layer, and monitoring, often leads to a stronger solution. In contrast, introducing a cutting-edge transformer adds little value if the training, deployment, and operational aspects are unclear.

The real signal is not how advanced the model is, but how well the entire system works end to end.