ML system design interviews are open-ended by nature. There's no single correct architecture, no perfect model choice, no objectively right answer. The interviewer gives you a vague problem ("Design a recommendation system for YouTube") and watches how you navigate ambiguity under time pressure. Without a framework, it's easy to spend 20 minutes talking about model architectures while never mentioning how you'd actually serve predictions or handle data freshness. A structured approach keeps you on track and ensures you cover what matters.

A typical ML system design interview runs 45-60 minutes. The interviewer presents a broad problem, and you're expected to drive the conversation. They might nudge you toward specific areas, but the structure is yours to impose.

Interviewers evaluate three things. First, structured thinking: can you break a vague problem into concrete, solvable pieces? Second, trade-off analysis: can you compare approaches and explain why you'd pick one over another? Third, breadth and depth balance: can you sketch the full system while also going deep on the parts that matter?

The biggest misconception is that these interviews are about ML knowledge. They're not, or at least not primarily. They're about systems thinking applied to ML problems. An answer that uses a simple model but nails the data pipeline, serving architecture, and monitoring story will beat an answer that describes a state-of-the-art model but ignores everything around it.

The framework has six phases. Each one builds on the previous, and together they cover everything an interviewer expects to see.

The phases aren't rigid walls. You'll often loop back (a deep dive question might surface a requirement you missed), and the interviewer may steer you to spend more time in one area. But this sequence gives you a reliable starting point.

Phase 1: Clarify Requirements (5-7 minutes)

Never start designing immediately. The problem statement is intentionally vague, and the interviewer expects you to narrow it down. "Design a recommendation system" could mean YouTube's home feed, Netflix's "Because You Watched" row, or Amazon's "Customers Also Bought" widget. Each is a fundamentally different system.

Ask questions across four categories:

A short dialogue might look like this:

Candidate: "When you say recommendation system, are we talking about the home feed, or recommendations within a specific context like 'related videos'?"

Interviewer: "Let's focus on the home feed, the personalized set of content a user sees when they open the app."

Candidate: "What's the scale we're designing for? Millions of users, or smaller?"

Interviewer: "Think large-scale. 100 million daily active users, 500 million items in the catalog."

Candidate: "And the primary business goal, are we optimizing for watch time, clicks, or something else?"

Interviewer: "Watch time is the primary metric, but we also care about content diversity."

In three exchanges, you've narrowed a vague problem into something concrete: a personalized home feed for 100M DAUs with 500M items, optimizing for watch time with a diversity constraint. Every subsequent design decision is now grounded in these specifics.

Phase 2: Define Metrics (3-5 minutes)

Metrics anchor your design. Without them, there's no way to evaluate whether your system is actually working. You need two types: offline metrics (measured before deployment) and online metrics (measured in production).

Offline metrics tell you whether your model learned useful patterns. For a ranking problem, you might use NDCG (Normalized Discounted Cumulative Gain) to measure ranking quality, or AUC to measure how well the model separates relevant from irrelevant items. For a classification problem, precision and recall capture different aspects of accuracy.

Online metrics connect model performance to business outcomes. A model might have great offline AUC but not actually increase user engagement. Online metrics like click-through rate, average session duration, or revenue per user measure what actually matters. You set these up through A/B tests: serve the new model to a percentage of users and compare against the current system.

The connection between offline and online metrics is where things get interesting. A model with higher offline NDCG doesn't always produce higher online engagement. Maybe it ranks perfectly according to historical data but doesn't surface enough novel content, so users get bored. Mentioning this tension shows the interviewer you understand that model quality and system quality aren't the same thing.

Phase 3: Propose Architecture (8-10 minutes)

This is where you draw the system. Start with a high-level diagram that shows how data flows from collection to prediction serving. Don't try to design every component. Lay out the major pieces and explain why each one exists.

A generic ML system architecture looks something like this:

Walk through the diagram as you draw it. Data flows in from event logs, user profiles, and the item catalog. A data pipeline processes raw events into clean features, which feed into both a feature store (for serving) and training datasets (for model training). The training pipeline produces model versions stored in a registry. At serving time, candidate retrieval narrows millions of items to a few thousand, the ranking model scores those candidates, and a re-ranking layer applies business rules and diversity filters.

You don't need to get the architecture perfect on the first pass. The goal is to establish a shared mental model that you and the interviewer can build on.

Phase 4: Data and Features (8-10 minutes)

Data decisions often matter more than model decisions. A simple model with great features beats a complex model with poor features almost every time. Spend real time here.

Start with data sources: what signals are available? User interaction logs (clicks, views, purchases), user profiles (demographics, preferences), item metadata (categories, descriptions, popularity), and contextual signals (time of day, device, location).

Then organize features into categories:

Discuss how features are computed. Batch features (updated hourly or daily) vs real-time features (computed at request time). Mention training-serving consistency: the features the model trains on must be computed the same way at serving time.

For training data, address labeling. In a recommendation system, implicit feedback (clicks, watch time, skips) is abundant but noisy. Explicit feedback (ratings) is clean but sparse. Discuss this trade-off briefly.

Phase 5: Model Selection and Training (5-8 minutes)

Start with a baseline. For a recommendation problem, a collaborative filtering approach or a gradient-boosted tree model on hand-crafted features is a reasonable starting point. Explain why: it's fast to train, easy to debug, and gives you a benchmark to beat.

Then propose a more sophisticated approach if the scale or requirements demand it. For a 100M-user recommendation system, you might use a two-tower neural network for candidate retrieval (one tower for users, one for items) and a deeper ranking model that incorporates all feature categories.

The key is justifying complexity. Don't jump to a complex model because it sounds impressive. Explain the specific limitation of the simpler approach that the more complex one addresses. "Collaborative filtering struggles with cold-start items because it needs interaction history. A content-based tower can recommend new items based on their metadata alone."

Briefly mention the training pipeline: how often does the model retrain? What data window does it train on? How do you evaluate a new model before promoting it to production? These operational details signal that you're thinking about a production system, not a research prototype.

Phase 6: Deep Dive (10-15 minutes)

This is the most important phase. The first five phases establish that you can structure a problem and design a reasonable system. The deep dive is where you demonstrate genuine depth.

The interviewer will either ask you to go deep on a specific topic or let you choose. Either way, pick areas with real trade-offs, problems where reasonable engineers would disagree on the approach. Good deep dive topics include:

Each deep dive should follow a pattern: state the problem, present 2-3 approaches, compare their trade-offs, and recommend one with a clear rationale.

Pacing matters. Spending too long on requirements leaves no time for depth. Rushing through architecture means your design has gaps the interviewer will probe.

These times are guidelines, not rigid rules. If the interviewer spends 10 minutes on requirements because the problem is complex, adapt your pacing for the remaining phases. The important thing is being aware of how much time you're spending so you don't run out before reaching the deep dive.

Starting with the model. "I'd use a transformer-based architecture with multi-head attention..." in the first minute is a red flag. It signals that you're thinking about models, not systems. Always start with requirements and metrics.

Ignoring scale. Designing a system for 1,000 users is very different from designing one for 100 million. If you don't ask about scale, your architecture decisions aren't grounded in reality. A brute-force scoring approach that works at small scale falls apart at large scale.

No metrics. Designing without defining success criteria means you can't justify any of your design decisions. Why did you pick this model? Why this feature set? Why this serving architecture? The answer should always connect back to your metrics.

Monologuing. Going 15 minutes without checking in with the interviewer is risky. They might want you to go deeper on something you glossed over, or they might want you to skip something you're about to spend five minutes on. Pause after each phase: "Does this look reasonable, or would you like me to go deeper on any of these components?"

All breadth, no depth. Covering every component at a surface level shows you can name the parts of an ML system but not that you can design one. The deep dive phase exists specifically so you can show expertise on specific problems. Picking two topics and going deep beats covering six topics shallowly.

Treating it as a research paper. Interviewers don't want to hear about loss function gradients or backpropagation mechanics. They want to understand your system design reasoning: what are the trade-offs, what breaks at scale, how do you handle failures? Keep the focus on engineering, not math.

Here's how the framework applies to a concrete problem: "Design a video recommendation system for a platform with 100M daily active users."

Requirements. After clarifying, you've scoped to the home feed. 100M DAUs, 500M videos, optimizing for watch time with a diversity constraint. Latency target: under 200ms end-to-end.

Metrics. Offline: NDCG@20 for ranking quality, recall@500 for retrieval coverage. Online: average watch time per session, content diversity (unique categories shown per user), and user retention as a guardrail.

Architecture. You sketch the standard pipeline: event logs feed a data pipeline that populates a feature store and generates training data. A two-stage serving system handles retrieval (narrowing 500M videos to 1,000 candidates) and ranking (scoring those 1,000 to produce the final 20).

Data and features. User features from watch history and profile data. Video features from metadata and engagement stats. Context features like time of day and device. Cross features capturing user-video affinity from past interactions.

Model. Baseline: a matrix factorization model for retrieval plus a gradient-boosted tree ranker. Production target: a two-tower model for retrieval (user tower + video tower producing embeddings for approximate nearest neighbor search) and a deep ranking model incorporating all feature categories.

Deep dive. You go deep on two topics. First, the cold-start problem: new videos have no interaction history, so the retrieval model can't generate useful embeddings. You propose a content-based fallback using video metadata and creator features, with an exploration budget that intentionally surfaces new videos to collect initial signals. Second, latency: how to keep the full pipeline under 200ms. You break down the latency budget across retrieval (20ms ANN lookup), feature fetching (10ms parallel reads from the feature store), ranking (30ms batched GPU inference), and re-ranking (5ms rule-based filters).

This walkthrough covers the full framework in a few minutes. The interview deep dive chapters later in the course go much deeper on problems like this, but the framework ensures you don't miss any critical phase.

Q1: Why should you define metrics before proposing an architecture?

Metrics shape the architecture. If your primary metric is real-time CTR, you need a low-latency serving path and near-real-time feature updates. If it's long-term user retention, you might prioritize content diversity and exploration over pure engagement optimization, which affects your re-ranking logic and training objective. Without metrics, you're making architecture decisions in a vacuum. Defining them first also gives you a concrete way to evaluate trade-offs later: "Approach A gives us better NDCG but 3x higher serving latency. Given our 200ms budget, approach B is the better fit." Every design decision should trace back to a metric.

Q2: A candidate spends 25 minutes on model architecture and 5 minutes on everything else. What's wrong with this approach?

The model is one component of a much larger system. Spending 25 minutes on it signals that the candidate is thinking like a researcher, not a systems engineer. In practice, model choice is rarely the bottleneck. Data quality, feature engineering, serving infrastructure, and monitoring are where production systems succeed or fail. An interviewer hearing 25 minutes of model details still doesn't know how the candidate would handle data freshness, cold-start users, or serving latency. The framework allocates only 5-8 minutes to model selection because the system around the model matters more than the model itself.

Q3: How do you decide which topics to deep dive on?

Pick topics with genuine trade-offs, problems where there are multiple valid approaches with different strengths. Serving latency is a good deep dive because there are real choices (batching vs streaming inference, CPU vs GPU, model distillation vs caching). Cold start is good because it pits exploration against exploitation. Avoid topics where the answer is straightforward or where you'd just be reciting textbook knowledge. If the interviewer doesn't specify, pick the 2-3 topics where you can demonstrate the strongest reasoning about trade-offs. Depth on two topics beats surface coverage of five.

Q4: When should you deviate from the framework?

The framework is a default, not a script. If the interviewer says "I'm particularly interested in how you'd handle the serving infrastructure," don't mechanically walk through requirements and metrics for 10 minutes. Spend 2-3 minutes on a quick problem scoping, then jump to architecture and the serving deep dive. Similarly, if you're 30 minutes in and haven't reached the deep dive, compress model selection to a few sentences and move on. The goal is demonstrating structured thinking and depth, not completing a checklist. Adapt to the interviewer's signals: if they're nodding and want you to move on, move on. If they're asking follow-up questions, go deeper.

With the framework in place, the next section builds the ML vocabulary you'll use at every stage: the types of ML problems, common algorithms, and the evaluation techniques that come up in every interview.