A retrained fraud detection model can pass every offline evaluation, show higher AUC, and still fail in production. The most common failure mode is over-aggressive predictions that hurt real users. Example: conversion dropped 12% within days because legitimate transactions were being blocked.

Without safeguards like rollback, traffic splitting, and staged deployment, recovery is slow and expensive. Teams end up scrambling to restore the previous model while the business takes a hit.

This is the core issue. Deploying ML models is fundamentally riskier than deploying application code, and it demands different strategies to manage that risk.

When a software deployment goes wrong, the failure is usually obvious. An API returns 500 errors, a page fails to render, a service crashes. Monitoring catches it immediately, and a rollback fixes it.

ML failures are subtler. A bad model doesn't crash. It serves predictions that are slightly worse, and that degradation hides inside aggregate metrics for hours or days before anyone notices.

A recommendation model that starts surfacing less relevant content won't trigger any error alerts. Click-through rates will quietly drop, and by the time someone investigates, the model has been serving bad predictions to millions of users.

The root cause is also different. A software bug is deterministic: given the same input, it always produces the same wrong output. An ML failure is statistical. The model might perform well on 95% of requests but catastrophically on a specific user segment that wasn't well-represented in evaluation. Or the training data distribution might have shifted just enough that the model's decision boundary no longer aligns with real-world patterns.

This gap between offline evaluation and production behavior is why deployment strategies matter so much for ML. You need mechanisms to test models in production conditions, limit the blast radius when something goes wrong, and revert quickly when it does.

Blue-green deployment is the simplest strategy for reducing deployment risk. You maintain two identical serving environments: "blue" (currently serving production traffic) and "green" (idle, ready for the new version). When you deploy a new model, you load it into the green environment, verify it's healthy, and then switch all traffic from blue to green in one step.

The key advantage is rollback speed. If the new model misbehaves after the switch, you route traffic back to the blue environment, which still has the old model loaded and warm. Total rollback time is seconds, not minutes.

For ML systems, blue-green deployments need a few adaptations beyond what standard software deployments require:

Blue-green works well when you have high confidence in the new model and want fast rollback capability. It's less ideal when you want to gradually validate the model in production, because the switch is all-or-nothing: either 100% of traffic goes to the new model or 0%.

Canary deployment solves the all-or-nothing problem by routing a small percentage of traffic to the new model and gradually increasing it as confidence builds. The name comes from the "canary in a coal mine" metaphor: if the canary (small traffic slice) shows problems, you stop before exposing the full user base.

The critical design decision in canary deployments is the rollout schedule: how much traffic to route at each stage, how long to wait, and what metrics to evaluate before promoting to the next stage.

The durations at each stage aren't arbitrary. At 1% traffic, you're checking for catastrophic failures: does the model crash, does latency spike, are predictions in the expected range? You don't need statistical significance for that. But at 5-25%, you're comparing business metrics between the canary and production populations, and that requires enough samples to detect meaningful differences.

A common mistake is promoting too quickly. If your model serves 10 million predictions per day, 1% gives you 100,000 samples, which sounds like a lot. But if the metric you care about is purchase conversion rate (say 2%), you need the canary to run long enough to accumulate enough conversions to detect a 5-10% relative change with statistical significance. Rushing through stages defeats the purpose.

Here's what automated canary evaluation logic looks like in practice:

The function checks three things: latency hasn't regressed, error rates are acceptable, and the primary business metric hasn't significantly degraded. The key word is "degraded," not "improved." During a canary, you're primarily looking for regressions rather than trying to prove the new model is better. Proving improvement with statistical confidence often requires more traffic and time than a canary stage allows.

Shadow deployment (also called "dark launching") takes a more conservative approach: the new model receives production traffic and generates predictions, but those predictions are never served to users. Instead, they're logged alongside the production model's predictions for offline comparison.

Shadow mode gives you production-quality evaluation data without any user-facing risk. You can compare the new model's predictions against the production model's predictions on real traffic, catch edge cases that offline evaluation missed, and measure latency under real load.

This is particularly valuable for high-stakes models where even a small canary carries meaningful risk. A fraud detection model that lets through fraudulent transactions during a canary test costs real money. A content moderation model that misses harmful content during a canary test exposes real users to harm. Shadow mode lets you evaluate these models with zero user impact.

But shadow deployment has an important limitation: it only works when you can evaluate predictions independently of whether they were served. For a recommendation model, the shadow model's predictions never reach users, which means you never observe whether users would have clicked on the shadow model's recommendations. You can compare the prediction distributions (did the shadow model rank different items higher?), but you can't measure the actual business outcome.

This limitation applies to any model where the prediction influences the outcome you're measuring. Ad ranking, content recommendation, and search ranking all have this feedback loop problem.

Shadow mode works well for classification models where ground truth arrives independently: fraud detection (the transaction is either fraudulent or not regardless of the prediction), medical diagnosis (the patient's condition exists regardless of the model's output), and credit scoring (the applicant's creditworthiness doesn't change based on which model scored them, assuming both models make the same final decision).

A practical approach for recommendation-like systems is to combine shadow mode with interleaving. Run the shadow model's predictions alongside the production model's predictions, randomly interleave them in the served results, and track which model's items get more engagement. This gives you production-quality signals without the full risk of a canary rollout.

Feature flags decouple deployment from release. You deploy the new model to production infrastructure (it's sitting there, loaded, ready to serve) but a feature flag controls whether any traffic actually reaches it. This separation is powerful because it lets you deploy during low-risk hours (Tuesday morning) and activate during high-attention hours (Monday during business review) with a single configuration change.

For ML systems, feature flags go beyond simple on/off toggles. They enable targeted rollouts that are more nuanced than canary traffic splitting.

One ML-specific pattern is the "model experiment" flag. Instead of a single new model behind a flag, you run multiple model variants and use the flag system to allocate traffic across them. This turns deployment infrastructure into experimentation infrastructure. Teams at Netflix and LinkedIn use this pattern extensively, running dozens of model experiments simultaneously with feature flags controlling the allocation.

Feature flags also provide a clean solution for the "model dependency" problem. If Model B depends on Model A's output as a feature, updating Model A can break Model B. With feature flags, you can deploy the updated Model A behind a flag, route a fraction of traffic to it, and verify that Model B still performs well on Model A's new outputs before fully activating the change.

The main risk with feature flags is flag debt. Stale flags that no one removes create a combinatorial explosion of possible system states. For ML systems, this means old model versions sitting on servers consuming memory, and configuration complexity that makes debugging harder. Establish a discipline of cleaning up flags within 2-4 weeks of full rollout.

Rollback is the safety net behind every other deployment strategy. When canary metrics degrade, when shadow evaluation reveals problems, or when a fully deployed model starts underperforming, you need to revert to the previous version quickly and cleanly.

For traditional software, rollback means deploying the previous code artifact. For ML systems, it's more involved because a model's behavior depends on more than just the model artifact.

Three things define a model's behavior in production:

Rolling back just the model artifact doesn't help if the feature pipeline has also changed. If model v2.2 was trained on a new feature that model v2.1 doesn't use, and the feature pipeline was updated to compute that feature, rolling back the model works.

But if model v2.2 changed how an existing feature is computed (say, switching from a 7-day to a 30-day window for average session length), rolling back the model while keeping the new feature computation will produce a model receiving features it wasn't trained on. This is a recipe for silent, hard-to-debug degradation.

Automated rollback triggers should cover both hard failures and soft degradation:

The distinction between automated and manual rollback matters. Automated rollback handles clear-cut failures where speed is critical: the model server is returning errors, latency has doubled, or prediction distributions have shifted dramatically. These are unambiguous signals that something is broken.

Manual rollback handles the harder cases. A 3% drop in click-through rate might be a model regression, or it might be a seasonal effect, a competing product launch, or noise. Having a human review the evidence before rolling back prevents unnecessary reverts that would disrupt ongoing experiments and erode team confidence in deployment automation.

One important practice: always maintain at least two previous model versions in a deployable state. The current production model and the one before it should both be loaded (or quickly loadable) in serving infrastructure. This eliminates the failure mode where you need to roll back but the previous model artifact is archived in cold storage and takes 30 minutes to retrieve and load.

In practice, teams combine these strategies into a deployment pipeline rather than using any single one in isolation. A typical production ML deployment pipeline looks like this:

The pipeline starts with shadow deployment to catch obvious prediction distribution issues with zero risk. If the shadow model's outputs look reasonable, it moves to a small canary (1%) for initial production validation. The canary percentage increases through stages, with automated metric checks at each gate. Feature flags control the traffic allocation at every step, and rollback is available at any stage.

Different types of models call for different pipeline configurations:

The level of caution should match the cost of a bad prediction. A search ranking model that surfaces slightly less relevant results for a few hours is annoying but recoverable. A fraud detection model that lets through fraudulent transactions during a canary costs real money with every bad prediction. Design your deployment pipeline to match the stakes.