Imagine a team that runs their ML workflow as six separate scripts: data validation, feature computation, training, evaluation, registration, and deployment. Each has its own schedule and owner.

One day, training starts before feature computation finishes. The model trains on stale data, passes evaluation, and gets deployed. Predictions quietly degrade for hours.

Nothing is “wrong” with any single script. The problem is coordination. There’s no system enforcing order or checking that inputs are ready before a step runs.

End-to-end ML pipelines fix this by turning the workflow into a single, dependency-aware system. Each stage runs only when its inputs are ready, and the entire process becomes reliable, traceable, and repeatable.

An ML pipeline is a directed acyclic graph (DAG) where each node is a processing step, and edges represent dependencies. A step runs only when all its upstream dependencies have completed successfully. This is a stronger guarantee than cron scheduling, which only knows about time, not about whether previous steps succeeded.

A data pipeline moves data from sources to feature stores and training datasets. A training pipeline takes prepared data and produces a trained model. An end-to-end ML pipeline orchestrates both of these, plus evaluation, registration, and deployment, as a single coordinated workflow.

The Full Lifecycle DAG

A production ML pipeline typically has six to eight stages, each depending on the outputs of the one before it.

Two quality gates control the flow. The data validation gate halts the pipeline if input data fails schema or distribution checks, preventing a model from training on corrupted data. The evaluation gate blocks deployment if the new model doesn't outperform the current production model on key metrics. Without these gates, pipeline automation becomes pipeline-automated-damage.

Each stage has its own internals covered in earlier chapters. What this chapter adds is the orchestration concern at each stage: what triggers it, what it consumes from upstream, what it produces for downstream, and what happens when it fails.

Pipeline Contracts Between Stages

Each stage needs a clear contract with its neighbors: what inputs it expects, what outputs it produces, and how downstream stages know it succeeded.

A data validation step, for example, doesn't just output "pass" or "fail." It produces a validation report that includes the schema version it checked against, distribution statistics for each feature column, and a list of any warnings. The feature engineering step consumes this report to confirm it's working with validated data.

In practice, these contracts are enforced through typed artifacts. Each step declares its input and output types (a dataset path, a model artifact, a metrics dictionary), and the orchestrator verifies that upstream steps produced the expected outputs before triggering downstream steps. This is the difference between "the DAG ran successfully" and "the DAG ran correctly."

The Data Pipelines chapter introduced orchestration tools for data workflows. ML pipelines have additional requirements that narrow the field: artifact passing between steps, step-level caching, GPU scheduling, and integration with experiment tracking systems.

Orchestrator Comparison for ML Workloads

Three orchestrators dominate ML pipeline workflows, each with different strengths.

The container isolation difference is significant. In Airflow, steps share a worker environment by default, which means dependency conflicts between a PyTorch training step and a scikit-learn evaluation step can cause subtle bugs. Kubeflow and Argo run each step in its own container, eliminating this problem entirely.

When to Use Which Orchestrator

The decision depends more on existing infrastructure than on feature checklists.

Airflow is the pragmatic choice when the team already uses it for data pipelines and wants one orchestration platform for everything. Its ecosystem is the largest, and adding ML pipelines to an existing Airflow deployment means no new infrastructure to manage. The trade-off is that ML-specific features (step caching, native artifact tracking) require custom implementation.

Kubeflow Pipelines fits teams that are Kubernetes-native and want an opinionated ML platform out of the box. It includes experiment tracking, artifact management, and pipeline comparison in a single UI. The downside is operational complexity: running Kubeflow reliably requires Kubernetes expertise, and the setup is not trivial.

Argo Workflows sits between the two. It runs on Kubernetes but doesn't prescribe an ML-specific workflow. Each step is a container, which gives maximum isolation and reproducibility. Teams that need multi-language pipelines (Python for training, Go for data processing, Java for serving) often land on Argo because it doesn't care what runs inside each container.

Defining Pipelines in Code

A Kubeflow pipeline definition makes the DAG structure and artifact passing explicit:

The key detail: train_model receives its data_path from validate_data's output. This creates an explicit dependency. Kubeflow won't start training until validation succeeds and produces a valid output path.

The equivalent in Argo Workflows uses a DAG template:

Argo's dependencies field defines the DAG edges, and artifacts flow between steps through the workflow engine. Each task runs in its own container with its own dependencies, so the training step can use PyTorch while the evaluation step uses a lightweight scoring library.

The DAG structure defines execution order. But two practical problems remain: how data and artifacts actually move between steps, and how to avoid redundant computation when inputs haven't changed.

Artifact Passing and Metadata

Three patterns exist for passing data between pipeline steps. The choice affects both performance and debuggability.

Most production teams use a hybrid: pipeline-native artifacts for small metadata (metrics, config files, validation reports) and file-based paths for large artifacts (training datasets, model weights). A 10 GB training dataset shouldn't flow through the orchestrator's artifact system, but a 2 KB evaluation report should.

Caching and Skipping Unchanged Steps

A daily ML pipeline doesn't always need to run every step. If the training data hasn't changed since yesterday, feature engineering and training produce identical results. Skipping those steps saves hours of GPU time.

Step-level caching works by fingerprinting each step's inputs: a hash of the input data version, the step's code, and its configuration. If the fingerprint matches a previous successful run, the orchestrator reuses the cached output instead of re-executing.

In this run, data ingestion and validation execute because they always need to check fresh data. But feature engineering and training are cached because the validated data fingerprint matches yesterday's run. Evaluation still runs because it compares against the current production model, which may have changed.

Kubeflow Pipelines does this natively. When you enable caching, it hashes each step's inputs and skips execution if a matching output exists. For Airflow, you need to implement this manually: compute input hashes in a pre-check task, compare against a metadata store, and conditionally skip downstream tasks using BranchPythonOperator or ShortCircuitOperator.

The savings are substantial. A team running daily retraining on a dataset that changes meaningfully once or twice a week saves 5 out of 7 GPU-hours per week, roughly 70% of training compute, without any change to model quality.

Traditional software has well-established testing practices. ML pipelines need the same rigor, but the strategies look different. You can't unit-test a model's accuracy (that's what evaluation is for), but you can test that the pipeline correctly wires inputs to outputs, handles failures gracefully, and produces artifacts in the expected format.

Unit Testing Pipeline Components

Each pipeline step should be a function that takes inputs and produces outputs, testable without the orchestrator. A feature engineering step takes a dataframe and returns a transformed dataframe. A data validation step takes a dataframe and returns a validation report. Testing these in isolation is fast and catches most bugs.

The test runs in milliseconds, uses synthetic data, and verifies the output schema and value constraints. It doesn't test model accuracy or pipeline orchestration, just that the step does what it claims.

Integration Testing Pipeline Stages

Integration tests verify that two connected stages work together: the output of one stage is valid input for the next. This catches contract mismatches that unit tests miss.

This test runs the validation step and the feature engineering step in sequence on the same data, verifying that validation's "pass" output leads to features that the training step can consume. It catches issues like a validation step that passes data with a column renamed, which the feature step doesn't expect.

End-to-End Pipeline Tests

End-to-end tests run the full pipeline on a tiny dataset with a simple model. The goal is to verify the wiring, not model quality. Use 100-500 rows, train for 1 epoch, and mock the deployment step.

The pyramid applies to ML pipelines just like traditional software. Most coverage comes from unit tests that run fast and catch most bugs. Integration tests cover the boundaries. End-to-end tests are expensive but catch wiring issues that nothing else catches, like a step that writes to the wrong path or an artifact that gets serialized in an incompatible format.

One common mistake: using the full production dataset and model architecture in end-to-end tests. This makes tests take hours and break frequently for reasons unrelated to the pipeline (GPU OOM, training instability). Keep E2E tests small, fast, and focused on testing the pipeline, not the model.

When a model is live in production, and its predictions start getting worse, the obvious question is: what changed?

Was it the training data? A tweak in feature engineering? New hyperparameters? Or something in the deployment config?

Without pipeline versioning, answering this turns into a messy investigation across logs, code, and systems. With proper versioning, it’s simple. You look up the exact pipeline version and see what changed.

What to Version in a Pipeline

Three things define a pipeline version, and all three must be tracked together.

Pipeline definition: The DAG structure, step ordering, and dependency graph. This is typically code (a Python file for Kubeflow, a YAML manifest for Argo, a DAG file for Airflow) and should live in version control.

Step configurations: Hyperparameters, data paths, resource allocations, threshold values for quality gates. These often change independently of the pipeline structure. A team might keep the same DAG but increase the training learning rate or change the evaluation threshold.

Execution environment: Container images, library versions, hardware specifications. A pipeline that runs identically on paper can produce different results if one run uses PyTorch 2.1 and the next uses PyTorch 2.3, or if one runs on A100 GPUs and the next on V100s (due to different floating-point behavior).

Pipeline Lineage: From Prediction to Pipeline

Pipeline versioning closes the lineage loop. Data versioning tracks which data was used. Model versioning tracks which model is deployed. Pipeline versioning connects the two: which pipeline definition, using which data version and which configuration, produced which model.

With this chain, debugging a production issue becomes systematic. Model v23 came from pipeline run #847. That run used data version v445, pipeline definition v2.1 (git commit a3f7b2c), specific training hyperparameters, and a specific container image. Compare each element against the previous successful run to find what changed.

The practical implementation depends on your orchestrator.

Kubeflow tracks the most out of the box. Airflow combined with MLflow covers the same ground but requires explicit logging at each step. Argo provides execution-level tracking but needs custom code for ML-specific metadata like data versions and model metrics.

The minimum viable lineage implementation: tag every registered model with three values, the pipeline git commit hash, the pipeline run ID, and the data version. This takes a few lines of code regardless of orchestrator and answers 90% of "what produced this model?" questions.