A model is only as reliable as the pipeline that produced it. Small differences, like data order, random seeds, or a missed preprocessing step, can lead to different results even with the same data and architecture.

Training pipelines bring structure to this process. They turn the path from raw data to a deployable model into something repeatable, auditable, and automated.

A training pipeline is a sequence of stages that takes prepared data and produces a trained model artifact. Each stage has a clear input and output, and the pipeline as a whole is designed to run end-to-end without manual intervention.

Here's how these stages connect in a typical pipeline:

The flow is linear, but the evaluation stage acts as a gate. If the newly trained model doesn't meet quality thresholds (relative to the currently deployed model or absolute benchmarks), it doesn't proceed to storage or deployment. This prevents a bad training run from silently replacing a working model.

Each stage is worth understanding in depth, because the design choices at every step affect model quality, training speed, and operational reliability.

The first bottleneck in any training pipeline is getting data into memory fast enough to keep GPUs busy. A single NVIDIA A100 can process thousands of samples per second. If the data loading pipeline can only deliver hundreds, the GPU sits idle, and you're paying for expensive hardware that's mostly waiting.

Data Formats

The format you store training data in directly affects how fast you can read it.

For tabular ML (recommendations, fraud detection, ad click prediction), Parquet is the standard. It's columnar, so reading a subset of features is fast. It compresses well. And it's natively supported by Spark, PyArrow, and every major data tool.

For deep learning on unstructured data (images, text, audio), sequential formats like TFRecord or WebDataset perform better. They're optimized for streaming reads, which matches how training loops consume data: one batch at a time, in order.

The Data Loading Pipeline

Naive data loading, reading one batch from disk, preprocessing it, sending it to the GPU, then reading the next, creates a sequential bottleneck. Modern frameworks solve this by overlapping data loading with training.

The key techniques:

Here's a PyTorch data loading setup that applies these techniques:

The pin_memory=True flag allocates batches in page-locked memory, which speeds up the CPU-to-GPU transfer. Combined with prefetching and parallel workers, this keeps GPU utilization above 90% for most workloads.

Sharding for Large Datasets

When training data reaches terabytes, a single machine can't hold it all on local disk. The solution is sharding: splitting the dataset into hundreds or thousands of files (shards), stored in object storage (S3, GCS). Each training run reads a subset of shards, and workers divide shards among themselves.

Sharding also enables a simple form of data parallelism even before getting into distributed training. Multiple workers each read different shards, so there's no contention or duplicate reads.

Once data is loaded, it needs to be transformed into the format the model expects. Preprocessing sits between raw data and the training loop, and getting it right, or wrong, directly impacts model quality.

Common Preprocessing Steps

Preprocessing: Pipeline vs Ad-Hoc

A common mistake is applying preprocessing steps in ad-hoc scripts rather than as a defined pipeline stage. The problem shows up at serving time: the model was trained with a specific normalization scheme, but the serving code applies a slightly different one (different mean/std values, different handling of missing values), and predictions are silently wrong.

The fix is to treat preprocessing as a versioned, reusable component. Frameworks like scikit-learn's Pipeline, TensorFlow Transform (tf.Transform), and Feast's on-demand feature transforms define preprocessing once and apply it identically in both training and serving.

Saving the preprocessing and model together as a single artifact eliminates the class of bugs where training and serving preprocessing diverge. This is particularly important for normalization statistics: if you compute mean and standard deviation during training but hardcode them in serving code, any data distribution shift will cause a mismatch.

The training loop is where the model actually learns. It's conceptually simple: iterate over the data, compute predictions, measure the loss, compute gradients, and update weights. But the engineering around it determines whether training is reliable, efficient, and debuggable.

Checkpointing

Training runs fail. A machine runs out of memory. A spot instance gets preempted. A NaN appears in the loss. Without checkpointing, a failure after 10 hours of training means starting over from scratch.

Checkpoints save the current state, model weights, optimizer state, learning rate schedule, and the current epoch/step, to disk at regular intervals. When training resumes after a failure, it loads the latest checkpoint and continues from where it left off.

The trade-off is storage and I/O overhead. Saving a checkpoint for a large model (several GB) every 100 steps adds I/O latency. Saving every 10,000 steps risks losing hours of work on failure. A common strategy: save every N steps (where N balances I/O cost against potential lost work), and keep only the K most recent checkpoints to avoid filling up storage.

Logging and Monitoring

Training isn't a black box. At minimum, log the training loss, validation loss, and learning rate at each step. These metrics reveal whether training is progressing (loss decreasing), overfitting (training loss drops but validation loss rises), or diverging (loss spikes or hits NaN).

Beyond basic metrics, log resource utilization: GPU memory usage, GPU utilization percentage, and data loading throughput. If GPU utilization drops below 80%, the pipeline has a bottleneck, usually in data loading or preprocessing.

Logging frameworks like TensorBoard, Weights & Biases, and MLflow provide dashboards for tracking these metrics across runs. Experiment tracking, comparing metrics across different training runs, is covered in a later chapter.

Every training pipeline needs a quality gate: an evaluation step that decides whether the trained model is good enough to proceed. Without it, any training run, even one that diverged or overfit, produces an artifact that could end up in production.

The evaluation stage runs the trained model against a held-out validation set and computes the metrics that matter for the use case (AUC for ranking, precision/recall for classification, RMSE for regression). These metrics are compared against two baselines:

If either check fails, the pipeline halts: no artifact is stored, no deployment is triggered, and the team is alerted. This is the single most important safeguard in an automated training pipeline. Without it, periodic retraining (covered later in this chapter) becomes a liability rather than an asset.

A trained model is more than a file of weights. To be useful in production and debuggable after deployment, a model artifact needs to include everything required to understand, reproduce, and serve it.

What to Store

Everything produced by a training run flows into a model registry (MLflow Model Registry, SageMaker Model Registry, or a custom solution backed by S3 and a metadata database). The registry stores artifacts alongside metadata, making it possible to answer questions like "which dataset trained the model serving traffic right now?" or "what changed between model v41 and v42?"

Model versioning and registry design is covered in detail in a later chapter. For now, the key principle is: store everything needed to reproduce the model. If you can't recreate a model from its artifacts, you don't really have a training pipeline. You have a script.

Choosing where and how to run training is an infrastructure decision that affects cost, speed, and operational complexity.

GPU vs CPU

Not everything needs a GPU. XGBoost on a 10M-row tabular dataset trains in minutes on a 32-core CPU machine. Adding a GPU won't make it meaningfully faster because tree-building algorithms don't parallelize the same way matrix operations do. GPUs become essential when training involves large matrix multiplications: neural networks, transformers, embedding tables.

On-Premise vs Cloud

The hybrid approach is increasingly common. Teams use on-premise GPUs for steady-state training jobs (daily retraining of production models) and burst to cloud for experiments, hyperparameter sweeps, and one-off large training runs.

Spot and Preemptible Instances

Cloud GPU instances are expensive. AWS spot instances and GCP preemptible VMs offer the same hardware at 60-90% discount, but they can be terminated with little notice (2 minutes on AWS, 30 seconds on GCP).

For training pipelines, this is a good trade-off, if you have robust checkpointing. A training job with checkpoints every 30 minutes loses at most 30 minutes of work when preempted. It resumes from the last checkpoint on a new instance. The 70% cost savings more than compensate for the occasional lost work.

The pattern: run training on spot instances, checkpoint frequently, configure the orchestrator to automatically restart preempted jobs. Most ML platforms (SageMaker, Vertex AI, Databricks) support this natively.

Training the same pipeline on the same data should produce the same model. In practice, this is harder than it sounds.

Sources of Non-Determinism

Making Training Reproducible

Full determinism requires addressing all four sources:

Setting torch.use_deterministic_algorithms(True) forces PyTorch to use deterministic (but sometimes slower) implementations of GPU operations. This is the nuclear option: full reproducibility at the cost of 10-20% slower training. For production pipelines where auditability matters, the slowdown is worth it. For experimentation, most teams accept minor non-determinism and just fix the random seed, which gets you 99% of the way to reproducible results.

Environment Pinning

Code-level reproducibility means nothing if the environment differs. Pin everything:

Better yet, use Docker to capture the entire environment, OS, CUDA version, Python version, and all dependencies:

The combination of pinned random seeds, deterministic algorithms, and a frozen Docker environment gives you bit-for-bit reproducibility. Store the Docker image hash alongside the model artifact so you can recreate the exact training environment months later.

A training pipeline that runs manually isn't a pipeline. It's a script. Production models need regular retraining because the data they were trained on becomes stale as user behavior, market conditions, and content catalogs change.

Retraining Strategies

Two approaches, and most production systems use a combination:

Periodic retraining runs on a fixed schedule: daily, weekly, or hourly. It's simple to implement, easy to reason about, and predictable in cost. A daily retraining job that runs at 2 AM, trains on the last 30 days of data, evaluates against the current model, and promotes the new model if it's better, covers most use cases.

Trigger-based retraining fires when a condition is met: model performance drops below a threshold, data drift is detected, or a certain volume of new labeled data arrives. It's more efficient because it only retrains when there's a reason to, but it requires monitoring infrastructure to detect the triggers.

This flow combines both strategies. Periodic monitoring checks model health. When degradation is detected, the system inspects data drift to decide between full retraining (significant drift, the old model's assumptions are stale) and fine-tuning (minor shift, the model just needs a refresh on recent data).

An Automated Retraining DAG

Here's what a daily retraining pipeline looks like in Airflow:

The BranchPythonOperator at the evaluation step makes the pipeline self-governing. If the new model beats the current one, it's promoted. If not, the pipeline logs why it skipped and the existing model keeps serving. No human intervention needed for the happy path, but the team is notified when models are skipped so they can investigate.

Training Window Design

One decision that's easy to overlook: how much historical data should each training run use? This is the training window, and it involves a trade-off between stability and freshness.

The right window depends on how fast the underlying distribution changes. Ad click prediction models at Meta and Google retrain every few hours on very recent data because user intent shifts quickly. Fraud detection models use longer windows because fraud patterns evolve more slowly and rare events need more data to learn from.