An ETA prediction system estimates how long a journey will take from now until arrival, and serves that estimate to riders, couriers, dispatchers, and downstream systems like pricing and routing. What makes it an ML problem rather than a routing problem is that the physical shortest path gives you a distance, but the time depends on dozens of variables the graph alone doesn't see: traffic, weather, time of day, driver behavior, restaurant prep speed, whether the courier is still picking up from the restaurant or already en route to the drop-off. A good design is a stacked regressor that predicts each leg of the journey separately, quantifies uncertainty rather than returning a single number, and refines its estimate as fresh GPS pings arrive.

Clarifying Questions

ETA systems look deceptively similar across products but differ in what the prediction represents, which edges of the journey the ML model owns, and how the number is consumed. Pinning these down first changes almost every later design decision.

Candidate: "What product surface are we building ETA for? Rideshare, food delivery, package delivery, or something like Google Maps directions?"

Interviewer: "Food delivery on a platform that handles about 10 million orders a day across 30 countries. Think DoorDash or Uber Eats shape. The user-facing ETA shown at order time is the full journey from order placement to drop-off at the customer, not just the drive time."

Candidate: "That's a multi-leg journey: courier travels to the restaurant, waits for food, then drives to the customer. Are we predicting the total end-to-end time, or each leg separately?"

Interviewer: "That's exactly the design question. Product needs a single number to show the customer, but internally you can decompose however you want. The merchant-prep time varies a lot by restaurant, by time of day, and by order size, so treating it as a black box inside one model probably won't work."

Candidate: "Where is the ETA used? Is it shown only to the customer, or do dispatching and pricing depend on it too?"

Interviewer: "Three consumers. The customer sees the ETA at order time and gets live updates during the trip. Dispatch uses the per-leg estimates to assign couriers, because a courier who can reach the restaurant in 3 minutes is a better match than one who can reach it in 8. Pricing uses the expected delivery duration for dynamic fees and surge logic. Different consumers care about different things: the customer wants accuracy and calibrated uncertainty, dispatch wants fast refreshes at low latency, pricing tolerates more latency but wants unbiased estimates."

Candidate: "What's the latency budget? The customer-facing ETA at order time is probably more forgiving than dispatch."

Interviewer: "Dispatch runs it at 50,000 requests per second during peak meals with a p99 budget of 150ms because it scores thousands of courier-order pairs per dispatch cycle. Customer-facing prediction at order time can take up to 500ms. Mid-trip live updates run every 10 seconds and share the dispatch latency budget."

Candidate: "How is accuracy measured? MAE on delivery time, or something business-facing?"

Interviewer: "Both. Offline we use MAE and quantile loss. The online metric that matters most is the fraction of deliveries where actual time exceeds the customer-facing ETA by more than 5 minutes, because that's the complaint trigger. Under-promising pads the ETA and depresses conversion; over-promising breaks trust. The sweet spot is a slight pessimistic bias, and the product decides how much."

Candidate: "What's the feedback timeline? When do we know the actual time?"

Interviewer: "The drop-off confirmation happens in real time, so labels arrive within seconds of the trip ending. Unlike fraud, there's no 90-day wait. The main label quality issue is that some drops get confirmed by the courier on arrival while others get auto-confirmed after the fact, so there's a small amount of label noise on the trailing tens of seconds."

Candidate: "What about extreme conditions? Weather events, stadium lets out, a road closes because of an accident."

Interviewer: "All of them happen daily somewhere. The design needs to handle them without tripping over itself. Baseline models often blow up on rain days because traffic patterns change discontinuously."

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

The natural framing is per-leg regression with distributional outputs. Let a delivery consist of three legs: pickup drive (courier current location to restaurant), dwell (arrival at restaurant to departure with order), and drop-off drive (restaurant to customer). For each leg, the model predicts not a single number but a distribution over time-to-complete, typically parameterized by a handful of quantiles or the parameters of a mixture density. The total delivery ETA is the sum of the three leg predictions, with uncertainty propagated through the sum.

Two design choices in the framing matter more than the choice of architecture. First, training one model per leg is cleaner than training an end-to-end model on total time. The legs have different feature dependencies (dwell time depends almost entirely on restaurant and order attributes, drive time depends on road network and traffic), and a shared model would either split its capacity poorly or require complex gating. Per-leg models also allow independent iteration: improving dwell-time prediction doesn't require retraining the drive-time model.

Second, the loss function has to match the asymmetric cost of errors. Under-estimating by 5 minutes costs a customer complaint, over-estimating by 5 minutes costs conversion. Quantile loss at multiple quantiles (p10, p50, p90) captures the distribution, and the action is to show the customer a point between p50 and p70 depending on product preference. A plain MSE loss is the wrong objective because it optimizes a symmetric penalty and ignores the business asymmetry.

Metrics

The lateness event rate gets singled out because it's the metric that connects directly to customer satisfaction and also has the cleanest interpretation: a 12% lateness rate means 12% of deliveries in the last week ran more than 5 minutes late relative to the ETA we promised. It's the metric every review comment, support escalation, and executive dashboard eventually maps back to.

A delivery's ETA is computed many times over its lifetime. Once at order time when the customer sees the initial estimate, thousands of times during dispatch as the system evaluates candidate couriers, and then every few seconds during the trip as fresh GPS pings and state transitions arrive. The architecture has to serve all three with different latency and freshness characteristics while sharing features and models.

The routing service produces the physical route between two points as a sequence of road segments along with raw distance and the graph's baseline travel-time estimate assuming no traffic. Everything after that is ML. Feature fetch pulls segment-level traffic, restaurant-level prep priors, courier-level speed history, and global context (weather, day-of-week, lunch peak flag). The leg-wise predictor runs pickup-drive, dwell, and drop-off-drive models in parallel and returns three distributions. The aggregator sums them, applies the product's quantile preference (typically p65 for the customer-facing number), and returns a final ETA with an interval.

Three asynchronous loops keep the inputs fresh without blocking any single request.

The traffic loop is the fastest: every active courier's GPS pings flow into a streaming aggregator that computes segment-level speeds on rolling 2-minute windows, and the result lands in an online feature store keyed by segment ID. The prep-time loop updates per-merchant prep priors hourly, because restaurant speed doesn't change second by second but does vary across lunch versus dinner, weekdays versus weekends. The training loop retrains daily on joined feature and outcome logs from the last 30 days.

The state transitions matter for when to re-score. A single delivery might be scored 200 to 400 times across its lifetime: once at order submission, a few hundred times during courier dispatch, and then every 10 to 30 seconds from pickup through drop-off. Each of those scorings uses different features because more is known as time progresses, and the model needs to handle the changing feature availability gracefully.

Sources

The signal for ETA comes from five layers: the physical road network, real-time state of that network (traffic, incidents), the merchant side of the transaction, the courier side, and the global context that shifts all of them together (weather, events, time of day).

The courier GPS stream is the backbone. Without it, segment-level speeds would degrade to public traffic feeds that lag by 10 to 30 minutes and miss the micro-patterns that show up in delivery-specific corridors (shortcut alleys, pedestrian zones bikes can cut through). The pings are aggregated up to segment-level speed with privacy-preserving windowing: no individual courier's position is exposed downstream, and any segment with fewer than 3 distinct couriers in the window falls back to historical priors rather than revealing per-courier data.

Merchant order history is the second most important input and the most under-appreciated. A pizza place that takes 22 minutes to prepare an order on Friday nights but 9 minutes on Tuesday afternoons is the single biggest variance contributor in a city with predictable traffic. The prep-time aggregator rolls up order history with explicit breakouts by day-of-week, hour-of-day, order size, and menu category, smoothed against a global prior so new merchants with thin history don't blow up the tail.

Feature Engineering

Features group into six categories based on the layer of signal they come from and their freshness characteristics. The split matters because different categories have different failure modes and different update cadences.

Route features describe the physical path. Total distance (great-circle and driving), number of segments, count of turns, count of traffic signals, max road-class level along the route, presence of limited-access segments (highways, bridges, tunnels). These come directly from the routing service and are effectively free to compute per request.

Traffic features describe the current state of the road network. Per-segment speed relative to historical baseline, aggregated along the route to produce a weighted "traffic coefficient," incident flags on segments on the route, speed-of-traffic deceleration (a segment that was clear 5 minutes ago and is now slow), and distance to the nearest active incident. These are the most volatile features and have the highest correlation with same-day accuracy.

Temporal features encode when the trip happens. Hour-of-day bucketed into 30-minute bins, day-of-week, whether it's a meal peak window (lunch 11:30-13:30, dinner 18:00-20:30), holiday flag, school-in-session flag. Temporal features are cheap, stable, and carry strong signal because congestion patterns are deeply periodic.

Merchant features capture the dwell-time layer. Historical median prep time, tail percentiles (p90, p95), prep-time volatility, current order queue depth at the merchant (how many pending orders they're already preparing), recent confirm-time accuracy (how close their estimates track reality), and order complexity features (item count, custom instructions, menu category).

Courier features describe the entity traveling. Vehicle type, historical median speed on equivalent road types, tenure bucket, time since last delivery completion (warm versus cold), current location relative to route start, active batching state (is this courier already carrying another order?).

Weather and event features are the global context. Precipitation type and intensity, temperature, wind speed, visibility, and flags for ongoing events (stadium events within 2km of the route). Weather features are one-dimensional but high-leverage: a rainy Friday at 6pm in Manhattan has distributional properties unlike any other day, and omitting rain-hour interactions produces a model that fails predictably on the worst traffic days.

The full feature set is closer to 150 to 250 features once you include interactions and segment-level vectors, but the table above captures the representative types. The segment-level speeds are themselves a variable-length vector because routes have variable segment counts; handling that in the model is a key design choice covered in the deep dives.

Labels and Fast Feedback

Unlike classification problems where labels arrive slowly, ETA labels are fast: the drop-off confirmation fires within seconds of trip completion, and the actual elapsed time is just a subtraction. This is one of the reasons ETA systems can retrain aggressively, on timelines of hours rather than weeks.

There are still label quality issues to handle. Some drop-offs are confirmed by the courier on arrival (clean label), others are auto-confirmed after a timeout because the courier forgot to tap the button (label has up to a minute of spurious inflation). The training pipeline filters out auto-confirms with a noisy-label flag and either drops them or weights them down. Trips that get cancelled mid-flight become negative examples for cancellation models but are excluded from ETA training because they never completed.

A subtler issue is that the distribution of actual durations is censored: late trips sometimes get cancelled, picked up by a second courier, or manually resolved, and those don't appear in the training data. Censoring biases models toward optimism because the worst outcomes are missing from the training set. The fix is to retain censored trips in the training data with a censoring indicator, treating the censoring time as a lower bound on the true duration, and using a censoring-aware loss like a Tobit regression or a modified quantile loss that treats censored examples appropriately for the upper quantiles.

Baseline: GBDT with Quantile Loss

The baseline is a LightGBM regressor trained with quantile loss at three quantiles (p50, p70, p90) over the feature set above, producing three separate boosters whose outputs form the predicted distribution. Quantile loss is the pinball loss: for quantile q, the loss on residual r = y - y_hat is q * max(r, 0) + (1 - q) * max(-r, 0), which pulls the prediction toward the qth percentile of the conditional distribution.

Training runs per leg: a pickup-drive model, a dwell model, and a drop-off-drive model, each with its own feature selection appropriate to that leg. The pickup-drive and drop-off-drive models share most route and traffic features; the dwell model uses merchant and order features with almost no routing input. Output quantiles are summed across legs to produce the total delivery distribution.

The baseline is not a toy. In a mature production system, a well-tuned per-leg quantile GBDT reaches within 5 to 10% of the best deep-learning architectures on aggregate MAE, trains in under an hour, serves in under 5ms per prediction, and is easy to monitor and debug. The gap between the baseline and the advanced model is concentrated in tail accuracy (p95 and beyond) and in slices where the feature engineering budget is the bottleneck. For many platforms, the GBDT is the production model for years before deep models earn their place.

Advanced: Leg-Wise Stack with Road-Network Encoder

The production architecture in mature platforms combines three model types: a road-network encoder that turns a route into a vector, a sequence model that integrates temporal dynamics, and a mixture density head that emits a distributional prediction. Each leg has its own stack, and the stacks share the road-network encoder weights.

The road-network GNN encoder takes the route as a sequence of segment IDs plus each segment's static attributes (length, road class, speed limit) and real-time attributes (current speed, incident flag). A two-layer GraphSAGE aggregates neighborhood features to produce a per-segment embedding that captures local network structure. At scale (hundreds of millions of segments globally), the GNN uses neighborhood sampling just like the fraud-detection graph: at each layer, each segment attends to a sampled subset of its neighbors rather than the full graph. The embedding captures patterns like "this segment connects two high-speed arterials" or "this segment has three traffic signals within 500m" that per-segment features miss.

The Transformer layer takes the sequence of per-segment embeddings plus a pooled route context (total distance, turn count, temporal features) and produces a fixed-size route representation. The attention mechanism is where the model learns that the first few segments matter more than the later ones for near-term predictions (because traffic upstream of the courier is less relevant than traffic on the segment they're about to enter). A couple of transformer blocks is enough; deeper stacks start overfitting to frequent routes.

The prep-time sub-model is a separate small GBDT or MLP operating on merchant and order features. Dwell time has almost no dependency on the road network, so training it inside the main stack wastes capacity and dilutes gradients. Running it as a side model with independent features is cleaner and lets the merchant ops team iterate on it without retraining the big model.

The mixture density head on each leg emits parameters of a mixture of three Laplace or Gaussian distributions. Laplace is the more common choice because ETA residuals have heavier tails than Gaussian, and the Laplace likelihood is less punishing on rare large errors. At serving time, the quantiles are computed analytically from the mixture parameters, and the three leg distributions are convolved to get the total delivery distribution (or approximated by summing means and variances with a correction for correlation, which is good enough in practice).

Training

Training data is 30 days of completed deliveries joined to feature snapshots at the time of each scoring event. The joining is non-trivial: a single delivery has many scoring events (order time, dispatch, mid-trip updates), and for each event the feature values were whatever they were at that moment. Training on the order-time feature snapshot against the final duration is the canonical setup; training on mid-trip snapshots against the remaining duration is used for the live-update model, which is typically a separate head that shares the encoder.

The loss is a combination of quantile loss (for the quantile heads in the baseline or as an auxiliary loss for the advanced model) and the negative log-likelihood of the mixture density model (for the primary distributional head). A weighting factor balances the two so that training optimizes both the quantile-specific calibration and the full distributional fit.

Temporal splits matter for the same reason they matter in fraud: random shuffling leaks future patterns into training. The convention is to train on weeks 1 through 4, validate on week 5, and test on the most recent complete week. A rolling walk-forward variant is used for final benchmark numbers.

Class-imbalance issues are milder than in fraud because ETA isn't imbalanced in the positive-class sense, but the training distribution is imbalanced across contexts. Lunch-hour peaks are 4 to 6 times denser than 3am, rain-hours are rare events, and the worst-traffic cities generate a disproportionate share of hard examples. Upweighting rare-context examples (rain, meal peaks, extreme distance) brings their loss contribution in line with their operational importance, though it comes at some cost to aggregate MAE because easy examples are now relatively under-weighted.

Feature store serving requires training-serving consistency: the feature values seen during training must match the values served at prediction time. The standard defense is a point-in-time join that replays the feature store's log to reconstruct what each feature's value was at each historical scoring event, rather than joining against the current value of each feature.

Inference Pipeline

The serving path at dispatch time looks like this.

Routing runs first because most features depend on the route. The routing service is CPU-intensive and often cached: for the same origin-destination pair within a short window, the cached route is reused unless a new incident appears on it. Cache hit rates at peak run around 35 to 50%, which cuts the average routing cost significantly.

Feature fetch is the latency choke-point. The service issues parallel lookups to the segment-speed store (one lookup returns speeds for all segments on the route in a batched call), the merchant feature store, the courier profile cache, and the weather store. The request is assembled into a dense feature vector with a freshness bitmap indicating which features came from cache misses or stale entries, and the model has seen training examples with the same freshness patterns so it can implicitly downweight stale features.

Model scoring runs the three leg models in parallel. The pickup-drive and drop-off-drive models share the GNN encoder on their respective routes, but because the two routes are different (different segments, different traffic state), the encoder runs twice. Batching across concurrent dispatch requests is how throughput scales: a single GPU can score 200 to 500 route predictions per forward pass if batching is done right, and at 50K req/s the dispatch path needs aggressive micro-batching to stay in budget.

Latency Budget

The 150ms p99 dispatch budget breaks down like this.

The buffer is deliberately large because both routing and feature fetch are bimodal. Routing median is 8ms but cold-cache routes (new destinations, rerouted trips) can take 40ms. Feature fetch median is 12ms but fleet-wide Redis shards occasionally hit gc pauses that push individual requests to 60ms. A smaller buffer would make the SLA brittle under load.

Batch vs Real-Time

The dispatch-path scorer is hard real-time, so are the mid-trip refresh scorings. The segment-speed aggregator is streaming (seconds-to-minutes windows), and the merchant prep-time aggregator is hourly batch because merchant behavior doesn't drift faster than that. Model retraining is daily.

The interesting near-real-time component is the live-update scorer. During an active trip, the courier's GPS pings every 500ms to 2 seconds, but re-scoring ETA on every ping would be wasteful (traffic doesn't change that fast, and the customer only cares about updates of 30 seconds or more). The refresh policy is adaptive: re-score when the courier has traveled more than 100 meters, or when 30 seconds have passed, or when a major state transition fires (order ready at merchant, courier arrived at drop-off neighborhood). The resulting refresh rate is one scoring every 10 to 30 seconds per active trip, averaging to about 100K re-scorings per second across 800K concurrent trips.

Degraded Serving

When the model service is unavailable or feature fetch starts timing out, the fallback path is a lookup-table approach: segment-speed tables by hour-of-day and day-of-week, plus merchant prep priors, combined with a simple additive model. The fallback is not retrained on the fly and is roughly a stripped-down version of the 2018-era pre-ML ETA system, which is actually good news because it means the fallback is well-understood and battle-tested. Accuracy in fallback mode runs 20 to 30% worse on MAE than the full model, which is painful but much better than failing open.

Like fraud, the fallback is exercised weekly in a chaos-engineering drill. A single region's scorer is taken offline for 5 minutes, and the degraded-mode lateness rate, courier utilization, and customer complaint rate are measured against a baseline. Fallback paths that aren't exercised rot, and the first time a broken fallback shows up in a real outage is a bad day.

6.1 Route Representation: Segment-Based vs Graph-Based

The choice of how to represent the route to the model has outsized influence on both accuracy and infrastructure cost. The two main approaches are segment-based encoding and graph-based (GNN) encoding, and they sit at different points on a cost-accuracy curve.

In the segment-based approach, the route is flattened into a vector of hand-crafted features: total distance, count of high-class segments, count of signals, average speed along the route weighted by segment length, maximum current traffic slowdown, and so on. The GBDT eats this vector directly. Feature engineering is the development work, and each new feature is a code change plus a backfill. The approach is cheap (feature computation is linear in route length, and no GPU is needed) and interpretable (SHAP values show exactly which route attribute mattered most).

In the graph-based approach, the route is kept as a sequence of segment IDs, and the model runs a GNN over the road network to produce per-segment embeddings that get pooled into a route representation. The embedding captures structural information about each segment's context: how many high-class segments are nearby, how well-connected the segment is, whether it sits on a frequent shortcut route. These are hard to hand-engineer because they involve two-hop or three-hop reasoning about the graph.

The honest answer is that the GNN's edge is concentrated in three situations: routes through under-represented geographies, novel segments that haven't been in training data, and long routes where two-hop context matters more than single-segment features. For the middle 80% of routes on well-travelled corridors, a GBDT with segment-based features matches or beats the GNN on MAE while running at a fraction of the cost.

Most mature platforms end up running both, not as a choice but as a stack. The GBDT handles the high-volume fast path, and the GNN feeds embeddings into the GBDT as additional features or scores a fraction of traffic where the segment-based features are known to be weak (new geographies, atypical routes). The hybrid captures the GNN's generalization benefit without paying its serving cost on every request.

6.2 Uncertainty Quantification

Showing the customer a single number for ETA is a product choice that hides real uncertainty, and it's the wrong choice on rainy days, peak meal times, and for long-distance deliveries where variance is naturally larger. Designing the system to produce a calibrated distribution and then making product decisions about how to surface it is fundamentally better than training a point predictor and hoping the mean is useful.

Three approaches to distributional prediction cover the practical space.

Quantile regression trains separate models (or separate heads on a shared model) for each target quantile using the pinball loss. It's the simplest approach, works well with GBDTs, and is easy to calibrate per quantile. The downside is that the quantiles aren't guaranteed to be monotonic (p70 could be below p50 on an individual prediction), and the distribution is only represented at the quantiles you trained on.

Mixture density networks (MDN) train a neural network whose output parameterizes a mixture of parametric distributions (typically Laplace or Gaussian). Any quantile can be computed analytically from the mixture parameters, and the distribution is fully specified rather than just sampled at known quantiles. MDNs are more expressive but trickier to train: the NLL loss can be unstable, and mode collapse (where all mixture components learn the same mean) is a real risk without careful initialization and regularization.

Conformal prediction wraps any point predictor and produces a calibrated interval by calibrating against a held-out set. The interval is guaranteed to have the right coverage marginally, independent of the underlying model's quality. The downside is that the intervals are typically wider than model-specific intervals, because conformal doesn't exploit the model's confidence signals.

The production choice for ETA at scale tends to be quantile regression for the baseline and MDN for the advanced model, with conformal used as a post-hoc calibrator to correct any residual miscalibration. Ensembles are valuable offline for understanding where the model is uncertain and where disagreement matters, but they rarely survive to production as live-serving paths because the cost multiplies linearly with ensemble size.

Once the model produces a distribution, the product decision is which point to surface. Showing p50 means half of deliveries run late against the ETA, which feels like broken promises. Showing p90 produces obviously padded ETAs that depress order conversion. The operational sweet spot varies by product; most food-delivery platforms land between p65 and p75 for the customer-facing ETA, with the window adapted dynamically based on observed lateness rates. The same model powers dispatch (which uses p50 for optimal courier selection) and customer messaging (which uses p65+), which is only possible because the model emits a full distribution rather than a single number.

6.3 Multi-Leg Prediction and Dwell Time

The naive framing of "predict total delivery time" as a single regression target is worse than the leg-wise decomposition in several ways, and understanding why is the heart of the ETA design problem.

First, the legs have very different feature dependencies. Pickup drive depends on the road network, traffic, and courier behavior. Dwell depends on the merchant's current order queue, the order's complexity, and the menu items. Drop-off drive depends on a different road network (different starting point), different traffic (potentially in a different neighborhood), and customer-side features like building access complexity. A single model trying to learn all three either wastes capacity on feature interactions that don't apply, or develops gated components that effectively reimplement leg decomposition internally, but with less interpretability.

Second, the legs have very different noise characteristics. Drive-time variance is roughly proportional to driving distance: a 2km drive has standard deviation around 45 seconds, a 20km drive has standard deviation around 5 minutes. Dwell-time variance is dominated by the merchant and has a heavy tail unrelated to distance: some restaurants have a standard deviation of 2 minutes, others have 12 minutes because they batch orders inconsistently. Summing two noise sources that live in different regimes requires knowing them individually to do variance propagation right.

Third, iteration speed matters. The merchant ops team wants to improve prep-time prediction by adding features specific to the restaurant side: kitchen size, menu complexity, staffing levels. Those features don't belong in the drive-time model. Keeping the two models separate means the merchant team can retrain the dwell model on its own cadence without regression testing against the drive-time model, which is a major productivity multiplier.

The dwell-time model itself deserves focused attention because it's the component with the highest headroom in most ETA systems.

Dwell decomposes into three sub-intervals: acceptance lag (order placed to merchant acceptance), prep time (acceptance to ready), and hand-off lag (ready to courier pickup). Each has distinct drivers. Acceptance lag depends on whether the merchant is using the tablet actively; prep time depends on the kitchen; hand-off lag depends on courier arrival timing relative to food readiness. A single dwell model predicts the whole interval conditional on features from all three sub-sources, but some platforms decompose further and predict each sub-interval separately when the data supports it.

The merchant-prep sub-model is typically a small GBDT that takes merchant features (rolling prep-time percentiles conditioned on day and hour), order features (item count, menu categories, customizations), and current-queue features (how many pending orders this merchant is already preparing). The last one is the single biggest feature in the sub-model: a merchant with 2 pending orders at 6:30pm is fundamentally different from the same merchant with 12 pending orders. Adding queue depth as a feature typically cuts prep-time MAE by 15 to 25% over a model that ignores it.

One subtle trap in dwell modeling is that the prep-time label is right-censored by courier arrival. If the courier arrives before the food is ready, the observed "ready time" is actually "hand-off time," which over-states prep time. If the food is ready before the courier arrives, the observed ready time is accurate. Training on the observed hand-off timestamp without adjustment biases prep-time predictions upward by 10 to 30 seconds in peak hours. The fix is to have merchants tap a "ready" button at the actual ready moment, which gives a clean label but depends on merchant compliance, or to model hand-off lag separately and subtract it, which is more robust but requires both signals.

6.4 Mid-Trip Refinement and Adversarial Dynamics

The initial ETA at order time is a prediction made from limited information: the courier hasn't been dispatched yet, traffic on the specific route hasn't been sampled from this courier's perspective, and the merchant's current queue depth is an estimate. As the trip progresses, real information accumulates, and the live ETA needs to integrate that information gracefully.

The refinement policy is conceptually simple. Every 10 to 30 seconds, re-run the scorer with the latest features, which now include the courier's current position (updating the remaining pickup or drop-off distance), the latest segment speeds on the upcoming route, and any state transitions that have fired (merchant accepted, food is ready, courier arrived at merchant). The output is a new ETA distribution, and the customer sees a refreshed countdown.

The subtlety is how to handle large updates. If the new ETA is 3 minutes later than what the customer was shown 10 minutes ago, displaying the correction feels like breaking a promise. If the new ETA is 5 minutes earlier, the customer might not be ready to receive the food. Most platforms damp the displayed ETA with a moving average so it only steps in small increments (at most 60 seconds per minute of wall-clock), while the internal dispatch ETA remains the raw model output. The damped display is a product layer, not a model layer, but it has to be designed together with the model because the damping imposes a constraint on how fast the underlying model can change without violating the display policy.

A separate concern is adversarial behavior within the fleet. Couriers are rewarded (explicitly through pay, implicitly through ratings) for fast deliveries, and if the ETA model becomes part of the reward function, couriers have incentives to drive in ways that game the model rather than deliver fast. The classic example is a courier who detours through a fast back road but doesn't log through the app's map, making it look like they drove a longer nominal route in less time. If the model treats this as "this courier is fast on suburban routes," it might preferentially dispatch them on routes where the shortcut doesn't exist, producing worse outcomes.

The defense is to decouple courier-specific features from the operational model. Courier speed priors are useful but should be computed from GPS-observed speeds on common segments relative to the fleet average, not from observed total trip times. A courier who is reliably 10% faster on suburban residential streets gets a positive prior for residential-heavy routes; a courier whose total trip times are 10% below average without a segment-level explanation doesn't get a prior adjustment because the signal could be coming from unobserved shortcuts or from cherry-picking easy trips.

Related is the feedback loop between ETA and dispatch. Dispatch picks couriers based on predicted pickup time, which means couriers closer to the restaurant are preferred, which biases the training data toward "close couriers" and might under-sample training examples of longer pickups. The fix is random exploration: a small fraction of dispatches (typically 1 to 3%) override the dispatch decision with a randomized pick that includes couriers further from the restaurant. Those dispatches are more expensive in pickup time but generate training data that keeps the model calibrated across the full distance distribution. Without the exploration, the model slowly becomes biased toward short pickups and its error on long ones grows until the dispatcher starts mispicking even under normal operation.

Offline Evaluation

Offline evaluation uses temporal holdout (train on weeks 1 through 4, validate on week 5, test on the most recent complete week) with three headline metrics: MAE, quantile loss at the served quantile, and coverage of the 80% prediction interval. Aggregate numbers hide the interesting regressions, so slice evaluation is mandatory across city, hour-of-day band, distance band, weather condition, and distance from the training geography.

The specific slices that pay for themselves are meal peaks versus off-peak (peaks have 2-3x higher variance and different feature sensitivities), rain versus clear weather (traffic patterns are discontinuously different), long-tail distance bands (routes beyond 10km are rare but produce the biggest absolute errors), and new-merchant bucket (merchants with under 100 historical orders have unstable prep-time priors and often regress on new model versions).

A candidate model promotes only when it beats the champion on aggregate MAE and on every major slice within a tolerance of 2%. Strict slice gates sometimes block genuinely better models that regress modestly on one slice while improving many others, which is the right trade-off because slice regressions manifest as localized customer-experience issues that take months to rebuild trust after.

Calibration evaluation runs on the distributional outputs. The empirical coverage of the 80% interval should be 80% across the full test set and within 2 percentage points across each major slice. Deviation patterns tell a specific story: systematic under-coverage means the model is overconfident on uncertain cases, systematic over-coverage means the intervals are too wide and the customer-facing ETA is more padded than it needs to be. Both are fixable, but they require different root-cause investigations.

Online Evaluation

Online evaluation uses A/B testing at the dispatch-unit level. The assignment unit is the dispatch request (courier-order pair) rather than the user or the order, because the ETA model's immediate effect is on dispatch decisions and downstream effects (lateness, customer complaints) propagate from there. Running an A/B at the order level would conflate dispatch changes with subsequent ETA refreshes.

The primary A/B metrics are lateness rate (fraction of deliveries exceeding customer ETA by 5+ minutes), over-promise rate (fraction where actual time exceeds ETA at all), dispatch quality (fraction of dispatches where the chosen courier was within 2 minutes of the predicted pickup time), and conversion rate at checkout (catches over-padded ETAs). The guardrail metrics are customer complaint rate, courier utilization, and serving latency.

The rollout ramp is faster than for fraud because the feedback is fast. A typical rollout starts at 1% of dispatches, moves to 10% after 48 hours if guardrails are clean, then 50% after 72 hours, then 100% after another week. Rollbacks are triggered by lateness rate exceeding the champion's by more than 10 relative percent or by any guardrail metric regressing past its pre-set threshold.

The trickiest measurement problem in online ETA evaluation is long-term effects. A model that over-pads ETAs today might show no short-term complaint-rate change but slowly erodes conversion over weeks as customers compare against competitor platforms. Guardrails on conversion have to run on multi-week windows, which slows the decision cycle for changes that could affect it. The standard compromise is to ramp to 50% and hold for 4 weeks rather than 1 before final promotion, paying a slower rollout for a more reliable conversion signal.

Monitoring

Training-serving skew is the monitor that earns its keep most often, just like in fraud. A feature that's mean zero in training but mean 0.05 at serving silently shifts every prediction by a small amount, and since ETA has no single hard threshold (unlike fraud's decline), the error hides in aggregate MAE for days. The defense is a shadow recomputation job that reproduces serving-time features from logged raw inputs and compares them to what the online feature store served. Any top-20 feature diverging by more than 1% pages the on-call.

Drift in the lateness rate itself is the business-level signal that everything else rolls up to. When lateness spikes, the triage question is always "is this a model regression, a feature-pipeline outage, or an underlying operational shift?" Model regressions and pipeline outages show up in the training-serving skew monitor and the feature freshness monitors before lateness moves. Genuine operational shifts (a city just added 500 couriers, a weather event is causing widespread delays) show up in the feature distribution monitors and in raw operational metrics (dispatch volume, courier utilization). Distinguishing the two categories quickly is what keeps an incident from becoming an outage.

Retraining cadence is daily for the full stack and hourly for incremental updates to the segment-speed encodings. The daily retrain takes about 4 hours end-to-end on a mid-sized GPU cluster and another 2 hours for shadow evaluation before promotion. Emergency retrains (novel geographic rollouts, sudden infrastructure changes) run on a shortened 90-day window of recent data and a compressed 4-hour shadow.

Q1: Why predict the legs of a delivery separately instead of training one model to predict the total delivery time?

The legs have different feature dependencies, different noise characteristics, and different iteration cadences, and combining them into one model forces compromises on all three. Pickup and drop-off drive times depend on the road network and traffic but not on merchant features; dwell time depends on the merchant and the order but not on the road network. A single model either wastes capacity learning feature interactions that don't apply or develops internal gating that effectively reimplements leg decomposition with worse interpretability. Drive-time variance scales with distance while dwell-time variance is dominated by the merchant; summing the two correctly for uncertainty propagation requires modeling them individually. The separate-model design also lets the merchant ops team iterate on prep-time prediction without regression-testing against the drive-time stack, which is a major productivity gain.

Q2: How do you quantify uncertainty in ETA predictions, and why does it matter beyond just showing the customer a range?

Quantile regression, mixture density heads, and conformal prediction are the three workhorse approaches. Quantile regression trains separate heads at each target quantile using the pinball loss and is the simplest approach on tabular models. Mixture density networks output parameters of a mixture of Laplaces or Gaussians and produce fully distributional predictions, but they're trickier to train. Conformal wraps any point predictor and guarantees marginal coverage. Uncertainty matters beyond the customer display because dispatch uses the median for optimal courier selection, pricing uses the mean for unbiased expected-cost calculations, and customer-facing UX uses a higher quantile (p65 to p75) to balance lateness complaints against over-padding. A single point predictor forces all three consumers to share one number, which necessarily compromises at least one of them. A distributional output lets each consumer pick its own point without retraining.

Q3: When is a GNN over the road network actually worth it versus a GBDT with hand-crafted route features?

A well-tuned GBDT with engineered features (total distance, signal count, road-class distribution, route-weighted traffic speed) captures most of the signal on common routes and beats a GNN on cost by an order of magnitude. The GNN earns its place in three specific situations: routes through under-represented geographies where training data is sparse, novel segments that haven't been seen in training, and long routes where two-hop graph context (what's around the segments the courier is about to enter) matters more than single-segment features. In production at a mature platform, the two coexist: the GBDT handles high-volume common routes with sub-5ms serving, and the GNN either feeds embeddings as additional features into the GBDT or scores a fraction of traffic where the segment-based features are known to be weak. The GNN rarely replaces the GBDT outright because the cost-accuracy trade-off doesn't justify it on the bulk of traffic.

Q4: How do you prevent a feedback loop between the ETA model and the dispatch system from degrading training data?

Dispatch picks couriers based on predicted pickup time, which biases the training data toward short-pickup trips and under-samples examples where the courier was further from the restaurant. Over time the model's error on long pickups grows because it's rarely seeing them, and eventually the dispatcher starts mispicking even under normal operation. The standard fix is random exploration: 1 to 3% of dispatches override the ranked choice with a randomized pick that includes couriers further from the restaurant. Those trips are more expensive in pickup time but generate training examples across the full distance distribution, keeping the model calibrated. A related concern is courier-specific features: a courier whose total trip times look fast without a segment-level speed explanation might be cherry-picking easy trips or using off-map shortcuts, so courier priors are computed from GPS-observed segment speeds relative to the fleet average rather than from total trip durations.

Q5: Why is temporal censoring a problem in ETA training data, and how do you handle it?

Deliveries that run very late sometimes get cancelled, reassigned, or resolved manually, and those trips don't appear in the completed-delivery training set. The observed distribution is censored on the right tail: the worst outcomes are systematically missing. Training on the censored distribution biases the model toward optimism because the examples that would pull the upper quantiles to the right have been removed. A second form of censoring affects dwell time specifically: prep-time labels are bounded above by courier arrival time, so trips where the courier arrives early produce artificially inflated "ready" timestamps. The fixes are censoring-aware losses (Tobit regression or a modified quantile loss that treats censored examples as lower bounds on the true duration), explicit "ready" signals from the merchant to de-censor prep time, and retaining censored trips with a censoring indicator so the model can learn to handle them rather than just dropping them. Ignoring censoring is one of the most common reasons an ETA model's p90 quantile is systematically under-coverage.

Next we turn to notification relevance, which shares ETA's concern for user trust but flips the design problem: instead of predicting a quantity, the system decides whether to send a message at all and when, trading user engagement against the risk of notification fatigue.