Green but Wrong: A Naive Airflow Pipeline Design Failed All 121 Write-Path Fault Trials, a Hardened Design Failed None of 133
Year: 2026
Type: preprint
Abstract: An orchestrator can report success while the data it produced is wrong. Apache Airflow executes tasks at least once: a task can be killed after writing half its rows, retried, triggered twice for the same data interval, or re-run after late data arrives, and in each case the DAG run can end green while the sink holds duplicate or stale records. This study measures that gap directly, under a pre-registration frozen before any experiment code existed. On a real local Airflow 3.4.0 deployment (LocalExecutor, SQLite metadata database), five ML pipeline types (ingestion, embedding, evaluation, retraining, monitoring) ran in two arms that differ only in their write and read pattern. The naive arm (append inserts, commit per batch, latest-data reads, wall-clock keys) is the design a straightforward implementation produces. The hardened arm applies three documented design patterns: idempotent sinks with upsert on deterministic keys, one transaction per write task, and fixed-partition reads and writes. Six fault types were injected across 305 logged trials, and after every trial an independent checker re-derived the expected sink content from the corpus specification and counted duplicates, losses, and content mismatches. The naive arm failed every write-path fault trial: 50 of 50 kill-mid-write, 50 of 50 partial-write, 15 of 15 duplicate-trigger, and 6 of 6 late-data trials, 121 of 121 in total, producing 4,060 duplicate records in the kill and partial-write trials alone. Every one of the 100 failed naive kill and partial-write trials still ended in DAG run success: green-but-wrong in 100 of 100 cases. The hardened arm failed zero of its 133 fault trials; pooled over its 50 kill and 50 partial-write trials, the one-sided Clopper-Pearson 95 percent upper bound on its failure rate is 0.0582 per fault. A scheduler restart corrupted neither arm, and a mid-run code deploy corrupted neither arm when the DAGs shipped in a versioned Git bundle: all six Git-bundle runs executed under exactly one code version, while one exploratory run under an unversioned local bundle mixed two versions inside a single run. Hardened recovery from a write fault cost exactly one extra task execution in 100 of 100 trials, with a pooled median recovery time of 6.97 seconds. The failure mode throughout was duplication, not loss: lost records were zero in all 305 trials. All four pre-registered hypotheses were supported. The claims are bounded by the design: SQLite sinks, one machine, a deterministic embedder stand-in, and small trial counts on the platform-level faults, each reported with its exact upper bound.
Author: Ushno, Jesun Ahmad
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7236178186
