Work

Where MapReduce Catches Up: A Pre-registered Scale Ladder, the Small-File Effect Measured at 1,251.7 Percent, and a Spark Hypothesis Refuted

Year: 2026
Type: preprint
Abstract: "Just use a script on one machine" and "you need a cluster" are both standard advice for batch document processing, usually given without numbers. This study pre-registered the numbers. On one named machine (a Dell PowerEdge R610), Hadoop MapReduce on HDFS (3.3.6, Docker pseudo-distributed cluster capped at 8 CPUs and 24 GiB) is compared with a tuned single-node baseline (same tokenizer, aggregation in DuckDB) on two document jobs over the English Wikisource corpus (648,182 documents, 1.85 GiB of extracted text), at 0.1 GiB, 1 GiB, and the full corpus. Every run's output had to hash identically to the baseline's before its time counted. Four hypotheses were frozen in advance. H1, scale crossover, is CONFIRMED: the MapReduce-to-baseline wall-time ratio for the inverted-index job falls from 2.288 at 0.1 GiB to 0.963 at the full corpus, where MapReduce is (narrowly) faster. H2, the small-file effect, is CONFIRMED at a magnitude far beyond its 15 percent pass line: at 1 GiB, storing the corpus as 1,055 files of at most 1 MiB instead of 9 files costs 1,251.7 percent of job wall time (median 7,737.209 seconds against 572.420), a factor of about 13.5. The cell had been stopped unmeasured in an earlier phase for practicality and was measured in a later phase under the owner's explicit authorization, with the deviation record public. H3, replication cost, is INCONCLUSIVE: replication factor 3 costs 29.5 percent on ingest write time but minus 0.1 percent on job time, failing the registered joint rule. H4 is REFUTED: Spark on YARN, the optional arm, is slower than MapReduce at this scale (median 1,087.0 against 954.5 seconds), not 30 percent faster. The study's claims are deliberately narrow: one machine, one corpus, one capped cluster. Within those bounds, every number was registered before it was run and every deviation is on the record.
Language: English
Open Access status: green
OpenAlex ID: W7233763838
Cites:
Cited by:
Related to: 10