The joinless partition pattern: a spark-native physical execution pattern for multi-dimensional aggregations in apache spark
Year: 2026
Type: article
Abstract: Large-scale data aggregation pipelines in Apache Spark typically require multiple join operations to enrich fact tables with dimensional attributes before computing aggregates. Each join induces a costly shuffle operation that redistributes the entire fact table across the cluster. This paper introduces the Joinless Partition Pattern , a novel optimization technique that eliminates intermediate join shuffles by leveraging broadcast variables and partition-local lookups within mapPartitions. We demonstrate that for workloads involving multiple small-to-medium dimension tables (10MB--1GB each), our approach achieves 3--12× speedup and 20--400× reduction in shuffle volume compared to traditional join-based approaches. We provide formal correctness proofs, complexity analysis, and experimental validation on both synthetic benchmarks and three production workloads running on Azure Databricks. Our results show consistent performance improvements across varying data scales and cluster configurations, with particularly strong benefits for aggregation-heavy analytical queries. The pattern has been deployed in production for 6+ months with demonstrated annual savings of \$15,000--\$20,000 in compute costs.
Author: Rubén Alejandro Jaime
Institution: Universidad Nacional de La Plata
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Keywords: Apache Spark, dimension tables, analytical queries
Open Access status: green
OpenAlex ID: W7125517975
