Optimizing PySpark Jobs for High-Performance Large-Scale Data Processing
Year: 2026
Type: article
Abstract: Processing very large datasets efficiently is a major challenge in modern data engineering. Apache Spark is widely used for distributed data processing, but poorly tuned PySpark jobs often lead to long runtimes and inefficient resource usage. This paper presents practical optimization techniques applied to a real-world PySpark job processing approximately 500 million records. By improving partition strategy, tuning executor resources, optimizing joins, and minimizing shuffle operations, the job runtime was reduced from around 90 minutes to 55–66 minutes. The study demonstrates how practical and simple tuning methods can significantly improve performance in production-scale Spark workloads.
Author: Parashar, Sachi
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7130947605
FWCI:
Cites:
Cited by:
Related to: