Year: 2025
Type: article
Abstract: As enterprises transition from on-prem Hadoop to cloud-native platforms like AWS EMR, ensuring performance parity and operational efficiency is critical.This paper introduces a custom Shadow Testing Framework designed to compare production workloads across both environments.By replaying real queries from on-prem systems in EMR, the framework captures execution metrics, identifies configuration gaps, and supports parallel testing across multiple clusters.Key architectural shifts-such as moving to Tez, S3 storage, and enhanced SQL authorization-are addressed.Our results show that, with tuning, EMR can meet or exceed on-prem performance while delivering scalability, reliability, and cost benefits.This approach enabled a productiongrade EMR deployment supporting thousands of daily queries.
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4410166737