Work

Erasure Coding Aware Block Placement for Data-Intensive Applications

Year: 2025

Type: conference-paper

Abstract: Erasure Coding (EC) has recently been integrated and deployed in the Hadoop Distributed File System (HDFS) to provide the same fault tolerance guarantees as replication, but with significantly less storage overhead. When EC is used, data reads typically involve only data chunks. In this paper, we study the effect of data chunk distribution on the performance of reads and data-intensive applications, and present the design and evaluation of an erasure coding aware (EC-aware) block placement that balances the distribution of data chunks across nodes. Experimental results show that EC-aware block placement can reduce the execution time of Sort and WordCount applications by up to 25%.

DOI: 10.1145/3719330.3721229

Authors: Shadi Ibrahim, Jad Darrous

Institutions: Centre National de la Recherche Scientifique, Institut national de recherche en sciences et technologies du numérique, Institut de Recherche en Informatique et Systèmes Aléatoires, Université de Rennes, Laboratoire des Sciences du Numérique de Nantes, IMT Atlantique

Language: English

Topic: Advanced Data Storage Technologies

Subfield: Computer Networks and Communications

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Funder: Agence Nationale de la Recherche

Award: Big Data Processing: Beyond Hadoop!

OpenAlex ID: W4408738458

FWCI:

Cites: 21

Cited by:

Related to: 10