Year: 2019
Type: article
Abstract: High-performance computing (HPC) and massive data processing (Big Data) are two trends that are beginning to converge. In that process, aspects of hardware architectures, systems support and programming paradigms are being revisited from both perspectives. This paper presents our experience on this path of convergence with the proposal of a framework that addresses some of the programming issues derived from such integration. Our contribution is the development of an integrated environment that integretes (i) COMPSs, a programming framework for the development and execution of parallel applications for distributed infrastructures; (ii) Lemonade, a data mining and analysis tool; and (iii) HDFS, the most widely used distributed file system for Big Data systems. To validate our framework, we used Lemonade to create COMPSs applications that access data through HDFS, and compared them with equivalent applications built with Spark, a popular Big Data framework. The results show that the HDFS integration benefits COMPSs by simplifying data access and by rearranging data transfer, reducing execution time. The integration with Lemonade facilitates COMPSs’s use and may help its popularization in the Data Science community, by providing efficient algorithm implementations for experts from the data domain that want to develop applications with a higher level abstraction.
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,085
Awards: MASWEB - MODELOS, ALGORITMOS E SISTEMAS PARA WEB, Adaptive, Trustworthy, Manageable, Orchestrated, Secure, Privacy-assuring, Hybrid Ecosystem for REsilient Cloud Computing, G434337789, G8345342992, EUrope-BRAzil Collaboration on BIG Data Scientific REsearch through Cloud-Centric Applications, G997093836
OpenAlex ID: W2980327568