Parallel processing optimization strategy based on MapReduce model in cloud storage environment
Year: 2017
Type: conference-paper
Abstract: Currently, a large number of documents in the cloud storage process employed the way of packaging after receiving all the packets. From the local transmitter this stored procedure to the server, packing and unpacking will consume a lot of time, and the transmission efficiency is low as well. A new parallel processing algorithm is proposed to optimize the transmission mode. According to the operation machine graphs model work, using MPI technology parallel execution Mapper and Reducer mechanism. It is good to use MPI technology to implement Mapper and Reducer parallel mechanism. After the simulation experiment of Hadoop cloud computing platform, this algorithm can not only accelerate the file transfer rate, but also shorten the waiting time of the Reducer mechanism. It will break through traditional sequential transmission constraints and reduce the storage coupling to improve the transmission efficiency.
Source: AIP conference proceedings
DOI: 10.1063/1.4982581
Authors: Jianming Cui, Jiayi Liu, Qiuyan Li
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Keywords: MapReduce, cloud storage, parallel computing, MPI, Hadoop
Open Access status: bronze
OpenAlex ID: W2613990120
