Work

Design and Implementation of a Scalable Data Warehouse for Agricultural Big Data

Year: 2025

Type: article

Abstract: The rapid growth of agricultural data necessitates the development of storage systems that are scalable and efficient in storing, retrieving and analyzing very large datasets. The traditional relational database management systems (RDBMSs) struggle to keep up with large-scale analytical queries due to the volume and complexity inherent in those data. This study presents the design and implementation of a scalable data warehouse (DWH) system for agricultural big data. The proposed solution efficiently integrates data and optimizes data ingestion, transformation, and query performance, leveraging a distributed architecture based on HDFS, Apache Hive, and Apache Spark, deployed on dockerized Ubuntu Linux environments. This paper highlights the reasons why a DWH is irreplaceable for big data processing, without disputing the strengths of traditional databases in transactional use cases. By detailing the architectural choices and implementation strategy, this study provides a practical framework for deploying robust DWH solutions that are useful in supporting agricultural research, market predictions and policy decision-making.

Source: Sustainability

DOI: 10.3390/su17083727

Authors: Asterios Theofilou, Stefanos A. Nastis, Michail Tsagris, Santiago Rodriguez-Perez, Konstadinos Mattas

Institutions: Aristotle University of Thessaloniki, University of Crete, Idener (Spain)

Language: English

Topic: Big Data and Business Intelligence

Subfield: Management Information Systems

Field: Business, Management and Accounting

Domain: Social Sciences

SDG: Zero hunger

Open Access status: gold

Estimated APC paid: $2,896