Big data provenance: Challenges, state of the art and opportunities
Year: 2015
Type: conference-paper
Abstract: Ability to track provenance is a key feature of scientific workflows to support data lineage and reproducibility. The challenges that are introduced by the volume, variety and velocity of Big Data, also pose related challenges for provenance and quality of Big Data, defined as veracity. The increasing size and variety of distributed Big Data provenance information bring new technical challenges and opportunities throughout the provenance lifecycle including recording, querying, sharing and utilization. This paper discusses the challenges and opportunities of Big Data provenance related to the veracity of the datasets themselves and the provenance of the analytical processes that analyze these datasets. It also explains our current efforts towards tracking and utilizing Big Data provenance using workflows as a programming model to analyze Big Data.
Authors: Jianwu Wang, Daniel Crawl, Shweta Purawat, Mai Nguyen, Ilkay Altintas
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
Awards: ABI Development: bioKepler: A Comprehensive Bioinformatics Scientific Workflow Module for Distributed Analysis of Large-Scale Biological Data, AN OPEN RESOURCE FOR COLLABORATIVE BIOMEDICAL BIG DATA TRAINING, INTEGRATED END-TO-END PERFORMANCE PREDICTION AND DIAGNOSIS FOR EXTREME SCIENTIFIC WORKFLOWS (IPPD), Hazards SEES Type 2: WIFIRE: A Scalable Data-Driven Monitoring, Dynamic Prediction and Resilience Cyberinfrastructure for Wildfires, NATIONAL BIOMEDICAL COMPUTATION RESOURCE
OpenAlex ID: W2200241032
