Year: 2025
Type: dissertation
Abstract: Computational reproducibility and replicability are essential pillars of credible research, especially as computational experiments become increasingly prevalent across diverse scientific domains.Despite strong advocacy for Open Science and adherence to FAIR (Findable, Accessible, Interoperable, and Reusable) principles, achieving true reproducibility remains a formidable challenge for many researchers.While openness enhances transparency and accessibility, reproducibility is a fundamental requirement of science itself-even in closed or proprietary research contexts, results must be independently verifiable to be considered scientifically valid.Key issues such as complex dependency management, insufficient metadata, and difficult access to essential code and data severely hamper reproducibility efforts.Moreover, existing reproducibility tools frequently offer piecemeal solutions that fail to address the multifaceted needs of diverse and complex experimental setups, particularly those that span multiple programming languages and dependencies.This thesis addresses these challenges by presenting a comprehensive framework designed to enhance computational reproducibility across multiple scientific domains.Our approach involved a systematic review of existing reproducibility tools to identify prevailing gaps and limitations in their design and functionality.This review underscored the fragmented nature of these tools, each supporting only aspects of the reproducibility process, yet none offered an integrated and robust solution.To bridge these gaps, we introduced SCIREP, an innovative framework that automates essential aspects of the reproducibility workflow such as dependency management, containerization, and cross-platform compatibility.This framework was evaluated using a curated dataset of 18 computational experiments from diverse scientific domains, successfully reproducing 94% of them.Furthering the accessibility and usability of reproducible research, we developed SCICONV, a conversational interface that simplifies the configuration and execution of computational experiments using natural language processing.This interface significantly reduces the technical barriers traditionally associated with setting up reproducible studies, allowing researchers to interact with the system through simple, guided conversations.Evaluation results indicated that SCICONV successfully reproduced 83% of the experiments in our curated dataset with minimal user input, demonstrating low cognitive workload and high usability.These results highlight its potential to make reproducible research more accessible to a broader range of researchers, regardless of technical background.Moreover, recognizing the critical role of user studies in evaluating tools, methodologies, and prototypes, particularly in software engineering and behavioral sciences, this thesis also extends into the realm of experimental tool evaluation.We conducted a thorough analysis of existing tools used for software engineering and behavioral science experiments, identifying and proposing specific features designed to enhance their functionality and ease of use for conducting user studies.These proposed features were validated through a survey involving 102 members of the research i ii community, confirming their relevance and the need for their integration into existing and future tools.The contributions of this thesis are manifold, encompassing the development of a classification framework for reproducibility tools, the creation of a standardized benchmark dataset for assessing tool efficacy, and the formulation of SCIREP and SCICONV to significantly advance the state-ofthe-art in computational reproducibility.
Author: Lázaro Gabriel Barros da Costa
Institution: Universidade do Porto
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6998403984
