Year: 2025
Type: preprint
Abstract: Data is a valuable asset, and sharing it as a product across organizations is key to building comprehensive and useful insights in fields such as science and industry. Before sharing, data often requires transformation to comply with governance policies and meet the requirements of recipient organizations. By leveraging pipelines, these transformations can be modeled as chains of processes; however, designing such pipelines while ensuring their efficiency is complex. In this paper, we present a tool that supports the design of pipelines by identifying opportunities for reusing transformation processes across different pipelines and suggesting designs and configurations based on these opportunities. This tool also generates reports on the resource consumption of pipeline processes, enabling the estimation of potential resource savings achievable through reuse-based designs. It could serve as a foundation for more efficient and resource-conscious data transformation pipeline design and be used as a component in self-service data platforms.
Source: arXiv (Cornell University)
Author: Masoudi, Sepideh
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6979356647
Cites:
Cited by:
Related to: