Year: 2024
Type: article
Abstract: We present Codepickle, a groundbreaking solution designed to overcome Python version constraints and enhance portability, especially within distributed Python grids and volunteer computing environments. Unlike existing serialization libraries such as Cloudpickle and Dill, which rely on Python bytecode and are prone to version conflicts, Codepickle offers a robust alternative. Our methodology includes innovative adjustments for function serialization and shared variable management. Experimental results reveal challenges like code sourcing and nonlocal variable handling. Performance benchmarks highlight Codepickle's significant advantages over Cloudpickle, including better portability and reduced message sizes. Notably, Codepickle achieves message sizes that are 84% of those produced by Cloudpickle especially for small code segments, with comparable execution performance. Proposed enhancements target critical issues such as lambda functions and cross-version compatibility. This comprehensive study not only demonstrates Codepickle's transformative potential but also underscores the ongoing quest for advanced serialization techniques in Python's distributed computing landscape.
Authors: Ekasit Tangmunchittham, Krerk Piromsopa
Institution: Chulalongkorn University
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W4403994915