DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuning
Year: 2024
Type: preprint
Abstract: We propose a novel fine-tuning method to achieve multi-operator learning through training a distributed neural operator with diverse function data and then zero-shot fine-tuning the neural network using physics-informed losses for downstream tasks. Operator learning effectively approximates solution operators for PDEs and various PDE-related problems, yet it often struggles to generalize to new tasks. To address this, we investigate fine-tuning a pretrained model, while carefully selecting an initialization that enables rapid adaptation to new tasks with minimal data. Our approach combines distributed learning to integrate data from various operators in pre-training, while physics-informed methods enable zero-shot fine-tuning, minimizing the reliance on downstream data. We investigate standard fine-tuning and Low-Rank Adaptation fine-tuning, applying both to train complex nonlinear target operators that are difficult to learn only using random initialization. Through comprehensive numerical examples, we demonstrate the advantages of our approach, showcasing significant improvements in accuracy. Our findings provide a robust framework for advancing multi-operator learning and highlight the potential of transfer learning techniques in this domain.
Source: arXiv (Cornell University)
Language: English
Subfield: Statistical and Nonlinear Physics
Field: Physics and Astronomy
Domain: Physical Sciences
Open Access status: green
Awards: G1008116226, G1021149499, Collaborative Research: DMS/NIGMS 1: Understanding genomic organization in the nucleus via biophysical modeling, super-resolution imaging, and data-efficient operator learning, G2778948316, G3047343479, TRUSTWORTHY HETEROGENEOUS DATA-AWARE BAYESIAN FEDERATED LEARNING, G3723704744, Collaborative Research: Robust Deep Learning in Real Physical Space: Generalization, Scalability, and Credibility, Collaborative Research: Inference and Uncertainty Quantification for High Dimensional Systems in Remote Sensing: Methods, Computation, and Applications, TRUSTWORTHY HETEROGENEOUS DATA-AWARE BAYESIAN FEDERATED LEARNING, FEDNEMO: PHYSICS-INFORMED AND ENERGY-AWARE FEDERATED LEARNING OF NEURAL MULTI-OPERATOR LEARNERS AS SCIENTIFIC FOUNDATION MODELS, G6355769530, ASCENT: From sensors to multiscale digital twin to autonomous operation of resilient electric power grids, Elements: FourPhonon: A Computational Tool for Higher-Order Phonon Anharmonicity and Thermal Properties, ATD: Multimodal Transformer-based Model for Time-series Prediction and Spatiotemporal Analysis, Collaborative Research: Randomized Feature Methods for Modeling and Dynamics: Theory and Algorithms, CDS&E: Integrated Molecular Dynamics and Machine Learning Investigation of Gas-Liquid Interface in Transcritical Reacting Flows, G947038852
OpenAlex ID: W4404403105