Year: 2025
Type: preprint
Abstract: Modeling the joint distribution of data samples and their properties allows to construct a single model for both data generation and property prediction, with synergistic benefits reaching beyond purely generative or predictive models. However, training joint models presents daunting architectural and optimization challenges. Here, we propose Hyformer, a transformer-based joint model that successfully blends the generative and predictive functionalities, using an alternating attention mechanism and a joint pre-training scheme. We show that Hyformer is simultaneously optimized for molecule generation and property prediction, while exhibiting synergistic benefits in conditional sampling, out-of-distribution property prediction and representation learning. Finally, we demonstrate the benefits of joint learning in a drug design use case of discovering novel antimicrobial~peptides.
Source: arXiv (Cornell University)
Authors: Izdebski, Adam, Olszewski, Jan, Gawade, Pankhil, Koras, Krzysztof, Korkmaz, Serra, Rauscher, Valentin, Tomczak, Jakub M., Szczurek, Ewa
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6929560632
Cites:
Cited by:
Related to: