Year: 2025
Type: dissertation
Abstract: Language models for RNA and proteins have revolutionized biological research, yet the principles governing their scaling remain an underexplored frontier. While numerous studies continue to investigate new architectures and training strategies, a systematic understanding of how scaling laws influence their performance is still lacking. In this work, inspired by the scaling methodologies proposed by Hoffmann et al. [1] for Large Language Models we systematically explore different configurations of protein and RNA language models, with a particular emphasis on tokenization strategies and the interplay between model parameters and training tokens. Our findings reveal that, while both parameter count and dataset size contribute to model performance, the number of training tokens exerts a more profound influence. In the case of pLMs, we successfully broke the performance plateau observed in ESM by developing MMFreePLM, an architecture that achieves superior efficiency while reducing computational overhead. This demonstrates that architectural optimization can unlock substantial gains in scalability without necessitating an exponential increase in resources. Similarly, for rnaLMs, we show that ChaRNABERT, with only 50 million parameters, achieves performance on par with RiNALMo, a significantly larger model with 650 million parameters. These findings highlight a key insight that optimizing architectural design and tokenization strategies has a greater impact on biomolecular language model performance than merely increasing model size. By bridging critical gaps in the understanding of scaling laws for pLMs and rnaLMs, this study paves the way for the development of more computationally efficient and scalable biomolecular models. Our work not only enhances the feasibility of large-scale biological sequence modeling but also contributes to advancing biotechnological research aimed at decoding the language of life with greater precision and efficiency.
Author: Vázquez Reza, Raquel
Language: English
Subfield: Molecular Biology
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W7111744605