Year: 2026
Type: preprint
Abstract: Scaling laws are widely used to predict foundation model performance and guide compute allocation. However, tabular foundation models (TFMs) differ from conventional foundation models in that they are pretrained on virtually unlimited synthetic data from configurable distributions, while ultimately being evaluated on real-world datasets. We investigate whether scaling laws for TabPFNv2-style models reliably extrapolate to larger compute budgets and generalize across evaluation settings. Our analysis reveals differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. In particular, selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates. We further demonstrate that compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. Increasing the maximum numbers of rows and features sampled during pretraining improves performance and produces more predictable scaling trends. These insights guide the training of more efficient TabPFNv2-like models, using the open-source training pipeline of nanoTabPFN.
Source: arXiv (Cornell University)
Institutions: University of Freiburg, Delft University of Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: European High Performance Computing Joint Undertaking, European Commission, Deutscher Akademischer Austauschdienst, Deutsche Forschungsgemeinschaft, Bundesministerium für Bildung und Forschung, Technische Universität Dresden, China Scholarship Council, Hector Stiftung, HORIZON EUROPE Framework Programme, European Regional Development Fund
OpenAlex ID: W7236835174
