Stratified few-label protocols can overstate macro-averaged metrics for tabular foundation models
Year: 2026
Type: preprint
Abstract: Tabular foundation models classify a new table's rows from a few labelled rows. Their benchmarks often label equally many rows of each class; a practitioner without labels can only label random rows. We show that this stratified labelling overstates macro-F1, the F1 score averaged over all classes. A stratified draw is a random draw conditioned on equal class counts, so the gain splits into a coverage part, from random draws that miss a class, and a balance part, from unequal class counts. On tables kept unopened until a final analysis, stratified labelling raised macro-F1 by +0.030 to +0.075 for four models at one to ten labels per class (+0.013 to +0.058 at three to ten). The gain is large where a missed class is costly, and too small to confirm where it is cheap. On 39 tables published after every model tested, class counts alone ranked the gains (Spearman 0.72) but overstated their size; a model of this cost, fitted on benchmark tables and fed with each table's runs, tracked them (Spearman 0.86). Stratified labelling also lowers accuracy on imbalanced tables and can change which model wins. We propose reporting few-label results under random labelling, with the share of draws that miss a class. Preprint, not peer reviewed. Supplementary material in the second file.
Author: Coutinho, Felipe Provezano
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225654365
Cites:
Cited by:
Related to: