Conflict or Ignorance in the Rashomon Set? Certified Typed Verdicts for Variable Importance
Year: 2026
Type: preprint
Abstract: When many models are almost equally accurate, they can disagree about which variables matter. Reports of this disagreement, as a range or a distribution of importance values, rarely say whether it would disappear with more data or would remain with unlimited data, and the two situations call for different actions. We compare the range of a variable’s importance over the population ε-Rashomon set, read as a partially identified quantity, with an importance threshold, and return for each variable one of four verdicts: IMPORTANT and IRRELEVANT (every ε-good model is on the same side of the threshold), STRUCTURAL (ε-good models lie on both sides, a property of the population, the model class and the tolerance that does not change with the sample size) and IGNORANCE (an abstention, with a projection of the sample size that would settle it). The probability of certifying at least one wrong verdict is controlled at level α asymptotically. For linear regression the verdicts have a closed form, and, for coefficients on fixed (population) scales under a correctly specified homoskedastic model, collinearity inflates a coefficient’s identified half-width and its standard deviation by the same factor. For finite model classes, such as all depth-two trees on a split grid, we build outer and inner confidence sets for the Rashomon set from loss differences to a reference fixed on a held-out split, a paired permutation estimator with independent units, and a studentized multiplier bootstrap. In simulations, including designs with an importance endpoint within 0.005 of the threshold or a model about 0.001 from the tolerance boundary, one wrong verdict was certified in more than 2,400 runs, and the empirical family-wise error stayed below α in every setting; bounding raw losses instead of loss differences lowered the share of replicates certifying STRUCTURAL at n = 20,000 from 0.73–0.83 to 0.33–0.67, and adapted general concentration bounds of Fisher et al. (2019), with an empirical loss bound, certified one linear verdict in six at n = 100,000. In a diabetes-screening illustration (BRFSS 2015), the best single tree assigns zero importance to heart disease and walking difficulty, while the procedure certifies both as STRUCTURAL at a one-point error tolerance in two of three sample splits, and reports IGNORANCE, never a contradictory verdict, in the third. Code is released as the Python package typedvi.
Source: Preprints.org
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7226036359
