Year: 2025
Type: article
Abstract: The growing interest in machine learning in a critical domain like healthcare emphasizes the need for reliable predictions, as decisions based on these outputs can have significant consequences. This study benchmarks methods for assessing pointwise reliability, focusing on data-driven techniques based on the density principle and the local fit principle. These methods evaluate the reliability of individual predictions by analyzing their similarity to training data and evaluating the performance of the model in local regions. Aiming to establish a standardized comparison, the study introduces a benchmark framework that combines error rate evaluations across reliability intervals with t-distributed Stochastic Neighbor Embedding visualizations to further validate the results. The results demonstrate that methods combining density and local fit principles generally outperform those relying on a single principle, achieving lower error rates for high-reliability predictions. Furthermore, the study identifies challenges such as the adjustment of method parameters and clustering limitations and provides insight into their impact on reliability assessments.
Source: Information
DOI: 10.3390/info16040327
Authors: Cláudio Correia, Simão Paredes, Teresa Rocha, Jorge Henriques, Jorge Bernardino
Institutions: Polytechnic Institute of Coimbra, University of Coimbra
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,930
OpenAlex ID: W4409607479