Comparative Evaluation of Confidence Calibration and Selective Abstention Across Heterogeneous Security Decision Tasks
Year: 2026
Type: preprint
Abstract: Security decision systems increasingly combine machine-learning scores with domain-specific integrity evidence, yet raw confidence values are often used without checking whether they remain reliable under operating variation. This paper presents a controlled comparative study of confidence calibration and selective abstention across three representative security settings: biometric identity verification, communicationsecurity monitoring, and digital-content integrity checking. The study does not introduce a new classifier or security architecture. Instead, it evaluates standard logistic regression, random forest, extremely randomized trees, and multilayer perceptron models on reproducible synthetic benchmarks designed to express the types of trust signals commonly reported in these domains. An isotonic calibration stage is additionally applied to the Extra Trees model, and a simple confidence-based abstention policy is evaluated under both development and shifted operating conditions. Across three random seeds and the three domains, logistic regression obtained the highest mean development macro-F1 of 0.915 and AUROC of 0.971, while calibrated Extra Trees achieved macroF1 of 0.913 and AUROC of 0.969. Calibration reduced the Extra Trees expected calibration error from 0.114 to 0.015 without materially changing discrimination. Under controlled distribution shift, calibrated Extra Trees retained macro-F1 of 0.907 and AUROC of 0.967. With a confidence threshold of 0.90, selective prediction retained about 71.2% of development decisions while increasing accuracy on retained cases to 97.4%. The results support treating calibration and abstention as separate operational properties from classification accuracy when security decisions are made from heterogeneous trust signals.
Source: Research Square
Author: Raji fredo
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7237232157
