Artificial Intelligence Predictions and Human Perception: Comparing User Trust with Accuracy
Year: 2026
Type: book-chapter
Abstract: This study aims to examine the relationship between the technical predictive accuracy of artificial intelligence (AI) models and the confidence that expert users place in these predictions. Despite the current emphasis on evaluating AI technologies based only on technical criteria, the importance of user confidence as part of such evaluation process and the relation of accuracy to confidence are often underestimated. Taking this into account, the current paper aims at understanding the relationship between user perceptions regarding AI outputs and the actual accuracy of the models generating predictions. To accomplish the stated goal, data from 50 academics was analyzed. Participants of the research were shown 20 different prediction samples provided by the AI model and asked to evaluate their confidence on a 5-point Likert scale. The statistical methods involved in the analysis comprised descriptive statistics, correlation analysis, and an independent-samples t-test. The analysis resulted in the identification of a positive and statistically significant yet moderate relationship between prediction accuracy and user confidence. Particularly, low-accuracy predictions could trigger high user confidence, and vice versa – some high-accuracy predictions could be rated with low confidence scores. The findings confirm that user confidence might be miscalibrated and impacted by several psychological processes such as algorithm aversion and bias. At the same time, the outcomes support the position that confidence is connected with prediction accuracy, rather than other criteria, but is impacted by explainability and presentation style of the system under analysis. As far as XAI plays a crucial role in enhancing both of the factors mentioned above, Explainable Artificial Intelligence should receive particular attention in such studies. In general, the presented paper shows that the evaluation of AI technology must not focus only on technical performance of the system, but must also consider user-related criteria. Further research might analyze directly the effect of using XAI techniques on the level of trust in experiments among various professions.
Source: Özgür Yayınları eBooks
Institution: Istanbul Arel University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7208776250