Perceived and demonstrated generative AI literacy in engineering problem solving: the role of verification behavior
Year: 2026
Type: article
Study design: Observational Study
Abstract: Introduction Generative artificial intelligence is increasingly used in engineering education, but self-reported literacy may not establish whether students can evaluate its outputs. The distinction between perceived competence and performance on engineering-specific AI scenarios remains insufficiently examined. This study investigated their correspondence and whether reported verification frequency statistically accounted for the association between demonstrated literacy and engineering task performance. Methods A cross-sectional online survey and assessment involved 1,206 mechanical engineering students at three Chinese universities. Measures comprised a 20-item perceived literacy questionnaire, a 12-item scenario test, an eight-item verification frequency scale, and two engineering tasks. Mediation and moderated mediation models controlled gender, year of study, AI use, AI course experience, and university fixed effects. Results Perceived and demonstrated literacy were modestly correlated, r = 0.265, 95% confidence interval [0.212, 0.317]. Demonstrated literacy was associated with verification frequency, b = 0.065, and the adjusted indirect association with engineering performance was 0.124, 95% bootstrap confidence interval [0.096, 0.154]. A small negative interaction suggested a weaker verification-performance association at higher perceived literacy, but its practical importance remains uncertain. A reverse-order model also produced a nonzero indirect association. Discussion The findings support assessing self-perception and scenario performance separately and investigating the quality of verification. Because the assessment was unproctored, the scores do not establish unaided competence; the cross-sectional associations do not establish a causal verification mechanism.
Source: Frontiers in Psychology
Authors: Xiaoqing Liu, Chenliang Zhang, Junbing Pan
Institution: Shaanxi University of Technology
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: gold
Estimated APC paid: $3,801
OpenAlex ID: W7225274830