A brief peer-assessed OSCE for evaluating undergraduate abdominal ultrasound training: a three-cohort psychometric study
Year: 2026
Type: article
Study design: Observational Study
Abstract: BACKGROUND: Much of undergraduate ultrasound teaching takes place in student-led, near-peer courses, but learners' skills are rarely re-assessed once such a course is established. Full objective structured clinical examinations (OSCEs) often exceed the resources of these programmes, and brief peer-assessed formats need evidence before their scores can be interpreted. We examined which interpretations are justified, and for which purpose. METHODS: Psychometric observational study of three consecutive cohorts (2025 to 2026) of a three-day, student-led basic abdominal ultrasound course for fifth-semester medical students at one German faculty. Participation was voluntary. Each student completed two 10-minute stations on two of seven randomly allocated organs, scored on 13 criteria (maximum 14 points) by trained student tutors; a subsample was rated independently by two examiners. The primary question was whether aggregated criterion- and organ-level results are stable enough to inform course review. Supporting objectives concerned the interpretation of individual scores, the relation to a 15-item knowledge test and delivery. RESULTS: One hundred fifty-three students were scored on at least one station and 136 on both (total score 20.7 ± 4.1 of 28). The single-rater intraclass correlation coefficient (ICC) was 0.78 (95% CI 0.63-0.88; n = 39) and 0.73 (0.52-0.86; n = 33), and 0.65 and 0.69 after adjustment for organ; within-station Cronbach's alpha was 0.70 and 0.69, and 0.34 for the four interpretive criteria. Dynamic scanning criteria were hardest and among the most discriminating, basic criteria showed ceiling effects, and the difficulty ranking was concordant across cohorts (Kendall's W = 0.92). The allocated organ accounted for 22% of score variance, examiners for 7% in a crossed model. Between-station consistency was low (r = 0.17; organ-adjusted ICC 0.35); a reliability of 0.70 would require 5 organ-standardised stations (range 2 to 10). The total score correlated with the post-course knowledge test (r = 0.38; n = 79). CONCLUSIONS: Aggregated criterion- and organ-level results were reproducible across cohorts and can inform iterative review of a student-led course, within limits of precision, representativeness and transferability. Individual total scores from two stations are not reliable enough for consequential decisions about learners, regardless of agreement between examiners. Readiness for independent clinical scanning is not addressed.
Source: BMC Medical Education
Authors: Severin Pfeil, Mario Iasevoli, Florian Recker
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
Keywords: abdominal ultrasound, peer assessment, OSCE, near-peer teaching, medical students, ultrasound skills assessment, inter-rater reliability, psychometrics, course evaluation, abdominal organs, knowledge test, ceiling effect, medical education assessment, liver ultrasound, renal ultrasound, bladder ultrasound, bowel ultrasound
Open Access status: gold
Estimated APC paid: $2,990
OpenAlex ID: W7233737319
