Work

Gaze behavior during closed-captioned movie viewing adapts to absent audio through more frequent switching between text and scene

Year: 2026

Type: article

Abstract: Closed captions, originally for hearing-impaired audiences, are now increasingly used, adding an additional visual element to the multisensory experience of movie viewing. This study examines how gaze behavior adapts to the absence of auditory information by testing whether increased caption engagement reflects more frequent switching between closed captions and the visual scene, longer uninterrupted reading intervals, or both. Using eye-tracking, we analyzed saccades and fixations in designated areas of interest as participants viewed a 20-minute movie with alternating presence and absence of audio. Viewers integrated the two visual input streams by frequently shifting their gaze between the captions and the scene. The balance between reading captions and watching visuals shifts in the absence of sound by adjusting transition probabilities (i.e., increasing gaze switching between captions and scene, rather than changing the duration of reading intervals). This suggests that increased caption engagement in the absence of sound reflects more frequent sampling of captions, rather than longer uninterrupted reading. Modeling the probabilistic characteristics of fixations and saccades revealed more consistent and focused eye movement patterns on the caption text compared to the visual scene, highlighting distinct reading and gazing behaviours. Notably, these differences remained stable regardless of the presence or absence of sound. In addition to the availability of audio, we observed the presence of dialogue and the emotional content also influence caption engagement. These findings highlight how viewers adapt to missing audio through more frequent switching between text and scene while maintaining low-level properties of eye movements.

Source: Journal of Vision

DOI: 10.1167/jov.26.6.12