Search efficiency scales with semantic relatedness in audiovisual contexts
Year: 2023
Type: article
Abstract: Semantic information strongly influences attention in visual scenes. However, its role in audiovisual contexts remains unclear. Recent studies demonstrated that a task-irrelevant sound improves search for a matched visual target (e.g., hearing a bark allows participants to find an image of a dog more quickly). However, the extent to which a non-match but semantically related sound modulates attentional selection remains an open question. To elucidate the role of semantic processing beyond exact matches in audiovisual search, 109 participants searched for visual targets in an image array while a task-irrelevant sound played simultaneously. The task irrelevant sound could either exactly match the visual target or be one of 9 other sounds that varied in their relatedness to the visual target, as defined by a database of crossmodal semantic judgements (the Sight Sound Semantics database). As semantic relatedness between sounds and target images increases, search times decrease (r=-0.27; p=0.009). In prior studies, only sounds that exactly matched images increased search performance, which may reflect a prioritization of stimuli that co-occur in time and space (e.g., often when you hear a meow sound, you also see a cat so you prioritize all cat representations). Our results show that there is a graded effect of semantic relatedness, which means that sounds can modulate prioritization to images even outside of exact co-occurrence in time and space, suggesting a more robust influence of semantic information across modalities than previously thought.
Source: Journal of Vision
Institutions: George Washington University, University of East Anglia
Language: English
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $1,850
OpenAlex ID: W4386244366