The effects of auditory, visual, and gestural information on the perception on Mandarin tones
Year: 2017
Type: article
Abstract: In multimodal speech perception, strategic connections between auditory and visual- spatial events can aid in the disambiguation of speech sounds. This study examines how co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones with either congruent (C) or incongruent (I) Audio+Face (AF) and Audio+Face+Gesture (AFG) input. Mandarin perceivers performed at ceiling rates in the Congruent conditions, but showed a partially gesture-based response in AFG-I, revealing that gestures were perceived as valid cues for tone. The English group’s performance was better in congruent than incongruent AF and AFG conditions. Their identification rates were also highly skewed towards the visual tone when gesture was presented in the AFG compared to AF conditions. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that crossmodal resources can be recruited to aid auditory perception when phonetic demands are high.
Source: Summit (Simon Fraser University)
Author: Beverly Hannah
Language: English
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
Award: G1000907492
OpenAlex ID: W2775132676
