Work

The effects of auditory, visual, and gestural information on the perception on Mandarin tones

Year: 2017
Type: article
Abstract: In multimodal speech perception, strategic connections between auditory and visual- spatial events can aid in the disambiguation of speech sounds. This study examines how co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones with either congruent (C) or incongruent (I) Audio+Face (AF) and Audio+Face+Gesture (AFG) input. Mandarin perceivers performed at ceiling rates in the Congruent conditions, but showed a partially gesture-based response in AFG-I, revealing that gestures were perceived as valid cues for tone. The English group’s performance was better in congruent than incongruent AF and AFG conditions. Their identification rates were also highly skewed towards the visual tone when gesture was presented in the AFG compared to AF conditions. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that crossmodal resources can be recruited to aid auditory perception when phonetic demands are high.
Language: English
Field: Psychology
Open Access status: green
OpenAlex ID: W2775132676
FWCI:
Cites: 73
Cited by:
Related to: 10