Non-referential visual contexts effects on incremental semantic interpretation of abstract sentences : an eye-tracking reading study
Year: 2014
Type: dissertation
Abstract: This doctoral thesis focuses on two issues that, in the context of research into psycholinguistics, have been underexplored so far.First, we asked whether abstract language-which does not directly speak about objects and actions-could be affected rapidly and incrementally by visual information.Second, we ask whether visual contexts that are not a depiction of the object and events subsequently described in a sentence could affect processing time for that sentence.Additionally, this work aims to clarify the cognitive mechanisms that would allow rapid interaction between abstract language and what we denominated non-referential visual contexts.A necessary literature review (Chapter 2) revealed a trend of moving from overt relationships between what is seen and what is described in a sentence, to subtler relationships, such as word associations.We identified however, that most studies examining real-time sentence processing concentrated on concrete language, and that results with respect to abstract language are mixed or limited to word recognition.Studies that were not particularly concerned with online sentence processing, on the other hand, have examined the relationship between visual information (e.g., spatial distance, containment and spatial locations) and abstract concepts.These studies covered a number of different conceptual tasks, such as similarity judgments, fast categorization, or ratings among others.They explored several abstract domains, such as the concepts of similarity, time, social relations and affective valence.None of those studies showed evidence of the involvement of spatial representations during online language comprehension.We concentrated on two concepts, namely similarity and intimacy, for which we found compelling theoretical and empirical evidence suggesting a close link to spatial distance.We discussed existing behavioral evidence for this mapping and argue that it might have consequences for online abstract sentence processing.At the end of the chapter, we present vi two existing models of situated sentence processing, discussing their central cognitive mechanisms and evaluating whether they could accommodate potential non-referential visual context effects in online abstract sentence processing.We argue that these accounts in their current versions are not well prepared to accommodate how visual scenes that do not depict the objects or events described in a sentence could potentially affect the processing of such sentence.Using a novel eye-tracking experimental paradigm designed to investigate the four research questions outlined in Chapter 1, we conducted five experiments, in which we crossed two levels of spatial distance with two levels of abstract semantic content.Participants inspected a visual context with two playing cards that moved either close together or far apart.Immediately after, participants read a sentence that expressed semantic similarity between two abstract nouns (Chapter 3) or a social interaction between two characters (Chapter 4).Participants' eye-movements were recorded as they read critical sentences and subsequently reading measures were computed.The critical regions of the sentence were those in which similarity (either similar or dissimilar; Experiments 1, 2 and 3) or the type social relation (either friendly or unfriendly; Experiments 4 and 5) were implied or explicitly mentioned.The results of the first three experiments revealed significant interaction between spatial distance and semantic similarity; overall, sentences that expressed similarity between two abstract nouns were read faster when playing cards in the visual context moved close together, compared to when they moved far apart.In contrast, sentences that expressed dissimilarity between two abstract nouns were read faster when playing cards in the visual context moved far apart, compared to close together.Analysis of the eye-tracking data in Experiment 4 further extended these findings showing that spatial distance can also modulate sentences concerning social relations.Interestingly, these results showed the opposite pattern to those experiments on similarity; sentences that expressed friendly social interactions between two characters were read faster when playing cards in the visual context moved far vii apart, compared to when they moved close together, and vice versa for sentences expressing unfriendly social interactions.In Experiment 5, however, these effects were not replicated, suggesting that a fully non-referential visual context might modulate abstract language only in certain circumstances.In the last chapter, we discussed the literature review chapter, summarizing key findings, open issues and specific research questions.We then discussed the results of our five experiments concentrating on the extent with to which they could answer our research questions.Subsequently, we contrasted our results with assumptions and predictions from existing models of situated language comprehension, and discussed some of the challenges that these findings presents for them.We argued that these accounts are not capable in their current state to accommodate our findings.However, we suggest that an extension of existing mechanisms such as co-indexing and spreading activation could account for our data.Consequently, we described how these updated mechanisms could explain the effects of a non-referential visual context on online abstract language comprehension.At the end of the chapter, we presented a summary of the key conclusions that could be drawn from this research project, and outlined a number of new directions in which our findings could potentially guide research concerning visually situated abstract language.viii Kamide, 1999;Kamide, Scheepers & Altmann, 2003) and that words can direct attention to unmentioned objects (e.g., Huettig & Altmann, 2005, 2011;Mirman & Magnuson, 2009).Furthermore, some studies have showed that language can direct visual attention to locations on a blank screen where visual referents were (Altmann 2004;Knoeferle & Crocker, 2007) or could be (Altmann & Kamide, 2009).Such results highlight the role of mental representation and working memory as mediators of the interaction between language and the visual world.sentence processing has been limited to the study of visual contexts in which objects or actions are depicted, and to sentences that describe those objects and actions literally.In this sense, abstract language, such as abstract nouns (e.g., similarity, friendship) or abstract adjectives (e.g., similar, different, friendly, unfriendly), and their potential connections with visual information have not been assessed in the context of online sentence processing.
Author: Ernesto Guerra
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W144442250