Vision and language: Recoding of visual representations
Year: 2010
Type: article
Abstract: Although most researchers agree that language and vision are domain specific each with different representational primitives, it is also true that these systems interact allowing us, e.g., to talk about what we see. What is the mechanism, development and consequence of an interaction between language and vision? We describe one language-vision interaction and lay out a hypothesis and some evidence to explain the underlying mechanism. The visual system sometime fails to encode or bind relevant features of an object. In several experiments we asked whether language could help children and adults bind and maintain visual properties that are difficult to retain in visual memory. In a delayed-matching task, participants were shown squares that were split in half by color (e.g., a square with red-left, green-right) and after a delay, they were shown the target replica, the target reflection (e.g., red-right, green-left), and another distracter. We found that providing specific directional terms to children accompanying the target enhanced their ability to find target matches, avoiding foils. Non-linguistic attentional manipulations did not show these effects, nor did linguistic instructions that did not include the directional terms. Finally, when adults are given the same task as the children in one of three conditions (No-Shadow, Verbal-Shadow, and Rhythm-Shadow), they performed significantly worse in the Verbal-Shadow condition compared to the other shadow conditions. Language appears to play a crucial role in maintaining feature conjunction in memory for both adults and children. We propose the Asymmetric Recoding Hypothesis which states that given an object without an inherent figure and ground, language forces the creation of an asymmetry between the parts, and forces the encoding of a directional relationship between the parts. We argue that given visual and linguistic information, a hybrid representation is formed that combines crucial details from each source.
Source: Journal of Vision
DOI: 10.1167/8.6.419
Authors: B. Dessalegn, B. Landau
Institution: Johns Hopkins University
Language: English
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $1,850
OpenAlex ID: W2045292145