Year: 2023
Type: conference-paper
Abstract: Embodied language comprehension emphasises that language understanding is not only a matter of mental processing in the brain, but also involves interactions with the physical and social environment.With the explosive growth of Large Language Models (LLMs) and their already ubiquitous presence in our daily lives, it is becoming increasingly necessary to verify their real-world understanding.Inspired by cognitive theories, we propose POSQA: a Physical Object Size Question-Answering dataset with simple size comparison questions to examine the extremity and analyse the potential mechanisms of the embodied comprehension of the latest LLM.We show that even the largest LLMs today perform poorly under the zero-shot setting.We then push their limits with advanced prompting techniques and external knowledge augmentation.Furthermore, we investigate whether their real-world comprehension primarily derives from contextual information or internal weights and analyse the impact of prompt formats and report bias of different objects.Our results show that real-world understanding that LLMs shaped from textual data can be vulnerable to deception and confusion by the surface form of prompts, which makes it less aligned with human behaviours.
Authors: Chang Shu, Jiuzhou Han, Fangyu Liu, Ehsan Shareghi, Nigel Collier
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4389519908