Work

Where and When? Large Vision-Language Models for Entity Prediction in Multimodal News Events

Year: 2024
Type: dissertation
Abstract: The contemporary era has introduced a multimedia format in news communication, encompassing text, images, and video content. However, understanding multimodal events requires combining vision, language, and event knowledge, motivating to perform event-related tasks in the news domain, which have diverse applications. The recent emergence and success of Large Vision-Language Models offered new possibilities to address complex challenges in different areas of research, as they have shown to be effective in encoding and using world knowledge to perform several tasks. Although, the question remains whether these models are capable of reasoning over real-world event contexts and can be further used in event-related tasks. The core motivation of this thesis is to explore the state-of-the-art models’ behavior on multimodal events data, focusing on temporal and spatial information, contributing to the development of ubiquitous Artificial Intelligence assistants. We proposed a novel proxy task, involving both general and fine-grained evaluations, and diverse settings to align with the study’s objectives. Together with, a new dataset specifically created for this task, containing multimodal news data on both past events and more recent or future ones. Additionally, we conducted an agreement study to assess whether powerful models can evaluate the task, as humans would. The key findings of this study revealed that multimodal models leverage visual cues and outperform their Large Language Model backbones. Furthermore, the use of In- Context Learning prompting strategies enhances performance compared to zero-shot prompts, and fine-tuning is the most appropriate approach to achieve even better results. Generally, models behave better on spatial information than temporal and generalize well for event and location entities when reasoning about unseen data, but they show clear limitations with recent dates. Therefore, this thesis provides insights into the potential of Large Vision-Language Models for multimodal event-related tasks.
Author: Calvo, Bernardo Alexandre Vaz
Language: English
Open Access status: green
OpenAlex ID: W6988001566
Cites: 52
Cited by:
Related to: 10