Work

Conversational search for image and video with augmented labelling

Year: 2025
Type: dissertation
Abstract: The rapid growth of media archives—including text, speech, video, and audio—has driven strong interest in developing advanced search methods for multimedia content. In particular, conversational search has emerged as a promising approach, where users engage in a dialogue with an AI agent to support and enhance their search activities. While most existing systems focus on text-based archives, this research extends conversational search methods to image and video retrieval. Our approach involves developing an experimental framework to explore how conversational engagement can improve multimedia search. We introduce a prototype system that combines dialogue-based interaction with state-of-the-art visual indexing techniques. While multimedia information retrieval (MIR) has long been studied through conventional user-driven interfaces, the integration of conversational agents introduces a new layer of interactivity. The agent aims to assist users by suggesting relevant content and helping to filter out irrelevant results. Effective dialogue in this context requires the agent to demonstrate an understanding of the content and its relevance to the user’s needs. Although MIR techniques have advanced significantly, little attention has been paid to how retrieved content is represented and communicated during the search process. Our system addresses this gap by incorporating object detection to highlight key visual features, enhancing both the accuracy and contextual relevance of search results. To evaluate the framework, we conducted three user studies focused on the effectiveness of conversational engagement in multimedia search. These studies examined how AI-driven dialogue affects users’ ability to retrieve relevant image and video content and improves the overall search experience. Results indicate that conversational interaction not only refines retrieval accuracy but also increases user satisfaction by creating a more intuitive and responsive search environment.
Author: Potyagalova, Anastasia
Language: English
Open Access status: green
OpenAlex ID: W7112969976
Cites: 105
Cited by:
Related to: 10