Locate and Explain: Joint Multimodal Emotion Cause Extraction and Summarization in Conversation
Year: 2026
Type: conference-paper
Abstract: Multimodal emotion cause analysis in conversation aims to identify the causes of emotions by leveraging multimodal information.Existing studies mainly formulate this problem as either utterance-level emotion cause extraction, which provides clear cause localization but limited explanation, or multimodal emotion cause generation, which offers fine-grained explanations but lacks explicit traceability to source utterances.Moreover, existing datasets rely heavily on human judgment and lack well-defined structured theoretical guidance, leading to subjective and inconsistent annotations.To address these issues, we introduce joint Multimodal Emotion Cause Extraction and Summarization in conversation (MECES), a new task that simultaneously extracts emotion cause utterances and generates cause summaries, enabling both precise localization and interpretable explanations of emotion cause.We further construct a MECES dataset guided by the Activating events-Beliefs-Consequences theory from psychology.This dataset consists of 5,787 emotion utterances annotated with causes, comprising 12,231 emotion-cause pairs and 6,040 cause summaries.We also propose an effective endto-end joint learning approach for MECES task, establishing strong benchmark results for this newly introduced task and dataset.( U1,U2 , "Chuan Bai handed Guang Shi a gift, and Guang Shi didn't expect to receive one too.") Guang Shi："I got a gift too!"
Authors: Jikun Wan, Chen Gong, Guohong Fu
Institution: Soochow University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G1502338116, G1998348612
OpenAlex ID: W7166843240