CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity
Year: 2026
Type: conference-paper
Abstract: This paper presents our system for SemEval-2026 Task 4 on narrative similarity assessment.We participated in both Track A (pairwise narrative similarity ranking) and Track B (embedding generation for narrative similarity).Through comprehensive experimentation, we evaluated various approaches including zeroshot pre-trained models, prompt engineering with large language models, and multiple finetuning strategies using synthetic data.Our experiments revealed a surprising finding: pretrained sentence transformers in a zero-shot setting consistently outperformed all fine-tuning attempts.Specifically, our best system using Sentence-T5-XL (Ni et al., 2021) achieved 67.5% accuracy on the development set (95% CI: [61.0%, 74.0%]), while all fine-tuning approaches resulted in performance degradation of 2-18 percentage points.We provide a detailed analysis of why fine-tuning failed and discuss the implications for narrative similarity tasks.
Institution: SoftBank Corp. (Japan)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166840709