Team CV at SemEval-2026 Task 4: Prompting LLMs and Benchmarking Embedding Models for Narrative Story Similarity
Year: 2026
Type: conference-paper
Abstract: This paper describes Team CV's systems for SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning (Hatzel et al., 2026).For Track A (comparative judgment), we explore five prompting strategies-zero-shot, chain-of-thought, structured feature extraction, pairwise scoring, and few-shot-and QLoRA fine-tuning of smaller models.For Track B (narrative embeddings), we benchmark twelve dedicated text embedding models of varying dimensionality (384-4096) spanning open-source (E5-Large-v2, BGE, GTE, Qwen3 Embedding) and closed-source (OpenAI, Gemini, Mistral) families, and fine-tune Qwen3 Embedding 4B on task-specific triples.Fewshot prompting with Qwen-2.5 7B (64.00%) outperforms all fine-tuned variants (best 57.50%) on Track A; scaling to LLaMA-3.3-70B yields 75.00%.On Track B, Ope-nAI text-embedding-3-large (3072d) achieves the best dev accuracy (67.00%), while fine-tuning Qwen3 Embedding 4B (2560-d) on synthetic triples slightly decreases accuracy.Our final submission-LLaMA-3.3-70B(3-shot) for Track A and text-embedding-3-large for Track Bachieves 70.75% and 64.50%, exceeding the GPT-4o-mini and STORY-EMB baselines respectively.
Authors: Chandan Kumar R S, Vinay Ulli
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166876506