Work

Which Model, Which Prompt? Selecting and Configuring LLMs as Oracles for Ontology Matching

Year: 2026

Type: conference-paper

Abstract: Large language models (LLMs) are viable oracles for use during interactive ontology matching, but their adoption raises three practical questions: (i) Which model do we select? (ii) How should the prompt be structured? (iii) What should the prompt contain? This work responds to the first question by benchmarking a large selection of language models via LogMap’s interactive matching mode on OAEI tasks, recommending models that balance performance, cost, latency, reliability, openness, and provider diversity; we publish the benchmark as a community resource via an online results explorer. For the second and third, we build on prior work, extending existing prompt templates for class, property, and instance matching, investigate zero-shot versus RAG-based few-shot ontology-driven prompting, and introduce collective anchors for few-shot-RAG ontology-driven prompting. We apply these methods to an existing system design, LogMapLLM, and evaluate over five OAEI tracks, spanning a variety of domains. We find that no single configuration consistently wins out, but the oracle’s failure mode helps explain which interventions work: demonstrations recover sensitivity in conservative oracles, whereas mutual subsumption improves specificity in permissive ones. Ultimately, the optimal configuration is model-, prompt-, and domain-specific, motivating ongoing work on automatic selection, where the most preferable model × prompt configuration is recommended given a matching task.

Source: City Research Online (City University London)