Work

Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback

Year: 2025

Type: article

Abstract: Evaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resulting in significant computational overhead and limited practical applicability. In this paper, we propose a novel prompt optimization method that leverages LLM-based confusion matrix feedback to efficiently optimize prompts for the relevance evaluation task. Unlike previous approaches, our method systematically analyzes LLM predictions—both correct and incorrect—using a confusion matrix, enabling prompt refinement through a single-step update. Our experiments in realistic IR scenarios demonstrate that our method achieves competitive or superior performance compared to existing methods while drastically reducing computational costs, highlighting its potential as a practical and scalable solution.

Source: Applied Sciences

DOI: 10.3390/app15095198

Author: Jaekeol Choi

Institution: Hankuk University of Foreign Studies

Language: English

Topic: Information Retrieval and Search Behavior

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,896

Funder: Hankuk University of Foreign Studies