AI-MRS: A Governed Retrieval-Augmented Metric Recommendation System for Generative AI Systems
Year: 2026
Type: conference-paper
Abstract: Generative Artificial Intelligence (GenAI) systems are rapidly being adopted and deployed across highly regulated and sensitive domains. Evaluation failures in these domains can lead to significant consequences. The existing evaluation practices largely depend on generic, performance-centric metrics that fail to capture real-world concerns such as hallucinations, bias, and security vulnerabilities. Meanwhile, existing Responsible AI frameworks define high-level evaluation principles but provide limited direction on concrete, task-level metric selection. This gap results in fragmented, ad hoc evaluation across teams. We propose AI-MRS, a governed, retrieval-augmented metric recommendation system that standardizes context-aware evaluation metric selection for GenAI systems. Given a structured system description, AI-MRS recommends evaluation metrics that are relevant, governance-aligned, and balanced across performance and Responsible AI dimensions. Experiments across five heterogeneous deployment scenarios demonstrate consistent, grounded, and zero-hallucination metric recommendations, with a composite score averaging 0.885. AI-MRS offers a scalable, auditable solution for systematic GenAI evaluation.
Authors: Ajay D M, Ritu Maheshwari
Institutions: HCL Technologies (United States), HCL Technologies (India)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7202276594