Work

Adaptive Mutual-Human Evaluation: A Universal Framework for Multi-LLM Reciprocal Assessment, Human Evaluation, and Context-Aware Switching

Year: 2026
Type: preprint
Abstract: The rapid development of large language models (LLMs) has created increasing demand for reliable, scalable, and context-sensitive evaluation mechanisms. Existing evaluation approaches include automated metrics, single-model evaluation, LLM-as-a-Judge, multi-agent evaluation, peer assessment, human evaluation, and human–AI evaluation. Although these approaches provide important capabilities, evaluation systems may remain constrained by fixed evaluator configurations, predetermined combinations of evaluators, or insufficiently adaptive allocation of evaluation responsibilities. A fundamental question therefore emerges: should an evaluation task always be assessed by multiple artificial intelligence models, always be assessed by humans, or should the evaluation mechanism dynamically change according to the requirements of the task? This paper introduces Adaptive Mutual-Human Evaluation (AMHE), a function-specific theoretical framework developed from the Universal Mutual-Human Model (UMHM). AMHE establishes two distinct but combinable evaluation dimensions: Mutual Evaluation and Human Evaluation. Mutual Evaluation refers to reciprocal or distributed assessment among two or more distinct evaluators, particularly heterogeneous or homogeneous LLMs that independently assess, compare, critique, challenge, and refine evaluation judgments. Human Evaluation refers to assessment performed by human individuals or groups through their knowledge, expertise, contextual understanding, critical judgment, and evaluative authority. The framework establishes three operational configurations: Mutual-only Evaluation, Human-only Evaluation, and Mutual–Human Evaluation. The central contribution is an adaptive evaluation architecture that does not permanently assign evaluation to either artificial intelligence or humans. Instead, it introduces Adaptive Evaluator Selection, Adaptive Evaluation Switching, Reciprocal Evaluation, Human Intervention, Adaptive Evaluation Combination, and Evaluator-Structure Reconfiguration. A system may begin with reciprocal evaluation among multiple LLMs, switch to human evaluation when uncertainty, disagreement, or risk increases, move from human evaluation to multi-model assessment when scalable comparative analysis becomes appropriate, or combine both mechanisms when complementary judgments are necessary. The proposed architecture further introduces uncertainty-aware routing, disagreement-sensitive intervention, evaluation-budget allocation, independent judgment preservation, and context-dependent evaluation governance. A formal model is developed to represent evaluator configurations, evaluation quality, uncertainty, disagreement, switching costs, human participation, and adaptive decision policies. The paper also establishes theoretical propositions, a unified architecture, an adaptive evaluation algorithm, application scenarios, an experimental research design, and a multidimensional evaluation protocol. Importantly, AMHE does not assume that model consensus guarantees correctness or that human judgments are universally superior. Instead, it treats evaluator selection as a context-dependent decision that must account for reliability, independence, expertise, cost, risk, and accountability. To the best of our knowledge, this paper is the first to formulate, specifically within the UMHM theoretical foundation, an explicitly adaptive, function-general evaluation architecture that formalizes reciprocal multi-model evaluation and independent human evaluation as alternative and combinable mechanisms, with context-aware switching between them as a central design principle. This contribution concerns the particular theoretical integration and adaptive architecture proposed here, rather than claiming priority over established research on LLM-as-a-Judge, multi-agent evaluation, human assessment, or hybrid evaluation. The framework provides a theoretical foundation for next-generation evaluation systems in which the identity, number, interaction, and participation of evaluators can change dynamically according to evaluation requirements.
Language: English
Open Access status: green
OpenAlex ID: W7236594392
Cites:
Cited by:
Related to: 10