Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems
Year: 2026
Type: article
Abstract: Modern AI systems rely heavily on labeled data, yet labeling is often expensive and labor-intensive, especially when requiring special skills such as reading radiology images by physicians. To most efficiently use experts’ time for data labeling, one promising approach is human-in-the-loop active learning. However, traditional active learning methods are limited to single-label queries and fail to leverage more flexible query types in many real-world settings. In this work, we propose a novel active learning framework with significant potential for application in modern AI systems. This framework incorporates different query schemes and jointly determines which question to ask and which data points to query. We also introduce a method to integrate full and partial information obtained from these diverse queries. In addition, we develop an innovative model-agnostic exploration and exploitation framework to filter out redundant samples. Experiments on five datasets, including two real-world image datasets, demonstrate that the proposed framework substantially outperforms all other methods. These results highlight the framework’s promise for applications across a range of scientific domains.
Authors: Yiran Huang, Jian-Feng Yang, Haoda Fu
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Award: G7844691718
OpenAlex ID: W4406031083