Year: 2026
Type: conference-paper
Abstract: Human-annotated data remains fundamental to training frontier Large Language Models (LLMs).However, crowd-sourced annotations often suffer from quality issues stemming from annotator misunderstanding or lack of engagement.To address this, we introduce a real-time requirement adherence (RE-AD) framework that leverages LLMs to proactively validate labeling quality.Our methodology involves decomposing Standard Operating Procedures (SOPs) into atomic rules via self-reflection, categorizing them by complexity, and applying tiered validation strategies.Evaluated on a synthetic benchmark, the system achieved an F1 score of 0.749.Furthermore, production deployment resulted in annotators accepting and fixing 82% of the errors flagged by the framework.We include ablation studies to demonstrate the impact of our core design decisions.
Language: English
Subfield: Computer Science Applications
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166815437