D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Models
Year: 2025
Type: conference-paper
Abstract: Evaluating generative models with open-ended generation is challenging due to inconsistencies in response formats.Multiple-choice (MC) evaluation mitigates this issue, but generating high-quality distractors is time-consuming and labor-intensive.We introduce D-GEN, the first open-source distractor generator model that transforms open-ended data into an MC format.To evaluate distractor quality, we propose two novel methods: 1) ranking alignment, ensuring generated distractors retain the discriminatory power of ground-truth distractors, and 2) entropy analysis, comparing model confidence distributions.Our results show that D-GEN preserves ranking consistency (Spearman's ρ 0.99, Kendall's τ 0.94) and closely matches the entropy distribution of ground-truth distractors.Human evaluation further confirms the fluency, coherence, distractiveness, and incorrectness.Our work advances robust and efficient distractor generation with automated evaluation, setting a new standard for MC evaluation.
Authors: Grace Byun, Jinho D. Choi
Institution: Emory University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888694