Evaluating Logical Error Sensitivity in Text-Only Proxies for Generated Science Rubrics
Year: 2026
Type: preprint
Abstract: Automatically generated marking rubrics could reduce assessment-authoring effort, but educational usefulness depends on whether evaluation detects criteria that would misgrade student responses, not only fluent paraphrases. This study examines that measurement problem in a public, text-only proxy testbed built from SciQ support explanations; the references are constructed proxies, not educator-authored rubrics. A locally served Qwen3-14B instruction model generated two-point rubrics for 120 held-out science questions under three decoding seeds and was compared with answer-only, retrieval-edit, and reference-template controls. Its outputs contained the supplied answer on 0.992 of items, while support-concept coverage was 0.263 to 0.268 and point-F1 was 0.434 to 0.436. An answer-only baseline reached point-F1 0.547 under the same constructed metric. Deliberately negated rubrics still had lexical cosine 0.901 and ROUGE-L 0.849; the concept metric remained 0.724 because polarity was not represented. Across 360 model generations, reference-similarity scores correlated with point-F1. These corruption tests are not calibrated acceptance decisions, but they show that aggregate association is insufficient evidence of logical sensitivity. The study provides code, data hashes, raw generations, repeated-seed estimates, and controlled corruptions for developing evaluation protocols that test negation, omission, irrelevant support, and unsupported generic language.
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7224485327