prakulhiremath/CDUR: CDUR v1.0: Initial Release (Experiments, Dataset, and Analysis)
Year: 2026
Type: software
Abstract: CDUR v1.0 — Reproducible Code and Data for Calibration Drift Under Reasoning This release provides the full experimental pipeline, dataset, and analysis used in the paper: "Calibration Drift Under Reasoning: Overconfidence Dynamics in Chain-of-Thought Language Models." Contents Evaluation dataset of 47 reasoning-trap questions spanning multiple failure modes Prompt templates for four reasoning budgets (none, light, medium, heavy) Raw model outputs and parsed results Scripts for computing accuracy, Expected Calibration Error (ECE), and overconfidence gap Implementation of CABStop (calibration-aware stopping rule) Reproduced figures and summary tables Key Idea We study how calibration changes as a function of reasoning budget in large language models. We observe that increasing reasoning depth can improve accuracy initially, but may also increase confidence without proportional gains in correctness. We refer to this behavior as Calibration Drift Under Reasoning (CDUR). Reproducibility All reported results can be reproduced using the provided scripts: Run experiments: scripts/run_experiments.py Compute metrics: scripts/compute_metrics.py Results may vary slightly depending on API behavior and sampling. Notes Metrics are computed on valid (parseable) responses only Approximately 42% of outputs were retained after filtering 70B model results are limited to baseline evaluation
Authors: PRAKUL HIREMATH, Harshit-212
Institution: Visvesvaraya Technological University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7155385474