Frontiers: Reducing Disparity Between LLMs and Humans: Optimal LLM Sample Calibration
Year: 2026
Type: article
Abstract: Large language models (LLMs) are increasingly used to measure preferences and opinions, but outputs vary across prompts, personas, and model checkpoints, leaving analysts with multiple noisy elicitation variants and no principled way to combine them. Existing practice typically relies on heuristic aggregation---uniform averaging, population-weighted post-stratification, or selecting the single best subgroup---which uses fixed ex ante weights and can remain misaligned with human benchmarks. We propose Human Mimicry Calibration (HuMCal), a post-hoc calibration method that learns credibility weights over elicitation variants using only aggregate benchmark data, with no access to model internals or subgroup-level labels. HuMCal is model-agnostic and baseline-safe on the calibration split: its optimized loss weakly dominates any fixed-weight rule over the same subgroup set, including uniform weighting, post-stratification, and the best single subgroup. We evaluate HuMCal in a revealed-preference market-structure application using U.S. bank-card transactions and in a public-opinion application using Pew survey data. Across both settings, HuMCal improves alignment on the calibration split relative to all fixed-weight baselines. Out of sample, source-learned weights outperform heuristic baselines across time, geography, industry, and topic in the large majority of evaluations, although transfer is empirical rather than guaranteed. Recalibration with newer LLM checkpoints often improves alignment further. These results suggest that HuMCal can serve as a scalable calibration tool for LLM-based measurement when collecting fresh benchmark data for each target setting is costly.
Source: Marketing Science
Authors: Yan Leng, Yunxin Sang, Ashish Agarwal
Institution: The University of Texas at Austin
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W4395015967