Validating Synthetic Personas for Predicting Digital and AI Service Adoption: A Benchmark Against the Korea Media Panel Survey
Year: 2026
Type: preprint
Abstract: This study evaluates the predictive validity of an LLM-generated synthetic persona panel for estimating population-level patterns of digital and AI service adoption in South Korea. Synthetic personas are constructed from the NVIDIA Nemotron-Personas-Korea dataset and prompted to answer a fixed set of survey items using identical question and response-option wording. Their aggregated responses are then compared against ground-truth distributions from the Korea Media Panel Survey (KISDI), a nationally representative secondary dataset serving as the benchmark. The synthetic sample is drawn under a sex-by-age-group stratified design, with per-cell sample sizes set to clip(observed × 2, floor 200, ceiling 600) to guarantee a precision floor for small cells while capping oversampling in large cells (total ≈ 8,000). Responses are generated primarily with gemini-3.5-flash (thinking = low, top_p = 1.0; temperature 1.0 for the main run and 0.7 for a robustness run) via the Gemini Batch API, with EXAONE (an open-weight model served over an OpenAI-compatible endpoint) as a comparison model. Format-violating outputs are regenerated up to three times. Conditional items follow skip logic in which sub-items are omitted when the corresponding parent item indicates non-use. Because the benchmark relies on previously collected secondary data, the analysts specify that hypotheses and analysis plans were finalized without prior access to the outcome distributions being predicted. The study assesses how closely synthetic-persona response distributions recover the observed weighted benchmark estimates.
Author: Howard Kim
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7204277651