When Are Synthetic Consumers Reliable? Baseline Validity and Universal Understanding of Variance in Silicon Sampling with Large Language Models for Customer Satisfaction Research
Year: 2024
Type: article
Abstract: Background. Silicon sampling, the use of large language models (LLMs) as synthetic respondents, has been proposed as a low-cost alternative to primary data collection, and specifically for consumer and marketing research tasks such as questionnaire pretest, concept screening, and pilot segmentation (Sarstedt et al., 2024; Argyle et al., 2023; Horton, 2023). A recurring concern is that LLMs distort the variance of human responses, but the literature has not consistently distinguished between distortion as a property of the synthetic distribution alone and distortion as a function of the baseline of comparison employed. Objective. We ask, as a primary question, whether the direction and magnitude of the variance distortion in silicon sampling depend on the validity of the comparison baseline against which it is judged; and, as a secondary question, whether the post-training alignment regime of the LLM conditions the magnitude of that distortion. We use six LLMs grouped into three alignment classes and the canonical customer satisfaction instrument European Customer Satisfaction Index (ECSI). We compared each LLM against two explicit baselines: a synthetic baseline reconstructed from published summary statistics (the usual practice when microdata is not available) and the actual semPLS::mobi baseline of Tenenhaus et al. (2005), using a data loader with labeled provenance that refuses to silently substitute synthetic data. Method. We elicited 250 sets of ECSI responses from each of six LLMs: two preference- based RL models (Claude Sonnet 4 with constitutional oversight, and GPT-5-mini with reasoning enabled; both with details not disclosed by the provider), Mistral-7B-Instruct v0.3 (DPO without RL), and three SFT-only models (Dolphin 2.8, Dolphin 3.0, Hermes-3). Synthetic personas were constructed from the scores of latent constructs from the real semPLS::mobi benchmark (n = 250, 22 indicators following schema reconciliation, 7 latent constructs with a single-item Complaints construct following Eklöf, 2000). Inferential tests included (a) one-sample t-tests on construct-level (σ_modelo/σ_línea-base) ratios of variance at the construct level (df = 6, on crude and logarithmic scale) as the primary inference unit, supplemented by a respondent-level bootstrap IC of the mean ratio as a crossover check with higher power; (b) paired t-tests with Holm-Bonferroni correction across 15 model pairs; (c) two unilateral equivalence tests (TOST) with a pre-recorded margin |Δratio| ≤ 0.10; (d) 95% paired percentile bootstrap confidence intervals (B = 5.000) for the univariate W₁, sliced Wasserstein, and MMD-RBF distances; (e) PLS toolpath modeling with the validated Python PLSPM package (Humble, 2020). Results. All models produced ≤ 1 ratios of variance against the actual baseline, in stark contrast to the analysis against the synthetic baseline: both PrefRL models severely compressed (Claude μ = 0.56 [SD 0.13], t(6) = −8.86, p = .0001; GPT-5-mini μ = 0.60 [SD 0.12], t(6) = −8.71, p = .0001) and were equivalent to each other by TOST at margin 0.10 (Δ = −0.033, p_TOST = .008), while the four non-PrefRL models were clustered near the unit (Mistral-Instruct DPO 0.97; Dolphin 2.8 0.96; Dolphin 3.0 0.91; Hermes-3 0.89). A respondent-level bootstrap with higher power left Mistral-Instruct and Dolphin 2.8 consistent with the unit, but detected slight and statistically significant compression in the two SFT Llama-3.1 models (Dolphin 3.0, Hermes-3), such that four of the six models significantly underestimate human variance. Any PrefRL × non-PrefRL differences remained statistically robust under Holm correction (|d_z| = 1.27–2.24, p_holm <.001 for the eight interclass pairs). The equivalence between Mistral-Instruct (DPO) and SFT-only models is not resolved at the pre-registered construct level (TOST p = .099–.248; n = 7, underpowered), but a higher-powered respondent-level bootstrap TOST, without widening the pre-recorded margin ±0.10, establishes equivalence for all three pairs (90% CI of the difference within ±0.10; p_TOST = .0002–.033), so H5 is established in the highest power test. Dolphin 2.8 produces the lowest mean Wasserstein-1 distance per item to the actual baseline (1.33 [1.19, 1.52]); the two PrefRL models are adjacent (GPT-5-mini 1.77 [1.63, 1.92], Claude 1.79 [1.66, 1.93]). The PLS-PM structural fidelity is the highest for Dolphin 3.0 (MAE = 0.07, r = .93) and the lowest for GPT-5-mini (MAE = 0.18), whose severe compression distorts structural relationships more than SFT models with higher variance. Finally, a recalibration per split sample using a small true anchor sample restored the compressed variance almost exactly (e.g., 0.57 → 0.99 for preference-based RL models) and trimmed the marginal distributional distance between one-half and two-thirds off-sample, demonstrating that compression is a correctable bias and not an irreducible limit. Conclusions. Against a true baseline with verified provenance, all LLMs evaluated underestimate the variance of human responses (there is no overdispersion in any kind of alignment), so that a sample of silicon from consumers systematically underestimates the heterogeneity on which segmentation and tail preference analysis depend. The overscattering that appears for models trained with DPO and SFT alone when compared against a synthetic baseline is a methodological artifact: the variance of that synthetic baseline is itself lower than that of real human respondents; the sign of the reported distortion is therefore dependent on the baseline. As a secondary standard, the alignment regime conditions the magnitude of compression (preference-based RL models compress about twice as much), although this comparison carries a numerical precision confounding factor and is reported as provisional. The actionable guidance for marketing researchers is to: (a) report the LLM alignment class as a design variable, expecting all classes to compress the variance to varying degrees; (b) use a real baseline with verified provenance whenever possible; (c) treat any claims of "fidelity" that depend on a synthetic comparator as conditional on the validity of the comparator itself; and (d) restrict silicon sampling to variance-tolerant uses such as instrument pretest, and avoid it when the research question depends on consumer heterogeneity (segmentation, polarized segment detection, or tail preferences) because all LLMs evaluated systematically underestimate that heterogeneity.
Source: Cultura
Institution: Universidad Pedagógica y Tecnológica de Colombia
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W7213520842