Work

When 95% Isn't 95%: A Monte Carlo Investigation of Confidence Interval Reliability

Year: 2026
Type: preprint
Abstract: A nominal 95% confidence interval is expected to contain the true population parameter in approximately 95% of repeated samples. However, actual coverage depends on the statistical method, sample size, and characteristics of the underlying population distribution.This study uses Monte Carlo simulation to compare the empirical coverage probabilities and average widths of conventional Student's t confidence intervals and percentile bootstrap confidence intervals for estimating a population mean. The investigation examines three population distributions—normal, moderately skewed gamma, and strongly skewed exponential—across four sample sizes (n = 10, 30, 100, and 500). Each of the 12 experimental conditions was evaluated using 1,000 Monte Carlo replications and 1,000 bootstrap resamples per replication.Across all 12 conditions, Student's t intervals achieved coverage closer to the nominal 95% level than percentile bootstrap intervals. Differences were particularly pronounced for small samples and skewed distributions. Although percentile bootstrap intervals were generally narrower, this reduction in interval width was accompanied by lower empirical coverage. Differences between the two methods diminished as sample size increased.These findings illustrate that bootstrap confidence intervals do not necessarily provide more reliable coverage under non-normality. The results emphasize the importance of considering sample size, population skewness, and interval construction methods when interpreting confidence intervals.
Source: Figshare
Language: English
Open Access status: green
OpenAlex ID: W7233611218
Cites:
Cited by:
Related to: