How much does an AI answer change on its own? A small-sample measurement of generative-engine volatility
Year: 2026
Type: preprint
Abstract: Generative search engines return different answers to the same question at different times, with no change to the underlying web or to the queried brand. This makes before-and-after measurement of any visibility work hard to trust. We ran a small, fixed basket of buyer questions on a clean ChatGPT surface across three non-consecutive windows, changing nothing in between, and recorded how often the set of named businesses changed. Across 36 observations the answer changed 27.8 percent of the time (10 status changes). We report this as a directional, small-sample result, not a population statistic. We set it alongside the large-scale study by Petra Labs (7,200 responses), which measured within-day model non-determinism and reached a compatible conclusion: short-run before-and-after comparisons are usually indistinguishable from sampling noise. Our contribution is narrow but distinct. We measure change across weeks rather than within a single day, which captures drift in the web and the index as well as model non-determinism, and which matches the window a real client engagement spans.
Author: William, David
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7171310083
Cites:
Cited by:
Related to: