Agent A/B: Automated and Scalable A/B Testing on Live Websites with Interactive LLM Agents
Year: 2026
Type: conference-paper
Abstract: A/B testing is central to UI/UX design, yet our formative study with six industry practitioners revealed that it is slowed by scarce user traffic, long runtimes, and high operational costs. To address these challenges, we introduce Agent A/B, an end-to-end system that deploys large language model (LLM) agents with structured personas to interact with live webpages and generate scalable behavioral evidence before launch. In a case study on Amazon.com, Agent A/B simulated a between-subjects A/B test of filter panel designs with 1,000 agents and found that the reduced filter list produced more purchases, reproducing directional outcomes also observed in a parallel large-scale human experiment. Results further suggest that agent-based simulations can detect interface-sensitive behavioral differences and surface subgroup patterns while offering faster, lower-risk insights. We position Agent A/B as a complement to human testing, enabling earlier prototyping, pre-deployment validation, and hypothesis-driven UX evaluation. We will release our code upon paper acceptance.
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7153720518