Year: 2024
Type: preprint
Abstract: Often, a good explanation for a program's unexpected behavior is a bug in the programmer's code. But sometimes, an even better explanation is a bug in the programmer's mental model of the language or API they are using. Instead of merely debugging our current code ("giving the programmer a fish"), what if our tools could directly debug our mental models ("teaching the programmer to fish")? In this paper, we apply recent ideas from computational cognitive science to offer a principled framework for doing exactly that. Given a "why?" question about a program, we automatically infer potential misconceptions about the language/API that might cause the user to be surprised by the program's behavior -- and then analyze those misconceptions to provide explanations of the program's behavior. Our key idea is to formally represent misconceptions as counterfactual (erroneous) semantics for the language/API, which can be inferred and debugged using program synthesis techniques. We demonstrate our framework, WatChat, by building systems for explanation in two domains: JavaScript type coercion, and the Git version control system. We evaluate WatChatJS and WatChatGit by comparing their outputs to experimentally-collected human-written explanations in these two domains: we show that WatChat's explanations exhibit key features of human-written explanation, unlike those of a state-of-the-art language model.
Source: arXiv (Cornell University)
Authors: Chandra, Kartik, Collins, Katherine M., Crichton, Will, Chen, Tony, Li, Tzu-Mao, Weller, Adrian, Nigam, Rachit, Tenenbaum, Joshua, Ragan-Kelley, Jonathan
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CAPA: Collaborative Research: ARION: Taming Heterogeneity with DSLs, Approximation, and Synthesis, Graduate Research Fellowship Program (GRFP), A Center for Brains, Minds and Machines: the Science and the Technology of Intelligence, CAREER: Differentiable Programming for Visual Computing, G8972024003
OpenAlex ID: W4392678284