A Pilot Mechanistic Study of Cataphoric Pronoun Resolution in GPT-2
Year: 2026
Type: preprint
Abstract: Cataphora, a pronoun that precedes its antecedent (as in "Before she left, Maria locked the door"), requires maintaining an unresolved dependency rather than retrieving an already-established referent. Human sentence-processing research shows that comprehenders actively and persistently search for cataphoric antecedents, but to our knowledge no prior work examines this mechanistically in language models. We present a pilot study in GPT-2-small that combines behavioral measurement, activation patching, logit-lens analysis, linear probing and attention analysis. We find a large behavioral effect that is predominantly, though not entirely, attributable to syntactic structure; evidence that gender information is carried in the residual stream between the pronoun and its antecedent; heads and layers whose ordering suggests separate search and read-out stages; and an unexpected gender asymmetry in the model's default resolution behavior. We report these findings with attention to their limits.
Author: MS, NEHAA SHRI
Institution: Vellore Institute of Technology University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7234182271
