Year: 2026
Type: preprint
Abstract: Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways do model responses resemble human responses, and where do they systematically diverge? The sheer breadth and diversity of the tasks humans can perform and think about pose a challenge for scalable and rigorous comparison between humans and models. We introduce CogGym, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials. CogGym uses a semi-automated, human-in-the-loop pipeline to standardize diverse experimental paradigms into a task-agnostic Experiment Markup Language (EML), enabling reproducible and faithful comparison at scale. For initial release, we curate and standardize 258 cognitive experiments from 100 papers that focuses on human commonsense reasoning, and evaluate 50 large language models against human responses. We find a clear scaling trend where larger and more recent AI models better reproduce human judgments. Yet AI models' improvement on such common reasoning tasks is considerably slower than the gains observed on formal-reasoning benchmarks like math and coding, and model--human fit remains well below human splithalf reliability ($R^2 = 0.93$ on text, $0.95$ on image, and $0.92$ on video) with the best models achieving $R^2 = 0.59$ on text, $0.58$ on image, and $0.43$ on video experiments. We intend for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.
Source: arXiv (Cornell University)
Authors: Ying, Lance, Wu, Jinzhou, Wang, Yingshan Susan, Aarya, Shivam, Buschoff, Luca M. Schulze, Chen, Harry, Collins, Katherine M., de Varda, Andrea, Fu, Shuhao, Houlihan, Sean Dae, Jagadish, Akshay K., Jiang, Guangyuan, Kiegeland, Samuel, Kurumisawa, Tetsu, Liu, Rongzhi, Liu, Ryan, Ma, Ningshan, McGregor, Kathryn, Strittmatter, Younes, Tsvilodub, Polina, Vigly, Jacob Hoover, Wu, Sarah, Xu, Enjie, Yun, Yiling, Allen, Kelsey, Brooke-Wilson, Tyler, Christian, Brian, Fedorenko, Evelina, Frank, Michael C., Franke, Michael, Gao, Tao, Gershman, Samuel J., Hawkins, Robert D., Hu, Jennifer, Jara-Ettinger, Julian, Kleiman-Weiner, Max, Levine, Sydney, Linzen, Tal, Lu, Hongjing, O'Donnell, Timothy, Ong, Desmond C., Piantadosi, Steven T., Saxe, Rebecca, Schulz, Eric, Shu, Tianmin, Sosa, Felix A., Sucholutsky, Ilia, Zhi-Xuan, Tan, Ullman, Tomer, Xu, Fei, Yildirim, Ilker, Zhu, Jian-Qiao, Griffiths, Thomas L., Gerstenberg, Tobias, Smith, Kevin, Tenenbaum, Joshua B.
Institutions: Harvard University, Massachusetts Institute of Technology, Cornell University, Johns Hopkins University, Helmholtz Munich, Massachusetts General Hospital, Santa Fe Institute, Dartmouth College, Stanford University, Princeton University, École Polytechnique Fédérale de Lausanne, University of Chicago, University of Tübingen, Canadian Institute for Advanced Research, University of California, Los Angeles, University of British Columbia, Vector Institute, Yale University, University of California, Berkeley, University of Washington, New York University, Mila - Quebec Artificial Intelligence Institute, McGill University, The University of Texas at Austin, National University of Singapore, University of Hong Kong
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7213926897