Year: 2026
Type: preprint
Abstract: Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the three-day workshop Benchmarks in Leipzig with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. We present the resulting collection of 100~questions. We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs and their predecessors, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models. After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions. This demonstrates that the mathematical reasoning capabilities of LLMs are becoming impressive. In September 2026, we added a fourth stage in which the next generation of models attempted all 100 questions once more, after which only 1 question remains unsolved.
Source: arXiv (Cornell University)
Authors: Balakin, Andrei, Bóna, Miklós, Brandenburg, Marie-Charlotte, Briand, Clara, Cortes, Veronica Calvo, Cox, Shelby, De Loera, Jesus A., Deligeorgaki, Danai, Friedman, Hannah, Gehrunger, Tim, Giardino, Chiara, Griffeth, Stephen, Hashemi, Baran, Hoster, Elena, Ivanov, Alexander, Jain, Nupur, Jal, Aryaman, Kayser, Leonie, Koefler, Joris, Kühn, Kevin, Kummer, Mario, Lotter, Felix, Marczinzik, René, Miller, Victor S., Morales, Alejandro, Panova, Greta, Petrella, Gianni, Pflueger, Nathan, Ramesh, Lakshmi, Rieke, Nikolas, Rodriguez, Carlos, Rosana, Andrea, Salizzoni, Flavio, Schmidt, Otto T. P., Schmitz, Sven Ulf, Marin, Lina Maria Simbaqueba, Sodomaco, Luca, Stump, Christian, Sturmfels, Bernd, Blomenhofer, Alexander Taveira, Telen, Simon, Tuchel, Philipp, Verkama, Emil, Waller, Carl Felix, Weigert, Julian, Werner, Annette, Williams, Nathan, Zibrowius, Claudius
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7163676334