Math Neurosurgery: Isolating Language Models’ Math Reasoning Abilities Using Only Forward Passes
Year: 2025
Type: conference-paper
Abstract: Math reasoning is an active area of Large Language Model (LLM) research because it is a hallmark of artificial intelligence and has implications in several domains, including math education.However, few works have explored how math reasoning is encoded within LLM parameters and if it is a skill that can be isolated within models.Doing so could allow targeted intervention to improve math performance without altering non-math behavior and foster understanding of how models encode math reasoning.We introduce Math Neurosurgery (MathNeuro), a computationally efficient method we use to isolate math-specific parameters in LLMs using only forward passes.MathNeuro builds on existing work by using weights and activations to calculate parameter importance, but isolates math-specific parameters by filtering out those important for general language tasks.Through pruning parameters MathNeuro identifies, we delete a LLM's math reasoning ability without significantly impacting its general language ability.Scaling the identified parameters by a small constant improves a pretrained or instruction-tuned LLM's performance by 4-17% on GSM8K and 5-35% on MATH while leaving non-math behavior unaltered.Math-Neuro is also data efficient: most of its effectiveness holds when identifying math-specific parameters using a single sample.MathNeuro highlights the potential for future work to intervene on math-specific parameters. 1
Institution: University of Virginia
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412945148