Year: 2024
Type: preprint
Abstract: In recent years many methods have been developed to understand the internal workings of neural networks, often by describing the function of individual neurons in the model. However, these methods typically only focus on explaining the very highest activations of a neuron. In this paper we show this is not sufficient, and that the highest activation range is only responsible for a very small percentage of the neuron's causal effect. In addition, inputs causing lower activations are often very different and can't be reliably predicted by only looking at high activations. We propose that neurons should instead be understood as a linear combination of concepts, and develop an efficient method for producing these linear explanations. In addition, we show how to automatically evaluate description quality using simulation, i.e. predicting neuron activations on unseen inputs in vision setting.
Source: arXiv (Cornell University)
Authors: Oikarinen, Tuomas, Weng, Tsui-Wei
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), RI: Medium: Foundations of Recourse Verification in Machine Learning, Towards Deploying a Science-Driven Regional Interoperable Science DMZ, CC* NPEO: Toward the National Research Platform, CCRI: ABR: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), Category II: A Prototype National Research Platform, Collaborative Research: SHF: Medium: Analog EDA-Inspired Methods for Efficient and Robust Neural Network Design, CC*DNI DIBBs: The Pacific Research Platform, CCRI: ENS: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI)
OpenAlex ID: W4396913060