Work

Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

Year: 2024

Type: preprint

Abstract: The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2402.12233

Authors: Qiu, Zihan, Huang, Zeyu, Huang, Youcheng, Fu, Jie

Institutions: University of Edinburgh, Sichuan University, Hong Kong University of Science and Technology, Tsinghua University

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Institute for Catastrophic Loss Reduction

OpenAlex ID: W4391988183

Cites: 1

Cited by: 7

Related to: 10