Work

Note on the quadratic penalties in elastic weight consolidation

Year: 2018
Type: letter
Abstract: Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel algorithm designed to safeguard against catastrophic forgetting in neural networks. EWC can be seen as an approximation to Laplace propagation (Eskin et al, 2004), and this view is consistent with the motivation given by Kirkpatrick et al (2017). In this note, I present an extended derivation that covers the case when there are more than two tasks. I show that the quadratic penalties in EWC are inconsistent with this derivation and might lead to double-counting data from earlier tasks.
Language: English
Open Access status: green
OpenAlex ID: W2792568620
Cites: 12
Cited by: 210
Related to: 9