Year: 2022
Type: conference-paper
Abstract: Previous literature has proved that Pretrained Language Models (PLMs) can store factual knowledge.However, we find that facts stored in the PLMs are not always correct.It motivates us to explore a fundamental question: How do we calibrate factual knowledge in PLMs without re-training from scratch?In this work, we propose a simple and lightweight method CA-LINET to achieve this goal.To be specific, we first detect whether PLMs can learn the right facts via a contrastive score between right and fake facts.If not, we then use a lightweight method to add and adapt new parameters to specific factual texts.Experiments on the knowledge probing task show the calibration effectiveness and efficiency.In addition, through closed-book question answering, we find that the calibrated PLM possesses knowledge generalization ability after fine-tuning.Beyond the calibration performance, we further investigate and visualize the knowledge calibration mechanism.The code and data are available at https://github.com/dqxiu/CaliNet.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G3077791970, G7138386155
OpenAlex ID: W4385574113