Year: 2024
Type: conference-paper
Abstract: Nowadays, many works have attempted to introduce the large language models (LLM) into the fact-checking task by in-context learning (ICL), which has three labels: real, fake, and not enough information (NEI).The vanilla ICL method used by these works lets the LLM do the 3-label classification task and performs badly.We conducted experiments on multiple LLMs and found that LLMs can not identify the NEI label well, but can perform very well while only needing to classify real and fake labels.So based on the characteristics of the fact-checking task, we propose a framework named TTFact, which has two main ideas:Task-Transform and Template-Transform.The Task-Transform extracts the relevance scores of retrieved article sentences as features to judge whether the test case is the NEI label.The Template-Transform treats the authoritative retrieved sentences as news text with the real label to get a better-performed prompt template.Besides, we propose a contrastive learning method to finetune the retrieval model and get better relevance scores to improve the classification of the NEI label.The experiment on the widely used fact-checking dataset CHEF shows that our TTFact achieves the best LLM-based fact-checking method.When introducing the finetuned small language model (SLM) into our TTFact, our enhanced method TTFact-S achieves SOTA.
Author: Zhenghan Tang
Institution: University of Science and Technology of China
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: gold
OpenAlex ID: W4406497332