Year: 2024
Type: conference-paper
Abstract: This paper investigates the capabilities of Large Language Models (LLMs) in the context of understanding their knowledge and uncertainty over questions.Specifically, we focus on addressing known-unknown questions, characterized by high uncertainty due to the absence of definitive answers.To facilitate our study, we collect a new dataset with Known-Unknown Questions (KUQ) and establish a categorization framework to clarify the origins of uncertainty in such queries.Subsequently, we examine the performance of open-source LLMs, fine-tuned using this dataset, in distinguishing between known and unknown queries within open-ended question-answering scenarios.The fine-tuned models demonstrated a significant improvement, achieving a considerable increase in F1-score relative to their prefine-tuning state.Through a comprehensive analysis, we reveal insights into the models' improved uncertainty articulation and their consequent efficacy in multi-agent debates.These findings help us understand how LLMs can be trained to identify and express uncertainty, improving our knowledge of how they understand and express complex or unclear information.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Funder: National Science Foundation
OpenAlex ID: W4402683869