Year: 2023
Type: conference-paper
Abstract: Large language models (LLMs) have recently shown great advances in a variety of tasks, including natural language understanding and generation.However, their use in high-stakes decision-making scenarios is still limited due to the potential for errors.Selective prediction is a technique that can be used to improve the reliability of the LLMs by allowing them to abstain from making predictions when they are unsure of the answer.In this work, we propose a novel framework for adaptation with self-evaluation to improve the selective prediction performance of LLMs.Our framework is based on the idea of using parameter-efficient tuning to adapt the LLM to the specific task at hand while improving its ability to perform self-evaluation.We evaluate our method on a variety of question-answering (QA) datasets and show that it outperforms state-of-the-art selective prediction methods.For example, on the CoQA benchmark, our method improves the AUACC from 91.23% to 92.63% and improves the AUROC from 74.61% to 80.25%.
Institutions: University of Wisconsin–Madison, Google (United States)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G1323911732, CAREER: Towards Better Understanding, Robustness, and Efficiency of Deep Learning, G1574964069, SaTC: CORE: Frontier: Collaborative: End-to-End Trustworthiness of Machine-Learning Systems, Collaborative Research: RI: Small: Theoretical Foundations: TheAdvantage of Deep Learning over Traditional Shallow Learning Methods, CAREER: Algorithmic Foundations and Modern Applications for Program Synthesis, G7452299184, FMitF: Collaborative Research: Formal Methods for Machine Learning System Design
OpenAlex ID: W4389519963