Work

Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining

Year: 2026

Type: conference-paper

Abstract: Continual Learning (CL) is essential for enabling self-evolving large language models (LLMs) to adapt and remain effective amid rapid knowledge growth.Yet, despite its importance, little attention has been given to establishing statistical reliability guarantees for LLMs under CL, particularly in the setting of continual domain pretraining (CDP).Conformal Prediction (CP) has shown promise in offering correctness guarantees for LLMs, but it faces major challenges in CDP: testing data often stems from unknown or shifting domain distributions, under which CP may no longer provide valid guarantees.Moreover, when high coverage is required, CP can yield excessively large prediction sets for unanswerable queries, reducing informativeness.To address these challenges, we introduce an adaptive rejection and non-exchangeable CP framework.Our method first estimates the distribution of questions across domains in the test set using transformer-based clustering, then reweights or resamples the calibration data accordingly.Building on this, adaptive rejection CP allows the LLM to selectively abstain from answering when its confidence or competence shifts significantly.Extensive experiments demonstrate that our framework enhances both the effectiveness and reliability of CP under CDP scenarios.Our code is available at: https://github.com/AlearZhou/CPCL20 40 60 80 Quantile (%) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.

Source: Findings of the Association for Computational Linguistics: ACL (Findings of ACL)

DOI: 10.18653/v1/2026.findings-acl.388

Authors: Xiaofan Zhou, Lu Cheng

Institution: University of Illinois Chicago

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Science Foundation, Cisco Systems, National Institutes of Health

Awards: Collaborative Research: III: Medium: Knowledge discovery from highly heterogeneous, sparse and private data in biomedical informatics, SCH: Fair and Interpretable Prediction of Alzheimer's Disease and MCI using LLMs, G6671297155, Collaborative Research: FDT-BioTech: Advancing Alzheimer's Disease Understanding and Treatments through Digital Twin Modeling, CAREER: Conformal Methods for Responsible Language Models