Year: 2025
Type: conference-paper
Abstract: Large Language Models (LLMs) enhanced with external contexts, such as through retrieval-augmented generation (RAG), often face challenges in handling imperfect evidence.They tend to over-rely on external knowledge, making them vulnerable to misleading and unhelpful contexts.To address this, we propose the concept of context-robust LLMs, which can effectively balance internal knowledge with external context, similar to human cognitive processes.Specifically, context-robust LLMs should rely on external context only when lacking internal knowledge, identify contradictions between internal and external knowledge, and disregard unhelpful contexts.To achieve this goal, we introduce Grft, a lightweight and plug-and-play gated representation fine-tuning approach.Grft consists of two key components: a gating mechanism to detect and filter problematic inputs, and low-rank representation adapters to adjust hidden representations.By training a lightweight intervention function with only 0.0004% of model size on fewer than 200 examples, Grft can effectively adapts LLMs towards context-robust behaviors.
Authors: Shenglai Zeng, Pengfei He, Kai Guo, Tianqi Zheng, Hanqing Lu, Yue Xing, Hui Liu
Institutions: Michigan State University, Amazon (United States)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: Interactive, Individualized Professional Learning for Elementary School Teachers: Enhancing Content and Pedagogical Content Knowledge as a Basis for Improving Practice, NeTS: Small: Exploring the Non-Standardized Polices, Operations, and Requirements for 5G Cellular Networks and Beyond: Advancing the Modeling, Tools, and Evaluation, III:Medium:Computation and Communication Efficient Distributed Learning, Collaborative Research: SaTC: CORE: Medium: Safeguarding Next-Generation Emergency Services (NG-9-1-1) over Cellular Networks: From Design to Practice, Intelligent, Adaptive Program with Just-in-time Feedback for Preservice Teachers, Collaborative Research: III: Medium: Graph Neural Networks for Heterophilous Data: Advancing the Theory, Models, and Applications, TRTech-PGR: Maize Hub: An Open-source Simulation Platform & Resources Integrating Extensive Field Trial Data, DNA Sequences, and Historical Weather Records, TRTech-PGR: Connecting sequences to functions within and between species through computational modeling and experimental studies
OpenAlex ID: W4412945404