Mitigating Structural Knowledge Collapse in Domain-Specific LLMs via Morpheme-Aware KV-Aggregation
Year: 2026
Type: conference-paper
Abstract: Standard tokenizers over-fragment domain terms, disrupting morpheme semantics.We characterize this representational misalignment as Structural Knowledge Collapse (SKC), where attention mechanisms fail to reconstruct coherent concepts from fragmented inputs.While existing input-centric solutions like vocabulary expansion address this, they necessitate expensive embedding retraining and neglect internal attention compositionality.To this end, we introduce Morphemeaware KV-aggregation Attention (MorphKA), a lightweight adapter that dynamically consolidates fragments without tokenizer changes.Bypassing tokenizer retraining, MorphKA employs a dual-phase strategy-Input-Level Morpheme Aggregation (IMA) and Context-Aware KV-Aggregation (AMRF)-to stabilize morpheme spans and synthesize higher-order concepts.Experiments on medical and legal benchmarks show MorphKA outperforms vocabulary adaptation baselines by 3.2-4.6%,reaching 7.9% on high-fragmentation terms.Moreover, MorphKA reduces catastrophic interference on general capabilities by 18-22% with ∼80% fewer parameters than embedding retraining approaches.
Authors: Yuxuan Si, Zheqi Lv, Chengxi Zang, Zhengyu Chen, Fei Wu
Institutions: Zhejiang University, Cornell University, Meituan (China)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166881211