Year: 2026
Type: conference-paper
Abstract: While critical for alignment, Supervised Fine-Tuning (SFT) incurs the risk of catastrophic forgetting, yet the layer-wise emergence of instruction-following capabilities remains elusive.We investigate this mechanism via a comprehensive analysis utilizing informationtheoretic, geometric, and optimization metrics across model scales (1B-32B).Our experiments reveal a distinct depth-dependent pattern: middle layers (20%-80%) are stable, whereas final layers exhibit high sensitivity.Leveraging this insight, we propose Mid-Block Efficient Tuning, which selectively updates these critical intermediate layers.Empirically, our method outperforms standard LoRA up to 10.2% on GSM8K (OLMo2-7B) with reduced parameter overhead, demonstrating that effective alignment is architecturally localized rather than distributed.The code is publicly available at https://github.com/lshowway/base.
Institutions: Hefei University, Lanzhou University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166860485