Work

BOLT: Building Open-Source LLMs for Your Target Domain via Automated Hierarchical Knowledge Distillation

Year: 2025

Type: article

Abstract: Adapting open-source large language models (LLMs) to specialized domains remains a critical challenge due to domain knowledge gaps, data scarcity, and reference hallucination. Existing approaches often neglect the structural characteristics of domain knowledge and fail to provide principled estimations of knowledge scope, resulting in data homogenization and suboptimal adaptation, while leaving reference hallucination unmitigated. This paper introduces BOLT(Building Open-source LLMs for your Target domain), a modular end-to-end framework that tailors open-source LLMs for domain-specific scenarios. BOLT systematically estimates domain scope, constructs structured hierarchical knowledge trees, distills diverse and semantically aligned training data from advanced teacher LLMs, and employs curriculum learning for progressive model optimization. To address reference hallucination, BOLT substitutes generative methods, which are susceptible to hallucinations, with a matching-based strategy, thereby alleviating the problem and significantly improving reference recommendation accuracy. Extensive experiments across diverse domains and models demonstrate that BOLT enables the efficient modeling of structured hierarchical domain knowledge and effectively enhances reference recommendation accuracy while preserving both training efficiency and robustness throughout the adaptation process.

Source: Applied Sciences

DOI: 10.3390/app152111393

Authors: Runze Lu, Zhaoyu Fan, Guanjie Wang, Qingjiang Shi

Institution: Tongji University

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,896

Funders: Tongji University, Fundamental Research Funds for the Central Universities