MiniMind-Dense: a Small Language Model that Supports HR Services by Adopting MiniMind with Supervised Fine-Tuning and LoRA
Year: 2025
Type: article
Abstract: Small and medium-sized enterprises (SMEs) form the backbone of Malaysia’s economy, yet they often lack resources to adopt advanced AI tools. Generative AI and large language models (LLMs) promise to transform human resources (HR) by automating tasks like policy guidance, recruitment support, and employee coaching [1]. However, generic LLMs trained on broad data do not capture local legal norms or HR-specific knowledge and may produce irrelevant or non-compliant advice. To address this, we adopted and adapted MiniMind [2] , a Malaysian HR-focused language assistant. Starting from a compact GPT-style base model, we implemented a three-stage refinement pipeline: (1) Supervised fine-tuning (SFT) on an expanded, HR-domain dialogue dataset; (2) Reinforcement learning from human feedback (RLHF) via Proximal Policy Optimization to align outputs with HR professionals’ preferences; and (3) LoRA parameter-efficient tuning to inject final domain expertise. Through the refinement pipeline, We have designed a new architecture, namely, “MiniMind-Dense”, by incorporating other Transformer improvements such as grouped-query attention, rotary embeddings, SwiGLU to achieve the goals of the research. Extensive evaluation on Malaysian HR queries shows dramatic improvements: e.g., BLEU score [3] rose from ~5% in the pretrained model to ~79% after LoRA tuning (and ROUGE‑L [4] from ~36% to ~92%). Qualitative analysis confirms highly fluent, relevant, and human-like responses, unlike the generic outputs of the base model. These results demonstrate the feasibility of a localized, aligned SLM as an AI HR assistant. Future work will integrate retrieval (RAG) [5] for factual grounding and expand multilingual capabilities.
Author: Darren Chai Xin Lun
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7217866496