Star at PalmX 2025: Arabic Cultural Understanding via Targeted Pretraining and Lightweight Fine-tuning
Year: 2025
Type: conference-paper
Abstract: We present a two-stage framework for enhancing Arabic cultural understanding in small language models, specifically designed for PalmX 2025(Alwajih et al., 2025) Shared Task 1: General Culture Evaluation.Our approach combines continuous pretraining on a culturallyenriched Arabic corpus spanning 10 Arab countries and different cultural domains, followed by supervised fine-tuning on cultural questionanswering data.Using Parameter-Efficient Fine-Tuning (PEFT) (Zhang et al., 2025) with LoRA on the Qwen3-4B base model, we achieve 74% accuracy on the development set and 64% on the blind test set, ranking our team ninth in the competition.Our system demonstrates the effectiveness of targeted cultural pretraining for improving Arabic language models' cultural competency while maintaining computational efficiency.
Authors: Eman Elrefai, Esraa Khaled, Alhassan Ehab
Institutions: Alexandria University, Cairo University, Minia University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4416037747