Multimodal model of cultural relics based on multi-layer attention visual encoding and distributed language computing framework
Year: 2025
Type: conference-paper
Abstract: Current methods for cultural relic identification, description generation, and image generation still have many limitations in cross-modal information fusion, historical style consistency, and semantic alignment. In addition, due to the lack of a high-quality archaeological knowledge base, existing description generation methods based on LLMs often find it challenging to ensure the historical consistency and professionalism of the text content. Cultural relic image generation still lacks style constraints and material detail control, affecting the generated images' authenticity and archaeological credibility. Therefore, this paper proposes a multimodal model of cultural relics based on multi-layer attention visual encoding and distributed language computing, combined with Swin Transformer V2 for high-precision visual feature extraction, using CLIP for cross-modal alignment, and generating historically consistent cultural relic descriptions and optimized prompts through LLM, guiding StyleGAN to generate high-fidelity cultural relic images that conform to the historical style. Results show that the research method achieved excellent performance in cultural relic classification, description generation, and image generation tasks on the iMet Collection 2019 and 2020 datasets. The proposed method achieved 90.2% accuracy and a micro-average F1 of 0.759 in cultural relic classification. In description generation, CLIP + KIMI-1.5 outperformed GPT-4.0 and other baselines, with BLEU-4 of 45.1 and ROUGE-L of 52.3. Swin Transformer + CLIP + KIMI-1.5 + StyleGAN attained the lowest FID (12.5) and LPIPS (0.098) for image generation, demonstrating superior style consistency and visual quality.
Authors: ZiYuan Xu, DanJie Xu, Feng Shen, BiRui Zhao
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4412120079