A Multimodal Deep Learning Framework for Herbal Plant Classification with Large Language Model Integration
Year: 2026
Type: article
Abstract: Accurate identification of herbal plants remains challenging due to high species diversity and the need for specialized botanical knowledge. This study proposes a multimodal artificial intelligence framework that integrates deep learning-based image classification with a large language model to provide both plant identification and descriptive information. The objective is to develop a more informative and practical system that extends beyond conventional classification approaches. The method employs a ResNet50V2 architecture with transfer learning to classify herbal plant images, combined with a large language model to automatically generate descriptions of plant benefits. The model was trained on 19,213 images across 131 plant classes. Experimental results demonstrate an accuracy of 89.43%, with a macro F1-score of 0.85 and a weighted F1-score of 0.89, indicating strong and consistent performance across classes. The integration of visual recognition and language generation enables the system to deliver richer, user-oriented outputs compared to existing methods that only provide classification labels. In addition, the system achieves real-time inference performance, making it suitable for web-based applications. This study contributes a scalable and comprehensive multimodal framework for herbal plant identification, highlighting the effectiveness of combining computer vision and large language models to enhance both accuracy and usability.
Institution: Universitas Majalengka
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $100
OpenAlex ID: W7166852829
