Integrating Visual and Environmental Modalities for Rice Leaf Diseases Classification and its Severity Prediction with Dual-Output CNN
Year: 2026
Type: article
Abstract: Various living and non-living factors can significantly decrease the rice yield. Existing approaches to deep learning often fail to consider the influence of environmental factors on disease and severity classification. A multimodal deep learning model for predicting disease (10 classes) and severity (3 levels) of rice leaves continuously through a combination of visual disease features and environmental factors. The framework uses a dual-branch architecture: (i) a dual Convolution Neural Network (CNN)-based visual encoder to extract the features of the lesions, and (ii) a Multilayer Perceptron (MLP)-based environmental encoder to model the environmental conditions in the field. Fused images using cross-attention module, followed by dual head outputs for disease and severity prediction. Gradient-weighted Class Activation Mapping (Grad-CAM) visualization is used for the interpretation of lesion-focused decisions. The model is trained using multi-task loss and tested using five-fold cross-validation. This model obtained 97.9% accuracy, 97.6% macro-F1, and Cohen’s κ = 0.974 for disease classification, outperforming state-of-the-art baselines including ResNet50 (93.5%), EfficientNet-B4 (95.5%), and Swin Transformer (96.2%). For severity prediction, the results were 95.3% accuracy and a macro-F1 of 95.2% for the model. The addition of environmental features increased the accuracy by + 4.5% compared to the visual-only model. Grad-CAM confirmed that predictions were based on lesion regions associated with leaf disease. In future research, transformer-based multimodal fusion will be studied, real-time IoT-based environmental sensing will be added, and validation products for multi-location and multi-crop datasets will be studied.
Authors: Pritha Singha Roy, Vinay Kukreja
Institution: Chitkara University
Language: English
Topic: Smart Agriculture and AI
Subfield: Plant Science
Domain: Life Sciences
SDG: Zero hunger
Open Access status: hybrid
Estimated APC paid: $3,690
OpenAlex ID: W7128044093