Multimodal pill image recognition in natural environments using context-aware cross-modal attention
Year: 2026
Type: article
Abstract: Medication errors remain a major source of preventable harm, and identifying a pill from a photograph taken in an uncontrolled setting is still difficult because the informative cues-color, shape, surface texture, and imprinted text-are unequally reliable from one image to the next. We present a recognition framework for natural environments in which a fusion module, context-aware cross-modal attention (CACMA), weighs these four modalities separately for every image using a query formed from their joint global context, an explicit confidence branch that estimates per-modality reliability, and a three-way integration of attended, confidence-weighted, and residual features, on top of \(\hbox {U}^2\) -Net segmentation (98.06% validation IoU) and connected-component localization. All results follow a unified end-to-end protocol, with segmentation, localization, OCR, and fusion executed at inference time, over three random seeds on two real-world datasets. On CURE (196 classes) CACMA attains 91.13±0.27% Top-1 accuracy and \(95.50\pm 0.16\%\) mAP, exceeding the published results of Ling et al. and Rádli et al. by 22.83 and 5.41 pp and summation fusion by 3.15 pp ( \(p<0.001\) ); the 0.76 pp margin over weighted summation is not statistically significant, which we report as parity rather than superiority, while CACMA attains the smallest across-seed variability of any method evaluated ( \(\pm 0.27\) pp). On OGYEI-v2 (112 classes) it reaches \(83.23\pm 0.73\%\) Top-1 accuracy. What fixed-weight fusion cannot reproduce is directly measurable: the confidence assigned to the text modality rises from 0.362 to 0.602 when an imprint is legible ( \(p = 1 \times 10^{-55}\) ), with the released weight redistributed to the visual modalities. A fully fine-tuned ViT-B/16 is not significantly more accurate ( \(92.07\pm 0.64\%\) ; McNemar \(p=0.32\) ) while using 6.4 \(\times \) the parameters and offering no modality-level explanation, placing CACMA at a favorable accuracy-efficiency-interpretability operating point. Paired significance testing applied to the ablations shows that no individual component accounts for the result on its own under this segmentation quality; together with calibration, saliency and failure analyses, these tests delimit where multimodal attention fusion helps and where it does not.
Source: Scientific Reports
Authors: Linh Nguyen Thi My, Tham Vo, Vinh Truong Hoang
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7225828849
FWCI:
Cites:
Cited by:
Related to: