Work

Toward comprehensive real-time scene understanding in ophthalmic surgery through multimodal image fusion

Year: 2026

Type: article

Abstract: Purpose The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery. Methods We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a Yolo-NAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence. Results Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from $$284\,\upmu \hbox {m}$$ 284 μ m (OPMI only) to $$33\,\upmu \hbox {m}$$ 33 μ m (multimodal). Conclusion Feature fusion of multimodal imaging can enhance multitask prediction accuracy compared to single-modality processing, and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multimodal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.

Source: International Journal of Computer Assisted Radiology and Surgery

DOI: 10.1007/s11548-026-03711-2

Authors: Nikolo Rohrmoser, Ghazal Ghazaei, Michael Sommersperger, Nassir Navab

Institutions: Technical University of Munich, Carl Zeiss (Germany)

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

SDG: Good health and well-being

Open Access status: hybrid

Estimated APC paid: $4,890