A Training-free Modular Collaborative Framework: Application to Nasopharyngeal Endoscopy Report Generation
Year: 2025
Type: conference-paper
Abstract: Automatically generating accurate diagnostic reports from nasopharyngoscopy images holds significant clinical value. However, the lack of publicly available training datasets in this domain hinders the fine-tuning of pretrained models. Moreover, general-purpose large models often suffer from hallucinations due to the absence of empirical and structured prompt design. To address these challenges, we propose TfMC, a Training-free Modular Collaborative framework that synergistically integrates Multimodal Vision-Language Models (MVLMs) with Large Language Models (LLMs) for zero-shot nasopharyngoscopy report generation. We propose a diagnostic decision tree-driven prompt generation strategy based on clinical expertise, converting node routing decisions into visual question-answering to dynamically guide prompt generation. Additionally, we design a symptom-based image-node matching and knowledge generation strategy to enhance the evidential and targeted of prompts. Experimental results on a clinically dataset demonstrate that our framework effectively mitigates hallucination in zero-shot settings while ensure consistency between report content and diagnostic result, significantly improving report accuracy and interpretability.
Authors: Haoran Zhao, Xiaotian Ai, Wei Li, Zhe Xu, Huabin Wang, Tao Ren
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4413455910