Concept Learning: Understanding and Extracting Domain-specific concepts in medical Observations
Year: 2024
Type: dissertation
Abstract: In clinical environments, high-risk decisions require trust and understanding, which can hinder the use of high-performance models such as deep neural networks due to their complexity and opacity. Despite the development of numerous techniques aimed at enhancing interpretability and transparency, the explanations provided are not always human-interpretable and remain underexplored as a means of improving downstream tasks. Furthermore, clinical data often faces the challenge of limited annotated data, despite a large volume of unlabeled data. This study aims to extract human-interpretable concepts from X-ray images and respective reports using multimodal learning with self-supervision, which allows learning directly from the input data without annotations. Moreover, the objective is to enhance the interpretability and performance in clinical tasks. Therefore, the work integrates a vision language model with self-supervision techniques to extract domain-specific textual concepts from the joint representation of X-ray images and reports, while also investigating changes in the model framework to improve the alignment between the concepts and image features. Finally, these extracted concepts are incorporated as features into an X-ray image classification model to assess their impact on performance. A key contribution is the development of a novel dataset generator with controllable ground truth concepts, providing a flexible and scalable tool to test concept-based ap- proaches and identifying appropriate methods for medical data. Additionally, the results demonstrate that the incorporation of large language models within self-supervised vision-language frameworks significantly enhances the extraction of concepts, leading to direct improvements in diagnostic classification performance and interpretability. These findings highlight the potential benefits of integrating multimodal frameworks with self-supervision for concept extraction, thereby establishing a foundation for more inter- pretable, transparent, and reliable artificial intelligence-driven healthcare solutions.
Author: Silva, Matilde Gonçalinho
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7197058984