Work

Multimodal Food Image Classification with Large Language Models

Year: 2024

Type: article

Abstract: In this study, we leverage advancements in large language models (LLMs) for fine-grained food image classification. We achieve this by integrating textual features extracted from images using an LLM into a multimodal learning framework. Specifically, semantic textual descriptions generated by the LLM are encoded and combined with image features obtained from a transformer-based architecture to improve food image classification. Our approach employs a cross-attention mechanism to effectively fuse visual and textual modalities, enhancing the model’s ability to extract discriminative features beyond what can be achieved with visual features alone.

Source: Electronics

DOI: 10.3390/electronics13224552

Authors: Jun-Hwa Kim, Nam-Ho Kim, Donghyeok Jo, Chee Sun Won

Institutions: Konyang University, Dongguk University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,728

Funders: National Research Foundation, National Research Foundation of Korea, Ministry of Science and ICT, South Korea