Work

Investigating the effect of Input Modality on Reasoning in VLMs using Zero-shot evaluation

Year: 2026

Type: conference-paper

Abstract: This project investigates how input modality affects mathematical reasoning in vision-language models (VLMs). Using 100 samples from the GSM8K benchmark, we evaluate Qwen2-VL-2B-Instruct and LLaVA-v1.6-Mistral-7B across three conditions: text-only, rendered image, and a modality mismatch condition where image and text describe different problems. Both models show substantial accuracy drops under visual input — Qwen2 falls from 55% to 30% and LLaVA from 40% to 21%. In the mismatch condition, both models follow the text modality in over 75% of cases, revealing strong text dominance when modalities conflict. These findings suggest that rendering math problems as images meaningfully impairs reasoning in current VLMs, even when visual content is clean and legible.

Source: Digital Commons - University of South Florida (University of South Florida)

Authors: Ghosh, Rodela, Gupta, Aviral, Monjur, Ocean, Datta, Shrestha, Chhabra, Anshuman

Institution: University of South Florida

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7202322126

FWCI:

Cites:

Cited by:

Related to: 10