Work

Multimodal and Bilingual Comparison of ChatGPT Performance on Physics Concept Inventories

Year: 2025
Type: dissertation
Abstract: In this study, the purpose is to examine whether a chatbot performs better, worse, or equally well when answering physics questions in Swedish compared to English, as well as whether its performance varies depending on whether the input includes images or only text. Can conclusions be drawn about whether the chatbot uses different databases for different languages, or whether it translates the input to access a larger database? The potential roles of chatbots in teaching and learning processes are also explored. Since ChatGPT-4o is the leading LLM application and one of the most commonly used by students and educators, it was chosen for this study. To assess ChatGPT’s ability to answer physics questions in Swedish, multiple-choice assessment tools were used: TUG-K, ADT2, CSEM, DIRECT, EMCS, and FCI. Each prompt included a brief instruction in Swedish or English and a screenshot of a physics question. ChatGPT’s performance in English and Swedish was compared and benchmarked against typical results from studies involving high school, college, or undergraduate students. A vision-based analysis was also conducted, categorizing items as text only, required image, or redundant image. Furthermore, linguistic analysis was performed to explore differences in reasoning between English and Swedish responses. Word counts were compared for correct and incorrect responses in both languages. Results show that ChatGPT performs slightly better when answering in English, achieving marginally higher scores across all six assessment tools (59% vs. 58%). Some Swedish responses featured awkward word choices suggestive of direct translations, hinting at potential internal translation to English. Correct answers tended to include more thorough analysis and contained more words than incorrect ones. Overall, response length was shorter in Swedish as well. Performance dropped significantly when solving items that required interpreting images (just above 40%) but was high for text-only items (over 85%) and items where images were redundant (above 75%). ChatGPT’s performance varied considerably across assessment tools and was inconsistent, occasionally applying sound reasoning yet selecting incorrect answers.
Author: Dexenius, Sam
Language: English
Subfield: Education
Open Access status: green
OpenAlex ID: W7112428998
Cites: 33
Cited by:
Related to: 10