Work

Multilingual Multimodal Software Developer for Code Generation

Year: 2025
Type: preprint
Abstract: The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and flowcharts used in real-world software development. To bridge this gap, we introduce MM-Coder, a Multilingual Multimodal software developer. MM-Coder integrates visual design inputs-Unified Modeling Language (UML) diagrams and flowcharts (termed Visual Workflow)-with textual instructions to enhance code generation accuracy and architectural alignment. To enable this, we developed MMc-Instruct, a diverse multimodal instruction-tuning dataset including visual-workflow-based code generation, allowing MM-Coder to synthesize textual and graphical information like human developers, distinct from prior work on narrow tasks. Furthermore, we introduce MMEval, a new benchmark for evaluating multimodal code generation, addressing existing text-only limitations. Our evaluations using MMEval highlight significant remaining challenges for models in precise visual information capture, instruction following, and advanced programming knowledge. Our work aims to revolutionize industrial programming by enabling LLMs to interpret and implement complex specifications conveyed through both text and visual designs.
Authors: Chai, Linzheng, Yang, Jian, Liu, Shukai, Zhang, Wei, Wang, Liran, Jin, Ke, Sun, Tao, Liu, Congnan, Zhang, Chenchen, Zhu, Hualei, Liu, Jiaheng, Wu, Xianjie, Zhang, Ge, Liu, Tianyu, Li, Zhoujun
Language: English
Open Access status: green
OpenAlex ID: W6967174718
Cites: 89
Cited by: 10
Related to: 10