SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
Year: 2026
Type: conference-paper
Abstract: Can large language model agents develop industry-level mobile applications? We introduce SWE-Bench Mobile, a benchmark for evaluating coding agents on realistic software engineering tasks derived from a production iOS codebase. Unlike existing benchmarks that focus on isolated problems or bug fixes, SWE-Bench Mobile captures the full complexity of industrial development: multi-modal inputs (PRDs and Figma designs), a large-scale mixed Swift/Objective-C codebase, and comprehensive test suites. We evaluate 22 agent-model configurations across four coding agents: three commercial (Cursor, Codex, Claude Code) and one open-source (OpenCode), and find that even the best configurations achieve only 12% task success rate. Our analysis reveals that (1) agent design matters as much as model capability: the same model shows up to 6x performance gap across agents, (2) commercial agents consistently outperform open-source alternatives, and (3) simple ''Defensive Programming'' prompts raise the test pass rate by 7.4 points over the baseline, while overly complex prompts hurt task success. These findings highlight a significant gap between current agent capabilities and industrial requirements, while providing actionable insights for practitioners and researchers. We release SWE-Bench Mobile as a hosted benchmark challenge to prevent data contamination and ensure fair evaluation. The public leaderboard is available at https://swebenchmobile.com.
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7196930851