Work

Testing AI Chatbot Instructions in Simulation: Where ChatGPT, Gemini, and Claude Misstep When Guiding Arduino Beginners

Year: 2026
Type: preprint
Abstract: When beginners turn to artificial intelligence (AI) chatbots for Arduino help, it is unclear whether wiring and code work when a beginner follows every step exactly. To evaluate this, I tested ChatGPT, Gemini, and Claude across a range of beginner projects. The free versions were tested on 12 projects across 4 difficulty levels in Wokwi, a free online simulator, and I followed all instructions exactly. For tests that did not work as given, I sent up to 5 symptom-only complaints and repeated those pairs, so there were 52 counted tests. I also performed 5 tests on real hardware. Out of 36 initial tests, 28 (78%) worked as given, with ChatGPT and Gemini getting 9 out of 12 and Claude getting 10 out of 12. All 36 initial tests worked in the end, though across these 52 counted tests, 1 test never worked. The 22 failures came from part assumptions, specifically button legs (9), display type (9), and buzzer type (4). No fix created new problems, nothing was unsafe, and real hardware matched Wokwi in all 5 tests. The main problem is that chatbots assume specific part types or leg connections without telling the beginner. This makes it hard to tell what went wrong. To prevent this, beginners should name the exact part they are using, and chatbots should state their assumptions or ask questions before giving instructions.
Language: English
Open Access status: green
OpenAlex ID: W7235241488
Cites:
Cited by:
Related to: 10