Year: 2026
Type: conference-paper
Abstract: We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source 1 benchmark designed to systematically evaluate the reliability and limitations of LLMs for construction tasks in Minecraft.The MineCEraft benchmark comprises 723 domain-expert hand-crafted natural-language instructions with programmatically verifiable evaluation, spanning 17 distinct task categories, providing a safe and controllable experimental environment for assessing LLMs' ability to perform realistic construction engineering tasks.With this benchmark, we conduct an in-depth evaluation of state-of-the-art LLMs and perform a detailed error analysis, revealing key failure modes and practical challenges in applying LLMs to construction engineering tasks. Minecraft Agent … MineCEraft BenchmarkNatural Language Instruction System: You are builder, a construction engineer and also a worker at a construction site.… User: Build a stone arch bridge. A: Structural Element -FoundationTask: Install a 9×9 pile cap, 2 blocks thick, with six 1×1 drilled piles, each 5 blocks deep, with a 1-block span.Use stone blocks. Failed in Shape & Structural Stability.The drilled piles are built above the ground instead of below it. B: Building -Basic BuildingTask: Let's build a house with pillars made of stone, walls made of dirt, and a roof made of green. Passed in Material.Green blocks at the top layer, and the corner and edge materials at walking height are correctly placed. A: Structural Element -FrameTask: Build the frame (including stairs) of a 10-story building using concrete. Failed in Shape & Structural Stability.The stairs are not climbable, and the headroom between levels is insufficient. C: Architectural Element -AccessibilityTask: Build a wheelchair-accessible house for two elderly people and one care-giver... Failed in Shape.The path from inside the building to the outside is not flat, violating the accessibility requirement. C: Architectural Element -CreativityTask: Build a building with at least three different ceiling heights. Failed in Physical Plausibility.The roof is placed in midair without support.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7204886048