LLM-Modulo Planning Agents with External Plan Verification and Iterative Self-Repair
Year: 2024
Type: article
Abstract: Large language models can translate natural-language goals into action sequences, but an action sequence is not a plan unless every precondition holds and the final state satisfies the goal. This study evaluates a modulo-style planning agent that assigns proposal generation to an LLM and delegates correctness to an external symbolic verifier. The evaluation used the PlanBench repository artifacts for Blocksworld: every stored GPT-4 candidate in four domain-size conditions and five prompt forms (6,000 direct plans), together with all 50 verifier-feedback trajectories containing 286 sequential candidates. Each candidate was parsed into PDDL actions and independently checked with VAL against the corresponding domain and problem file. Across the direct corpus, 934 of 6,000 candidates were valid (15.57%). Validity reached 27.53% in standard Blocksworld but only 3.60% after semantic obfuscation. State-tracking prompts were strongest in both conditions, while PDDL-form prompts did not compensate for weak state-transition reasoning. On the matched 50-instance repair set, direct execution completed 4 tasks (8%) and would have executed 46 invalid plans. A verifier gate preserved the same 8% completion while reducing invalid execution to zero through abstention. Iterative verifier-feedback repair completed 41 tasks (82%), repaired 37 of 46 initially invalid plans (80.43%), accepted no invalid plan, and required 5.72 candidate generations on average. Completion saturated after the ninth candidate; the paired gain was 74 percentage points with a 62-86 point bootstrap interval and exact McNemar p = 1.46 x 10^-11. The results establish a clear division of labor: formal verification supplies sound acceptance, while bounded feedback-driven regeneration converts safe rejection into high task completion.
Authors: Ward, Tyler, Zhou, Yan, Gray, Stephanie
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7220230041
