Work

S3Eval: A Three-Layer Pipeline for Automated Evaluation of PDDL-Encoded Attack Paths

Year: 2026
Type: conference-paper
Abstract: Evaluating the quality of Planning Domain Definition Language (PDDL)-encoded attack paths is essential for ensuring that they faithfully represent real-world vulnerability exploitation. Existing automated tools primarily focus on surface-level properties, such as syntactic correctness and plan solvability, whereas the domain-specific semantics still rely on the costly, unscalable human experts inspection. To address this problem, we propose S3Eval, a three-layer automated evaluation pipeline that progressively assesses the Syntactic, Solvability, and Semantic dimensions of PDDL-encoded attack paths. The semantic layer applies embedding-based similarity and LLM-as-expert judgment guided by domain-specific rubrics for key criteria, including threat model compliance, exploitation mechanism fidelity, and attack-stage completeness. We evaluate S3Eval on both expert-authored references and controlled mutated negative samples. Notably, the best configuration achieves an $F_{1}$ of 0.94 for semantic evaluation. Additionally, the three layers provide complementary quality signals. S3Eval can reliably discriminate between high-and low-quality domains without human expert involvement.
Language: English
Open Access status: hybrid
OpenAlex ID: W7225919039
FWCI:
Cites:
Cited by:
Related to: 10