A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL
Year: 2026
Type: conference-paper
Abstract: Replication Package for EASE 2026 Research Track titled - A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL This dataset contains UML class models that are in the PlantUML (.puml) format and the USE (.use) format. All specifications are available in the specifications.json file. The templates the prompts are available in the prompts.json file. The script to execute the prompts on the various LLMs are available in jupyter notebooks, and python files The directory structure is as follows: Transformed Dataset │ ├── UML │ ├── Airport │ ├── Airport.puml │ ├── Airport.use │ ├── transformed_Airport.puml │ ├── transformed_Airport.use │ ├── EmploymentAgency │ ├── EmploymentAgency.puml │ ├── EmploymentAgency.use │ ├── transformed_EmploymentAgency.puml │ ├── transformed_EmploymentAgency.use │ ├── EURental │ ├── EURental.puml │ ├── EURental.use │ ├── transformed_EURental.puml │ ├── transformed_EURental.use │ ├── HealthRecord │ ├── HealthRecord.puml │ ├── HealthRecord.use │ ├── transformed_HealthRecord.puml │ ├── transformed_HealthRecord.use │ ├── InvoicingOrders │ ├── InvoicingOrders.puml │ ├── InvoicingOrders.use │ ├── transformed_InvoicingOrders.puml │ ├── transformed_InvoicingOrders.use │ ├── ISP │ ├── ISP.puml │ ├── ISP.use │ ├── transformed_ISP.puml │ ├── transformed_ISP.use │ ├── Mortgage │ ├── Mortgage.puml │ ├── Mortgage.use │ ├── transformed_Mortgage.puml │ ├── transformed_Mortgage.use │ ├── Person │ ├── Person.puml │ ├── Person.use │ ├── transformed_Person.puml │ ├── transformed_Person.use │ ├── Royal&Loyal │ ├── Royal&Loyal.puml │ ├── Royal&Loyal.use │ ├── transformed_Royal&Loyal.puml │ ├── transformed_Royal&Loyal.use │ ├── Tournament │ ├── Tournament.puml │ ├── Tournament.use │ ├── transformed_Tournament.puml │ ├── transformed_Tournament.use │ └── Vehicle │ ├── Vehicle.puml │ ├── Vehicle.use │ ├── transformed_Vehicle.puml │ ├── transformed_Vehicle.use │ ├── prompts.json │ ├── script.ipynb │ ├── transformation_script.ipynb | ├── transformed_specifications.json | ├── small_LLM_OCL_script.ipynb | ├── fine_tuning_LLM_for_OCL_script.ipynb | └── specifications.json
Author: Anonymous
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7125680684
FWCI:
Cites:
Cited by:
Related to: