Enhancing Regulation-Adherent Requirement Engineering with Contextual AI - An Empirical Study
Year: 2025
Type: dissertation
Abstract: Software projects in the medical device domain demand multi‑layered requirement specifications, yet authoring detailed lower‑level requirements is time‑consuming. This thesis investigates whether privacy‑preserving, on‑premise large language models (LLMs) can automate that decomposition while remaining compliant with data‑protection constraints. Five open‑source instruction tuned models ranging from 3 billion to 70 billion parameters were executed locally via the Ollama runtime, and four prompt strategies: minimal, regulatory‑context, example‑driven, and a retrieval‑augmented generation (RAG) variant, were evaluated across two transformation layers: user-to-system, which converts high-level user requirements into system-level requirements, and system-to-software, which refines system requirements into software requirements. The proposed JSON‑based pipeline batched higher‑level requirements from three commercial medical products, captured model outputs, and logged hallucinations and parsing errors. The quantitative evaluation relied on SBERT semantic-similarity scores and on counting differences between generated and human-written requirements. Qualitative review examined generated requirements from a requirement engineering perspective. Results show that all on‑device models produced syntactically valid, structured requirements when properly prompted. Example‑driven prompts yielded the highest mean similarity scores, while the RAG extension offered no measurable benefit, likely due to a small retrieval corpus. Model size and architecture mattered: R1 Distill Qwen 32B and Llama 3.3 70B consistently outperformed smaller models, though even the largest model often generated out‑of‑scope or insufficiently specific statements. The pipeline also revealed that numeric cues in prompts tightly constrained output length, exposing it as a practical tuning lever for requirement engineers. Although the study is limited by a single‑company dataset, one expert adviser, and GPU‑limited model sizes, it demonstrates the technical feasibility of on‑device LLMs for regulated requirement generation for medical devices and provides a reproducible framework for future work on larger models, richer corpora, and broader expert validation.
Author: Sirin, Aziz Orhan
Institution: Tampere University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7132979037