Year: 2026
Type: other
Abstract: This thesis evaluates how well modern large language models generate Python docstrings from function code alone, and how prompt design and evaluation method affect the conclusions. Two commercial models, Anthropic Claude Haiku 4.5 and OpenAI GPT-5.4-mini, were used to generate docstrings for 20,000 functions drawn from CodeSearchNet and CodeXGLUE. The main experiment compares two prompt conditions across both datasets: code only (C0) and code with extracted structural hints (C1). A matched follow-up experiment on 1,000 CodeSearchNet functions per model compares two improved prompt-engineering conditions: C2, which adds stronger instructions and a hard 100-word cap, and C3, which removes the hard cap while keeping the improved prompt structure. In total, 84,000 docstrings were generated and evaluated through four tiers: structural factuality checks, referencebased similarity metrics, semantic similarity metrics, and direct quality assessment by two LLM judges and human reviewers. The results show that both models can produce broadly useful documentation, but that the conclusions depend strongly on the evaluation method. Adding structural hints in C1 improves parameter-related factuality and usually improves judged quality, while reference-based metrics change very little. The C2/C3 follow-up shows that the hard brevity constraint in C2 improves conciseness but reduces completeness, and that removing this constraint in C3 improves judged completeness and usefulness without uniformly improving automated metrics. A central finding is that different metrics often disagree: BERTScore and CodeBERTScore do not always favor the same prompt condition, and judge-based assessment is more sensitive than reference-based metrics to differences in completeness and usefulness. Generated docstrings also sometimes surpass the original references in clarity and detail, which highlights a limitation of reference-based evaluation for this task. The thesis contributes a multi-tier evaluation framework for Python docstring generation and shows that model ranking, prompt ranking, and perceived documentation quality depend on how evaluation is defined, a result with direct implications for how future studies in this area should be designed.
Source: Doria (University of Helsinki)
Author: Röblom, Odin
Institution: Åbo Akademi University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162909923