Work

Cross-Attention or Concatenation? A Multi-Seed Empirical Study of Code-Comment Fusion for Pull Request Outcome Prediction

Year: 2026
Type: preprint
Abstract: Predicting whether a pull request (PR) will be merged, rejected, or left unre solved is a recurring problem in empirical software engineering, yet it remains unclear how much explicit fusion of code changes and reviewer comments adds over simpler designs, and conclusions drawn from a single training run can be misleading. We report a multi-seed comparison of encoder-based fusion strategies on a retrospective dataset of 33,955 PRs from 155 projects, three platforms, and 12 programming languages. A CodeBERT–RoBERTa hybrid with bidirectional cross-modal attention is compared with concatenation and late-fusion variants, four alternative code encoders (GraphCodeBERT, UniXcoder, CodeT5, CodeReviewer), unimodal baselines, and two re-implemented prior systems, using five random seeds per model. The cross-attention hybrid reaches 88.11% mean accuracy (sample SD 0.39 percentage points, pp) and outperforms its code-only counterpart in every seed (+1.85 pp; Holm-adjusted paired p = 0.008). How-ever, the seven best models lie within a 0.42 pp band, and cross-attention exceeds concatenation by only 0.34 pp on average across five encoders (positive for four of five, not significant for any single encoder); at one seed (42) the ordering of the two reversed. Review comments alone are about as informative as code alone, and combining them adds roughly 1.6–1.9 pp. Project identity is strongly associated with the outcome (Cram´er’s V = 0.60), so the random split used here probably overstates generalisation. All analyses are retrospective, use features that become available only after review, and exclude LLM-based baselines.
Author: Ahmed Hussen
Language: English
Open Access status: green
OpenAlex ID: W7236241355
Cites: 21
Cited by:
Related to: 10