MIUN BiasPatrol at SemEval-2026 Task 13: Why TF-IDF can Beat Transformers for OOD Code Detection
Year: 2026
Type: conference-paper
Abstract: The increasing use of AI-generated code underscores the need for effective detection systems.However, their performance often deteriorates when faced with distribution shifts.This paper presents our system for SemEval-2026 Task 13:A, which focuses on binary classification of human-written versus machinegenerated code across various programming languages and domains.We systematically compare traditional classifiers, such as Random Forest and XGBoost, which utilize statistical and TF-IDF features, against pipelines that leverage frozen embeddings from advanced code transformers like UniXcoder and Graph-CodeBERT.Our results reveal a notable tradeoff, i.e., while transformer-based pipelines excel in in-distribution validation (reaching up to 0.89 Macro F1), they experience severe performance drops up to 77%; when applied to out-ofdistribution languages and domains.In contrast, models employing TF-IDF with tree-based classifiers demonstrate significantly greater stability.We identify this fragility as a result of a bias toward superficial formatting, particularly whitespace, which is accentuated by transformers.By implementing simple space normalization, we enhance the out-of-distribution robustness of traditional models; however, this also highlights the ongoing dependence of embeddings on these non-semantic features.Our findings suggest that for creating generalizable code detection systems, straightforward, wellnormalized lexical features may be more reliable than complex, unrefined embeddings.
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166840175