Work

Phishing in the LLM Era: Which Website Features Still Separate Phishing from Legitimate When Attackers Clean Up Their Act?

Year: 2026
Type: preprint
Abstract: Classical phishing detectors were trained against sloppy attackers: IP-literal hosts, hyphen-stuffed domains, URL shorteners, absurd lengths. Large language models and cheap tooling have industrialised the opposite kind of phishing — grammatically clean, lexically tidy, HTTPS-served. This paper asks which parts of a classical feature set still earn their keep when the lexical sloppiness is removed. On the UCI Phishing Websites corpus (11,055 sites, 30 ternary features), we partition the features into three families — lexical/URL-structure (LEX, 14), host and reputation (HOST, 9), and HTML/JavaScript content behaviour (CONTENT, 7) — and run a graded ablation over three tabular models (logistic regression, random forest, gradient boosting) with operating thresholds calibrated on a separate validation split at 1% and 5% false-positive rates. Three findings survive every model. First, the CONTENT family is nearly worthless on this corpus: deleting it costs at most 0.003 AUC, and alone it reaches only 0.59–0.61 AUC, barely above chance. Second, LEX and HOST are both load-bearing and partially redundant: each alone reaches 0.92–0.97 AUC, and deleting either costs 0.017–0.052. Third, the clean-attacker test splits by how it is run. Among real test samples, the 127 phishing pages with the fewest lexical flaws are still caught at 96.1% recall (random forest, 5% FPR threshold) — their host features betray them. But under a counterfactual edit that sets the lexical features of every phishing page to legitimate-like values, detection at the strict 1% FPR threshold falls to zero for every model, and at the 5% threshold only the random forest degrades gracefully (recall 0.984 → 0.852 for a 9-feature clean-up, → 0.420 for a full lexical clean-up) while logistic regression and gradient boosting collapse (0.19 and 0.10). We also report a negative result about the dataset itself: it contains not one phishing sample that is clean on all nine strict lexical features, so the truly clean attacker is, in 2015 data, a counterfactual — and conclusions about the LLM era drawn from this corpus must be read with that limit in view.
Language: English
Open Access status: green
OpenAlex ID: W7226032378
Cites:
Cited by:
Related to: 10