Work

Evaluating Artificial Intelligence Tools for Forensic-Linguistic Analysis of Arabic Judicial Texts: A Layer-Based Comparative Case Study of Saudi Judgments

Year: 2026
Type: preprint
Abstract: This study evaluates GPT-4o (Omni; version 2024-05-13), supported by Arabic preprocessing with CAMeL Tools, against expert forensic-linguistic annotation of Saudi judicial judgments. Using a comparative evaluation design, the study analyzed 10 Saudi criminal judgments from Majmūʿat al-Aḥkām al-Qaḍāʾiyya (Ministry of Justice, 1435 AH/2013–2014 CE), from which 24 representative segments involving psychiatric evidence, legal capacity, and criminal responsibility were extracted. Each segment was coded across the five P-DIM layers, producing 120 segmentby-layer classification decisions. Two expert coders established the adjudicated benchmark after pre-adjudication reliability testing (88% agreement; pooled Cohen's kappa = .82). The AI arm used deterministic prompting (temperature = 0.0), a system-level forensic-linguistic persona, the full P- DIM codebook, and three few-shot gold-standard examples per layer. Results showed strong AI performance for formulaic and structurally explicit layers, especially legal formulae and institutional weighting, but weak performance for pragmatic-doctrinal interpretation. Layer 4 produced the main pragmatic bottleneck (F1 = .41; kappa = .28). The corrected fold-level ANOVA used 20 F1 observations (four validation folds for each of the five P-DIM layers) and showed a significant layer effect, F(4, 15) = 63.04, p = 3.367e-09, η² =.944. Tukey HSD confirmed that L4 was significantly lower than all other layers, while also showing additional significant non-L4 contrasts. Overall AI-expert agreement was fair (kappa = .275, N = 120), reflecting aggregate instability caused mainly by Layer 4. The study therefore supports a human-in-the-loop verification protocol rather than autonomous AI interpretation in Arabic forensic-linguistic analysis. Keywords: Arabic forensic linguistics; Saudi judicial judgments; AI-assisted annotation; expert coding; legal NLP evaluation; pragmatic-doctrinal inference; diglossia
Language: English
Subfield: Law
Open Access status: green
OpenAlex ID: W7225987817
Cites: 30
Cited by:
Related to: 10