Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects
Year: 2026
Type: preprint
Abstract: This preprint evaluates 67 systems on 43 Arabic-focused tasks across nine task families using a shared zero-shot protocol. It compares Jev with other decision models and large language models, and examines content-matched differences between Modern Standard Arabic and Arabic dialects within Jev 1.14.0. Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B. Relative performance varies across task families. The analyses are descriptive; aggregate scores should be interpreted alongside task coverage, task-family results and matched language comparisons. The record includes the manuscript and a companion archive containing LaTeX source, complete primary scores and coverage, sensitivity analyses, matched correctness summaries, system and dataset identifiers, templates, recorded per-request processing rates and offline validation scripts. Timing definitions distinguish request wall time from amortized LLM batch processing. Underlying dataset texts and full model responses are not included. Historical hypothesis outputs are retained with their validation limitations explicitly documented.
Language: arb
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225251997