Work

Evaluating Reasoning vs. No‑Reasoning Models on Insider‑Threat Datasets

Year: 2025
Type: dissertation
Abstract: Security operations centres (SOCs) are tasked with analyzing enormous volumes of asymmetrical log data that is unevenly distributed. Actual insider threat attacks are extremely hard to detect. This thesis asks one question: do “reasoning-oriented” LLMs (used as RL-post-trained / RL-distilled model families) work better than non-reasoning models for insider-threat detection on the CERT r6.2 dataset? We reformat raw log events into concise 4W session summaries (When, Where, What, Which), train models with careful attention to class imbalance data, and evaluate performance using AUPRC and AUROC. To keep training comparable, we adapt the 7B and 8B parameter models using LoRA with INT4 quantisation, fitting them onto consumer-grade GPUs typical in SOCs. Our findings indicate that the ‘reasoning’ model—DeepSeek R1 Distill Qwen 7B with LoRA—achieves higher AUROC than the ‘no-reasoning’ model – LlaMA 3.1 8B Instruct with LoRA—on a 200k validation slice. However, baseline encoder models were evaluated on the full 1.41M-session validation set in the initial draft, which limits direct cross-model ranking. Therefore, conclusions are framed as protocol-specific until an apples-to-apples re-evaluation is completed on a fully harmonised evaluation set.
Author: Sun, Lihao
Language: English
Open Access status: green
OpenAlex ID: W7211494357
Cites: 25
Cited by:
Related to: 10