Year: 2026
Type: other
Abstract: Small, on-device language models are now used in products where privacy and latency matter, and the safety stack varies by deployment. This thesis measures jailbreak robustness of two compact, locally run models—Qwen3-4B and Llama 3.2-3B—in single-turn, text-only settings. The evaluation is implemented in lfuzz, a local framework that runs the models through Ollama, wraps each harmful goal in a manual attack template, and applies prompt-only defenses. The experiments use a frozen 100-goal snapshot from JailbreakBench Behaviors and two seeds (42 and 43), yielding 200 evaluations per model and defense. Each goal is evaluated under two conditions: no defense and a “safety sandwich” that combines a real system policy with a user-message wrapper that marks untrusted text. Responses are labeled by a hybrid evaluator that combines refusal string matching with a local LLM judge; uncertain cases are flagged and not counted as jailbreaks. The baseline Attack Success Rate (ASR) is 25.0% for Qwen3-4B and 38.0% for Llama 3.2-3B. With the safety sandwich, ASR drops to 0.5% for Qwen3-4B and 23.5% for Llama 3.2-3B, with Wilson 95% confidence intervals reported for each estimate. A matched-token sensitivity run (4096 tokens for both models) preserves these conclusions and removes truncation for Llama. A manual audit of 400 seed-42 outputs shows the automated judge is high-precision but moderate-recall (precision 0.93, recall 0.74). The thesis argues for transparent reporting of uncertainty and judge reliability rather than a single safety score, and it provides a reproducible workflow for on-device evaluation. The code repository is private because it contains harmful prompt material, but the method and artifacts are documented in the thesis for verification.
Source: Doria (University of Helsinki)
Author: Lagerqvist, Rasmus
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7160677126