HoneypotBench: A Curated 5-Class MITRE ATT&CK Benchmark for Evaluating Frontier Large Language Models on Honeypot Alert Classification
Year: 2026
Type: dataset
Abstract: # HoneypotBench — Complete Data and Code Release (v1.0) **Companion artefact to:** *HoneypotBench: A Curated 5-Class MITRE ATT&CK Benchmark for Evaluating Frontier Large Language Models on Cloud-Honeypot Threat Classification.* **Authors:** Sridhar G, J B Simha, Rashmi Agarwal (REVA University, Bengaluru) **Licence:** Data files (`.csv`, `.json`, `.jsonl`) under **CC-BY-4.0**. Code (`.py`) under **MIT**. Third-party rule snapshot (`emerging_rules_snapshot.tar.gz`) redistributed under its original BSD-style Emerging Threats Open license. **Cite as:** [DOI to be assigned on Zenodo publication] --- ## What this release contains A complete, reproducible artefact bundle for the HoneypotBench benchmark and every analysis reported in the manuscript. A reader can (a) inspect the benchmark itself, (b) reproduce every reported metric from the raw per-alert predictions, (c) re-run the classical baselines on their own machine in about 15 minutes, (d) extend the LLM evaluation to future frontier models, and (e) audit the multi-annotator hallucination adjudication. **All primary artefacts are included.** No files require external download except the ETOpen ruleset, which is bundled in `rules/` as the exact snapshot used to produce the reported Suricata baseline. --- ## Directory layout ```HoneypotBench_v1/├── README.md ← this file├── data/│ └── dataset.csv ← 1,947-alert HoneypotBench (primary artefact)├── predictions/│ ├── llm_predictions.jsonl ← 7,641 raw LLM predictions│ ├── sbert_baseline_predictions.jsonl ← 1,947 SBERT per-alert predictions│ └── suricata_predictions.jsonl ← 1,947 Suricata per-alert predictions├── baselines/│ ├── llm_results.json ← LLM aggregate results (all metrics)│ ├── sbert_baseline_metrics.json ← SBERT+XGBoost + SBERT+RF results│ └── suricata_baseline_metrics.json ← Suricata rule-content matcher results├── adjudication/│ ├── hallucination_adjudication_stratified_sample.csv ← 211-alert base sample│ ├── annotator_1_verdicts.csv ← Annotator 1 verdicts│ ├── annotator_2_verdicts_excluded.csv ← Annotator 2 (excluded, see §6.9)│ ├── annotator_3_verdicts.csv ← Annotator 3 verdicts│ ├── hallucination_adjudication_consensus_A1_A3.csv ← Merged A1+A3 with strict consensus│ ├── expert_adjudication_set_657rows.csv ← Section 7 expert-adjudication export│ └── Annotator_Brief.md ← Brief distributed to annotators├── code/│ ├── run_sbert_baselines.py ← Reproduces sbert_baseline_metrics.json│ ├── run_suricata_baseline.py ← Reproduces suricata_baseline_metrics.json│ └── hallucination_adjudication_export.py ← Regenerates the stratified sample├── rules/│ └── emerging_rules_snapshot.tar.gz ← Exact ETOpen ruleset used for reported Suricata└── figures/ ├── fig_calibration.png ├── fig_confusion.png ├── fig_cost_accuracy.png └── fig_perclass_f1.png``` --- ## File-by-file description ### `data/dataset.csv` (573 KB) The 1,947-alert HoneypotBench corpus. **This is the primary artefact.** The schema is described in Table 1 of the manuscript. Every alert carries the following columns: | Column | Type | Description ||---|---|---|| `alert_id` | string | Unique identifier: `TRI-0001` through `TRI-1947` || `synthetic` | bool | Always `True` — flags the record as benchmark-generated || `category` | enum | Ground-truth class: `brute_force`, `c2_malware`, `web_exploitation`, `reconnaissance`, `benign_noise` || `first_seen`, `last_seen` | ISO 8601 | Time span (within 18 Oct – 7 Nov 2025) || `event_count`, `distinct_paths` | int | Session-level activity summary || `target_service` | string | Application-layer service || `target_honeypot` | string | Deployed honeypot component || `source_ip` | string | RFC 5737 documentation IPv4 address (safe to redistribute) || `severity` | enum | `CRITICAL` / `HIGH` / `MEDIUM` / `INFO` || `confidence` | float | Template-assigned confidence in [0, 1] || `mitre_techniques` | string | Semicolon-separated MITRE ATT&CK IDs || `evidence_sample` | string | Realistic service log excerpt || `template_id` | string | Identifier of the generation template || `holdout_70_30` | enum | Pre-computed 70/30 stratified split: `train` / `test` || `cv_fold_5` | int | Pre-computed 5-fold stratified assignment (0–4) || `grouped_fold_5` | int | Alternative 5-fold grouped by `source_ip_hash` | **Class balance:** 390 brute_force, 390 c2_malware, 389 web_exploitation, 389 reconnaissance, 389 benign_noise. **Service distribution:** webtrap 860, snare 443, mysql 295, jupyter 123, cowrie 107, smtp 50, k8s 41, redis 20, conpot 8. **Nine honeypot services in total.** ### `predictions/llm_predictions.jsonl` (4.4 MB, 7,641 records) Raw per-alert LLM predictions across all four experimental conditions for all three frontier models. Breakdown: - **Baseline evaluation:** 5,841 records (3 models × 1,947 alerts) — primary evaluation, main-text tables- **Few-shot ablation:** 1,500 records (3 × 500-alert stratified subset) — Section 6.4- **Naive adversarial suffix:** 150 records (3 × 50) — Section 6.5- **Sophisticated adversarial suffix:** 150 records (3 × 50) — Section 6.5 Each line is a JSON object with fields: `alert_id`, `model_key` (`claude` / `gpt` / `gemini`), `tag` (`baseline` / `fewshot` / `adv_naive` / `adv_sophisticated`), `parsed` (the model's parsed JSON response with `classification`, `attack_category`, `mitre_attack_technique_id`, `severity`, `confidence`), `raw_response`, and cost fields where the API exposes them. Every aggregate metric in `llm_results.json` derives from this file. ### `predictions/sbert_baseline_predictions.jsonl` (389 KB, 1,947 records) Per-alert predictions from SBERT+XGBoost and SBERT+Random Forest, produced by 5-fold stratified cross-validation. Each line has: `alert_id`, `ground_truth`, `fold`, `sbert_xgb_pred`, `sbert_xgb_confidence`, `sbert_rf_pred`, `sbert_rf_confidence`. Aggregate metrics in `sbert_baseline_metrics.json`. ### `predictions/suricata_predictions.jsonl` (312 KB, 1,947 records) Per-alert predictions from the Suricata rule-content matcher against the December 2025 ETOpen ruleset (52,160 rules parsed, 37,352 mapped to HoneypotBench categories). Each line: `alert_id`, `ground_truth`, `suricata_pred`, `matched_rule_msg`. Note the degenerate behaviour discussed in the manuscript §9 — broad C2 rules cause the classifier to collapse all predictions to `c2_malware`, yielding 0.20 accuracy. This measures the rule-selection protocol, not Suricata's operational capability, and is included for reproducibility. ### `baselines/llm_results.json` (65 KB) Aggregate LLM results across all four tags for the three frontier LLMs. Contains per-model: accuracy with 95% bootstrap CIs, macro-F1 with CIs, per-class metrics, confusion matrices, calibration bins with ECE, per-service breakdown, ensemble analysis, prompt-sensitivity deltas, adversarial-robustness deltas, cost breakdown per tag, and reasoning-token counts. **Authoritative source for Tables 4–14 in the manuscript.** Baseline aggregate (for quick verification): | Model | Accuracy | Macro-F1 | 95% CI (accuracy) ||---|---|---|---|| GPT-5.6-sol | 0.5727 | 0.5161 | [0.5501, 0.5948] || Gemini 3.1 Pro Preview | 0.5280 | 0.4633 | [0.5054, 0.5501] || Claude Fable 5.1 | 0.4643 | 0.4335 | [0.4412, 0.4864] | ### `baselines/sbert_baseline_metrics.json` (4 KB) Classical ML baseline results — the surprising 0.957 accuracy for SBERT+XGBoost that the manuscript §6.7 discusses. Contains per-class precision/recall/F1, confusion matrices, bootstrap 95% CIs, and McNemar tests against GPT-5.6-sol (b=36, c=785, χ²=681.5, p<10⁻¹⁵⁰). ### `baselines/suricata_baseline_metrics.json` (2 KB) Suricata baseline metrics. Reproduces the 0.20 accuracy figure discussed in §9 Limitations. ### `adjudication/hallucination_adjudication_stratified_sample.csv` (60 KB, 211 rows) The base stratified sample used for the hallucination adjudication (§6.9). Alerts drawn from four buckets: confident-and-wrong (30% target), confident-and-right (30%), low-confidence (20%), unusual-MITRE (20%). Type-1 (structural MITRE format) is pre-flagged. ### `adjudication/annotator_{1,2,3}_verdicts.csv` (61–66 KB each, 211 rows each) Verdicts from three security-domain annotators using the five-type rubric in `Annotator_Brief.md`. **Annotator 2's file is included with the `_excluded` suffix per the openness principle** — the manuscript §6.9 discusses why Annotator 2's verdicts were excluded from the headline analysis (systematic labelling artefact: all six flagged rows were type_2, all in Claude, no notes provided; κ against Annotator 3 was essentially zero at −0.019). Readers can inspect the raw verdicts and reproduce or challenge our exclusion decision. ### `adjudication/hallucination_adjudication_consensus_A1_A3.csv` (73 KB) Merged file with Annotators 1 and 3 verdicts side-by-side, plus a computed `consensus_verdict` column under the strict rule (both annotators agree on both flag and type). Supports Table 16 and the Cohen's κ = 0.08 (5-way) and 0.18 (binary) headline numbers. ### `adjudication/expert_adjudication_set_657rows.csv` (261 KB) The 657-alert stratified export used for the expert adjudication in §7 (which found 486 unanimous LLM-vs-benchmark disagreements, headline finding of that section). Contains per alert: ground-truth label, all three models' predictions, and `expert_verdict` / `notes` columns for external reviewers. ### `adjudication/Annotator_Brief.md` (7 KB) Exact brief distributed to the annotators. Documents the 5-type hallucination taxonomy (Type 1: fabricated MITRE ID; Type 2: phantom entity attribution; Type 3: unsupported severity inflation; Type 4: wrong kill-chain phase; Type 5: temporal/contextual confabulation), the review workflow, and the confidentiality terms. ### `code/` (three Python scripts) - **`run_sbert_baselines.py`** — SBERT+XGBoost and SBERT+RandomForest baselines, 5-fold stratified CV, bootst
Author: G, Sridhar
Institution: REVA University
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7213758201