Work

Advanced DNS tunneling detection: a hybrid reinforcement learning and metaheuristic approach

Year: 2026

Type: article

Abstract: Introduction DNS tunneling remains a critical network threat, exploiting the inherent trust in the DNS protocol for unauthorized communication, data exfiltration, and firewall evasion. Methods Addressing this challenge, this paper introduces a novel, hybrid feature selection framework that integrates the Random Forest classifier with an Enhanced Reinforcement Learning-Guided Grey Wolf Optimizer (EnhancedRLGWO). The EnhancedRLGWO employs a Dueling Deep Q-Network and strategic Opposition-Based Learning to intelligently navigate the feature space and identify an optimal, minimal subset. Results Evaluated against the benchmark CIRA-CIC-DoHBrw-2020 dataset, the proposed approach achieved a state-of-the-art accuracy of 99.82% and a weighted F1-score of 99.79% using a highly compact subset of only 12 features. This performance significantly outperforms existing machine learning-based DNS tunneling detection systems, such as a hybrid feature selection model achieving 98.3% accuracy and a full 28-feature Random Forest baseline (98.50% accuracy). The experimental results showed the robustness of this method in identifying various types of DNS tunneling attacks, including Iodine, DNS2TCP, and DNScat2, while maintaining performance and accuracy.

Source: Frontiers in Computer Science

DOI: 10.3389/fcomp.2025.1728980

Authors: Mahmoud Sammour, Mohd Fairuz Iskandar Othman, Aslinda Hassan, Omar Bhais, Mohammed Saad Talib

Institutions: Universiti Teknikal Malaysia Melaka, University of Babylon

Language: English

Topic: Network Security and Intrusion Detection

Subfield: Computer Networks and Communications

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,418

Funders: Universiti Teknikal Malaysia Melaka, Ajman University