Network Intrusion Detection Using Big Data Analytics: a Pyspark and Hive Approach for Unsw-nb15
Year: 2025
Type: article
Abstract: Network intrusion detection remains a critical challenge as cyber threats continue to evolve in complexity and scale. This study investigates the application of big data analytics for intrusion detection using the UNSW-NB15 dataset. Apache Hive was used for large-scale querying and feature analysis, while PySpark was used for advanced analytics, including descriptive statistics, correlation, hypothesis testing, and dimensionality reduction. A RF classifier was developed and evaluated for both binary and multi-class intrusion detection tasks. The experimental results demonstrate a 99.99% accuracy in binary classification and 98.62% in multi-class classification, highlighting the effectiveness of combining Hive and PySpark for scalable intrusion detection. These findings underscore the importance of big data frameworks in strengthening cybersecurity defence systems
Author: Okon, Daniel U.
Institution: University of Port Harcourt
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7081945831
FWCI:
Cites:
Cited by:
Related to: