Research on the Framework of Network Security Threat Intelligence Crawler
Year: 2025
Type: conference-paper
Abstract: In light of the global proliferation of advanced persistent threats (APTs) and ransomware attacks, cyber threat intelligence (CTI) has emerged as a pivotal component of proactive defence strategies. However, existing cyber crawling frameworks face three challenges: high technical costs, performance bottlenecks and scalability issues, and ecological sustainability risks. This paper therefore proposes an artificial intelligence-driven, distributed web crawling framework based on cybersecurity threat intelligence. This framework overcomes the limitations of existing open-source tools with regard to dynamic rendering, high technical costs, low scalability, anti-crawling measures and intelligent analysis. Experiments demonstrate that the proposed framework enhances crawling efficiency by 70% and reduces resource utilisation by 50% compared to the Scrapy approach. This research provides a scalable paradigm for building an autonomous, controllable, cybersecurity-intelligent infrastructure, offering a new direction for cyber-crawling research. Its applications can be extended to key areas such as financial risk control and public opinion monitoring.
Authors: Wenxuan Gan, Bin Liu
Institution: Panzhihua University
Language: English
Topic: Web Data Mining and Analysis
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4413740980