Work

Mathematical Model and Algorithm for Accurate Main Content Extraction From News Websites

Year: 2024

Type: article

Abstract: Irrelevant elements like ads, menus, and footers in web pages hinder data extraction and reduce the performance of Retrieval-Augmented Generation (RAG) systems in Large Language Models (LLMs). This paper tackles the challenge of accurately identifying and extracting the main content from web pages to enhance the efficiency of these systems. We present a novel mathematical model and algorithm that leverages the Document Object Model (DOM) structure, effectively isolating relevant content with high accuracy. Our approach is language-neutral and performs well across diverse languages, including those with complex tokenization, such as Arabic. To validate the model, we created a dataset from 500 websites, allowing for comprehensive evaluation and benchmarking. The algorithm’s practical application demonstrates a reduction in token usage for LLM tasks, contributing to cost-effectiveness. This work introduces a robust, open-source tool for the academic and commercial communities, fostering further innovation in web content extraction and information retrieval.

Source: IEEE Access

DOI: 10.1109/access.2024.3524656

Authors: Hamza Salem, Hadi Salloum, Manuel Mazzara

Institution: Innopolis University

Language: English

Topic: Web Data Mining and Analysis

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,995

Funder: Analytical Center for the Government of the Russian Federation