Work

Distributed indexing of web scale datasets for the cloud

Year: 2010

Type: conference-paper

Abstract: In this paper, we present a distributed architecture for indexing and serving large and diverse datasets. It incorporates and extends the functionality of Hadoop, the open source MapReduce framework, and of HBase, a distributed, sparse, NoSQL database, to create a fully parallel indexing system. Experiments with structured, semi-structured and unstructured data of various sizes demonstrate the flexibility, speed and robustness of our implementation and contrast it with similarly oriented projects. Our 11 node cluster prototype managed to keep full-text indexing time of 150GB raw content in less than 3 hours, whereas the system's response time under sustained query load of more than 1000 queries/sec was kept in the order of milliseconds.

DOI: 10.1145/1779599.1779600

Authors: Ioannis Konstantinou, Evangelos Angelou, Dimitrios Tsoumakos, Nectarios Koziris

Institution: National Technical University of Athens

Language: English

Topic: Cloud Computing and Resource Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Sixth Framework Programme

Award: G8881705538

OpenAlex ID: W2038879656

FWCI: 3.17

Cites: 12

Cited by: 22

Related to: 10