Work

Data Warehouse Query Log Analysis using MapReduce

Year: 2011
Type: dissertation
Abstract: Current auditing techniques on the data warehouse of Credit Suisse, a large Swiss multinational nancial service provider, are either not exible enough to answer ad-hoc queries or very costly to maintain.This master thesis proposes a new scalable solution in analyzing the raw query logs based on distributed computing using Hadoop and Pig.Our approach acts on raw data, hence allows total exibility in formulating ad-hoc queries.In addition, it does not rely on maintaining a dedicated data warehouse, but can run on any cluster infrastructure on a utility basis, thus being very cost eective.Our solution has been implemented and tested on synthetic test data, as well as on a set of production data at Credit Suisse.An estimation has shown that we are able to process 93% of all the query log data, even in the presence of malformed data.In experiments, our implementation has proven to be highly scalable when processing synthetic query logs for a generic use case.Its performance however is in need of improvement, but might be compensated by the high scalability in the meantime.
Author: Lai, Willy
Language: English
Open Access status: green
OpenAlex ID: W2528018424
Cites:
Cited by:
Related to: 10