Benchmarks for measuring duplicate detection methods for biological databases
Year: 2016
Type: other
Abstract: Duplication is a key data quality problem in many domains. It is related to redundancy (near-identical instances), incompleteness (fragmentary records), and inconsistency (records with contradictory information) – issues that undermine the efficiency of data retrieval and the accuracy of answers to queries. It is problematic in bioinformatics databases, where data volumes are rapidly increasing. The high data volume makes fully manual approaches, recognized as the most precise way to determine whether a pair is duplicate, infeasible. Automatic approaches may be feasible, but methods to date have examined only limited types of duplicates under simple assumptions. A further fundamental issue is that the definition of ‘duplicate’ is context dependent. A pair considered as duplicates by one community, or for one task, may not necessarily be so in another context; a duplicate detection method that achieves high performance in a dataset gathered under restrictive assumptions may not necessarily be effective on another dataset. We have collected records that can be regarded as duplicates under a range of assumptions, and created related benchmarks. We built three DNA sequence database benchmarks, based on information drawn from a range of resources, including information derived by mapping between databases. Each benchmark has distinct characteristics. We quantitatively measure these characteristics and argue for their complementary value in evaluation of duplication detection techniques. The benchmarks collectively contain a vast number of validated biological duplicates; the largest has nearly half a billion duplicate pairs. They are also the first benchmarks targeting the primary nucleotide databases. The records cover the 21 most studied organisms in molecular biology research. Our quantitative analysis shows that duplicates in the different benchmarks, and in different organisms, have different characteristics. It is thus unreliable to evaluate duplicate detection methods against any single benchmark. For example, the benchmark derived from Swiss-Prot mappings identifies more diverse types of duplicates, showing the importance of expert curation, but is limited to coding sequences. Overall, these benchmarks form a resource that we believe will be of tremendous value for development and evaluation of duplicate detection methods. They also represent the diversity and complexity of duplication in biological databases.
Authors: Qingyu Chen, Justin Zobel, Karin Verspoor
Institution: The University of Melbourne
Language: English
Topic: Data Quality and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
OpenAlex ID: W2735891904