Work

A Simple Method to Determine if a Music Information Retrieval System is a “Horse”

Year: 2014
Type: article
Abstract: We propose and demonstrate a simple method to explain the figure of merit (FoM) of a music information retrieval (MIR) system evaluated in a dataset, specifically, whether the FoM comes from the system using characteristics confounded with the “ground truth” of the dataset. Akin to the controlled experiments designed to test the supposed mathematical ability of the famous horse “Clever Hans,” we perform two experiments to show how three state-of-the-art MIR systems produce excellent FoM in spite of not using musical knowledge. This provides avenues for improving MIR systems, as well as their evaluation. We make available a reproducible research package so that others can apply the same method to evaluating other MIR systems.
Author: Bob L. Sturm
Institution: Aalborg University
Language: English
Open Access status: green
OpenAlex ID: W2074619556
FWCI: 10.9
Cites: 50
Cited by: 219
Related to: 10