Information Retrieval Meta-Evaluation:Challenges and Opportunities in the Music Domain
J. Urbano
International Society for Music Information Retrieval Conference, pp. 597-602, 2011.
An extended version of this paper can be found at "Evaluation in Music Information Retrieval".
Abstract
The Music Information Retrieval field has acknowledged the need for rigorous scientific evaluations for some time now. Several efforts were set out to develop and provide the necessary infrastructure, technology and methodologies to carry out these evaluations, out of which the annual Music Information Retrieval Evaluation eXchange emerged. The community as a whole has enormously gained from this evaluation forum, but very little attention has been paid to reliability and correctness issues. From the standpoint of the analysis of experimental validity, this paper presents a survey of past meta-evaluation work in the context of Text Information Retrieval, arguing that the music community still needs to address various issues concerning the evaluation of music retrieval systems, pointing out directions for further research and proposals in this line.