The Treatment of Ties in AP Correlation

J. Urbano and M. Marrero
ACM International Conference on the Theory of Information Retrieval, (accepted), 2017.

Abstract

The Kendal tau and AP correlation coefficients are ubiquitous in Information Retrieval and related disciplines for comparing two rankings over the same set of items. Even though Kendall's tau was originally defined assuming that there are no ties in the rankings, two alternative versions were soon developed to account for ties in two different scenarios: measure the accuracy of an observer with respect to a true and objective ranking, and measure the agreement between two observers in the absence of a true ranking. These two variants prove useful in cases where ties are possible in either ranking, and may indeed result in very different scores. AP correlation was devised to incorporate a top-heaviness component into Kendall's tau, penalizing more heavily if the discrepancies occur between items at the top of the rankings, making it a very compelling coefficient for Information Retrieval problems. Unfortunately, the treatment of ties in AP correlation remains an open problem. In this paper we fill this gap, providing closed analytical formulations of AP correlation under the two scenarios of ties. In addition, we developed an R package that implements these coefficients.

Downloads