How do Gain and Discount Functions Affect the Correlation between DCG and User Satisfaction?
J. Urbano and M. Marrero
European Conference on Information Retrieval, pp. 197-202, 2015.
Abstract
We present an empirical analysis of the effect that the gain and discount functions have in the correlation between DCG and user satisfaction. Through a large user study we estimate the relationship between satisfaction and the effectiveness computed with a test collection. In particular, we estimate the probabilities that users find a system satisfactory given a DCG score, and that they agree with a difference in DCG as to which of two systems is more satisfactory. We study this relationship for 36 combinations of gain and discount, and find that a linear gain and a constant discount are best correlated with user satisfaction.