Ziv Bar-Yossef (Technion - Israel Institute of Technology and Google Haifa)
Maxim Gurevich (Technion - Israel Institute of Technology)
Abstract:
We address the problem of measuring relevance neutral search quality metrics, like corpus size, index freshness, and density of duplicates in the index. The recently proposed estimators for such metrics [Bar-Yossef and Gurevich, WWW2006][Broder et al, CIKM 2006] suffer from significant bias and/or poor performance, due to inaccurate approximation of the so called ``document degrees''.
We present two new estimators that are able to overcome the bias introduced by approximate degrees. Our estimators are based on a careful implementation of an approximate importance sampling procedure. Comprehensive theoretical and empirical analysis of the estimators demonstrates that they have essentially no bias even in situations where document degrees are poorly approximated.
Building on an idea from [Broder et al, CIKM 2006], we discuss Rao-Blackwellization as a generic method for reducing variance in search engine estimators. We show that Rao-Blackwellizing our estimators results in significant performance improvements, while not compromising quality.
Slot:
Alberta, Wednesday, May 9, 2007, 10:30am to 12 noon.