Shaw, W.M.; Burgin, R.; Howell, P.: Performance standards and evaluations in IR test collections : vector-space and other retrieval models (1997)
0.01
0.008167865 = product of:
0.01633573 = sum of:
0.01633573 = product of:
0.03267146 = sum of:
0.03267146 = weight(_text_:systems in 7259) [ClassicSimilarity], result of:
0.03267146 = score(doc=7259,freq=2.0), product of:
0.16037072 = queryWeight, product of:
3.0731742 = idf(docFreq=5561, maxDocs=44218)
0.052184064 = queryNorm
0.2037246 = fieldWeight in 7259, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.0731742 = idf(docFreq=5561, maxDocs=44218)
0.046875 = fieldNorm(doc=7259)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- Computes low performance standards for each query and for the group of queries in 13 traditional and 4 TREC test collections. Predicted by the hypergeometric distribution, the standards represent the highest level of retrieval effectiveness attributable to chance. Compares operational levels of performance for vector-space, ad-hoc-feature-based, probabilistic, and other retrieval models to the standards. The effectiveness of these techniques in small, traditional test collections, can be explained by retrieving a few more relevant documents for most queries than expected by chance. The effectiveness of retrieval techniques in the larger TREC test collections can only be explained by retrieving many more relevant documents for most queries than expected by chance. The discrepancy between deviations form chance in traditional and TREC test collections is due to a decrease in performance standards for large test collections, not to an increase in operational performance. The next generation of information retrieval systems would be enhanced by abandoning uninformative performance summaries and focusing on effectiveness and improvements in effectiveness of individual queries
