Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015)
0.04
0.044248845 = product of:
0.08849769 = sum of:
0.08849769 = sum of:
0.038503684 = weight(_text_:systems in 2591) [ClassicSimilarity], result of:
0.038503684 = score(doc=2591,freq=4.0), product of:
0.16037072 = queryWeight, product of:
3.0731742 = idf(docFreq=5561, maxDocs=44218)
0.052184064 = queryNorm
0.24009174 = fieldWeight in 2591, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
3.0731742 = idf(docFreq=5561, maxDocs=44218)
0.0390625 = fieldNorm(doc=2591)
0.049994007 = weight(_text_:22 in 2591) [ClassicSimilarity], result of:
0.049994007 = score(doc=2591,freq=4.0), product of:
0.1827397 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.052184064 = queryNorm
0.27358043 = fieldWeight in 2591, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=2591)
0.5 = coord(1/2)
- Abstract
- Purpose In a system-based approach, replicating the web would require large test collections, and judging the relevancy of all documents per topic in creating relevance judgment through human assessors is infeasible. Due to the large amount of documents that requires judgment, there are possible errors introduced by human assessors because of disagreements. The paper aims to discuss these issues. Design/methodology/approach This study explores exponential variation and document ranking methods that generate a reliable set of relevance judgments (pseudo relevance judgments) to reduce human efforts. These methods overcome problems with large amounts of documents for judgment while avoiding human disagreement errors during the judgment process. This study utilizes two key factors: number of occurrences of each document per topic from all the system runs; and document rankings to generate the alternate methods. Findings The effectiveness of the proposed method is evaluated using the correlation coefficient of ranked systems using mean average precision scores between the original Text REtrieval Conference (TREC) relevance judgments and pseudo relevance judgments. The results suggest that the proposed document ranking method with a pool depth of 100 could be a reliable alternative to reduce human effort and disagreement errors involved in generating TREC-like relevance judgments. Originality/value Simple methods proposed in this study show improvement in the correlation coefficient in generating alternate relevance judgment without human assessors while contributing to information retrieval evaluation.
- Date
- 20. 1.2015 18:30:22
18. 9.2018 18:22:56
