Rajagopal, P.; Ravana, S.D.; Koh, Y.S.; Balakrishnan, V.: Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment (2019)
0.01
0.008572079 = product of:
0.025716238 = sum of:
0.010929906 = weight(_text_:in in 5287) [ClassicSimilarity], result of:
0.010929906 = score(doc=5287,freq=12.0), product of:
0.059380736 = queryWeight, product of:
1.3602545 = idf(docFreq=30841, maxDocs=44218)
0.043654136 = queryNorm
0.18406484 = fieldWeight in 5287, product of:
3.4641016 = tf(freq=12.0), with freq of:
12.0 = termFreq=12.0
1.3602545 = idf(docFreq=30841, maxDocs=44218)
0.0390625 = fieldNorm(doc=5287)
0.014786332 = product of:
0.029572664 = sum of:
0.029572664 = weight(_text_:22 in 5287) [ClassicSimilarity], result of:
0.029572664 = score(doc=5287,freq=2.0), product of:
0.15286934 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.043654136 = queryNorm
0.19345059 = fieldWeight in 5287, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=5287)
0.5 = coord(1/2)
0.33333334 = coord(2/6)
- Abstract
- Purpose The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges' involvement. Then the study determines the variation in system rankings due to low effort relevance judgment in evaluating retrieval systems at different depth of evaluation. Design/methodology/approach Effort-based relevance judgments are generated using a proposed boxplot approach for simple document features, HTML features and readability features. The boxplot approach is a simple yet repeatable approach in classifying documents' effort while ensuring outlier scores do not skew the grading of the entire set of documents. Findings The retrieval systems evaluation using low effort relevance judgments has a stronger influence on shallow depth of evaluation compared to deeper depth. It is proved that difference in the system rankings is due to low effort documents and not the number of relevant documents. Originality/value Hence, it is crucial to evaluate retrieval systems at shallow depth using low effort relevance judgments.
- Date
- 20. 1.2015 18:30:22
Ravana, S.D.; Taheri, M.S.; Rajagopal, P.: Document-based approach to improve the accuracy of pairwise comparison in evaluating information retrieval systems (2015)
0.01
0.007504981 = product of:
0.022514943 = sum of:
0.007728611 = weight(_text_:in in 2587) [ClassicSimilarity], result of:
0.007728611 = score(doc=2587,freq=6.0), product of:
0.059380736 = queryWeight, product of:
1.3602545 = idf(docFreq=30841, maxDocs=44218)
0.043654136 = queryNorm
0.1301535 = fieldWeight in 2587, product of:
2.4494898 = tf(freq=6.0), with freq of:
6.0 = termFreq=6.0
1.3602545 = idf(docFreq=30841, maxDocs=44218)
0.0390625 = fieldNorm(doc=2587)
0.014786332 = product of:
0.029572664 = sum of:
0.029572664 = weight(_text_:22 in 2587) [ClassicSimilarity], result of:
0.029572664 = score(doc=2587,freq=2.0), product of:
0.15286934 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.043654136 = queryNorm
0.19345059 = fieldWeight in 2587, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=2587)
0.5 = coord(1/2)
0.33333334 = coord(2/6)
- Abstract
- Purpose The purpose of this paper is to propose a method to have more accurate results in comparing performance of the paired information retrieval (IR) systems with reference to the current method, which is based on the mean effectiveness scores of the systems across a set of identified topics/queries. Design/methodology/approach Based on the proposed approach, instead of the classic method of using a set of topic scores, the documents level scores are considered as the evaluation unit. These document scores are the defined document's weight, which play the role of the mean average precision (MAP) score of the systems as a significance test's statics. The experiments were conducted using the TREC 9 Web track collection. Findings The p-values generated through the two types of significance tests, namely the Student's t-test and Mann-Whitney show that by using the document level scores as an evaluation unit, the difference between IR systems is more significant compared with utilizing topic scores. Originality/value Utilizing a suitable test collection is a primary prerequisite for IR systems comparative evaluation. However, in addition to reusable test collections, having an accurate statistical testing is a necessity for these evaluations. The findings of this study will assist IR researchers to evaluate their retrieval systems and algorithms more accurately.
- Date
- 20. 1.2015 18:30:22
