-
Ravana, S.D.; Taheri, M.S.; Rajagopal, P.: Document-based approach to improve the accuracy of pairwise comparison in evaluating information retrieval systems (2015)
0.04
0.038388878 = product of:
0.076777756 = sum of:
0.076777756 = sum of:
0.045954376 = weight(_text_:p in 2587) [ClassicSimilarity], result of:
0.045954376 = score(doc=2587,freq=4.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.28089944 = fieldWeight in 2587, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0390625 = fieldNorm(doc=2587)
0.03082338 = weight(_text_:22 in 2587) [ClassicSimilarity], result of:
0.03082338 = score(doc=2587,freq=2.0), product of:
0.15933464 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.045500398 = queryNorm
0.19345059 = fieldWeight in 2587, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=2587)
0.5 = coord(1/2)
- Abstract
- Purpose The purpose of this paper is to propose a method to have more accurate results in comparing performance of the paired information retrieval (IR) systems with reference to the current method, which is based on the mean effectiveness scores of the systems across a set of identified topics/queries. Design/methodology/approach Based on the proposed approach, instead of the classic method of using a set of topic scores, the documents level scores are considered as the evaluation unit. These document scores are the defined document's weight, which play the role of the mean average precision (MAP) score of the systems as a significance test's statics. The experiments were conducted using the TREC 9 Web track collection. Findings The p-values generated through the two types of significance tests, namely the Student's t-test and Mann-Whitney show that by using the document level scores as an evaluation unit, the difference between IR systems is more significant compared with utilizing topic scores. Originality/value Utilizing a suitable test collection is a primary prerequisite for IR systems comparative evaluation. However, in addition to reusable test collections, having an accurate statistical testing is a necessity for these evaluations. The findings of this study will assist IR researchers to evaluate their retrieval systems and algorithms more accurately.
- Date
- 20. 1.2015 18:30:22
-
Reichert, S.; Mayr, P.: Untersuchung von Relevanzeigenschaften in einem kontrollierten Eyetracking-Experiment (2012)
0.04
0.037990816 = product of:
0.07598163 = sum of:
0.07598163 = sum of:
0.03899358 = weight(_text_:p in 328) [ClassicSimilarity], result of:
0.03899358 = score(doc=328,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.23835106 = fieldWeight in 328, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.046875 = fieldNorm(doc=328)
0.036988053 = weight(_text_:22 in 328) [ClassicSimilarity], result of:
0.036988053 = score(doc=328,freq=2.0), product of:
0.15933464 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.045500398 = queryNorm
0.23214069 = fieldWeight in 328, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.046875 = fieldNorm(doc=328)
0.5 = coord(1/2)
- Date
- 22. 7.2012 19:25:54
-
Rajagopal, P.; Ravana, S.D.; Koh, Y.S.; Balakrishnan, V.: Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment (2019)
0.03
0.031659015 = product of:
0.06331803 = sum of:
0.06331803 = sum of:
0.03249465 = weight(_text_:p in 5287) [ClassicSimilarity], result of:
0.03249465 = score(doc=5287,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.19862589 = fieldWeight in 5287, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0390625 = fieldNorm(doc=5287)
0.03082338 = weight(_text_:22 in 5287) [ClassicSimilarity], result of:
0.03082338 = score(doc=5287,freq=2.0), product of:
0.15933464 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.045500398 = queryNorm
0.19345059 = fieldWeight in 5287, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=5287)
0.5 = coord(1/2)
- Date
- 20. 1.2015 18:30:22
-
Munkelt, J.; Schaer, P.: Towards an IR test collection for the German National Library (2018)
0.02
0.022746254 = product of:
0.045492508 = sum of:
0.045492508 = product of:
0.090985015 = sum of:
0.090985015 = weight(_text_:p in 5780) [ClassicSimilarity], result of:
0.090985015 = score(doc=5780,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.55615246 = fieldWeight in 5780, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.109375 = fieldNorm(doc=5780)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Balog, K.; Schuth, A.; Dekker, P.; Tavakolpoursaleh, N.; Schaer, P.; Chuang, P.-Y.: Overview of the TREC 2016 Open Search track Academic Search Edition (2016)
0.02
0.022512956 = product of:
0.04502591 = sum of:
0.04502591 = product of:
0.09005182 = sum of:
0.09005182 = weight(_text_:p in 43) [ClassicSimilarity], result of:
0.09005182 = score(doc=43,freq=6.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.55044824 = fieldWeight in 43, product of:
2.4494898 = tf(freq=6.0), with freq of:
6.0 = termFreq=6.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0625 = fieldNorm(doc=43)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Schirrmeister, N.-P.; Keil, S.: Aufbau einer Infrastruktur für Information Retrieval-Evaluationen (2012)
0.01
0.01299786 = product of:
0.02599572 = sum of:
0.02599572 = product of:
0.05199144 = sum of:
0.05199144 = weight(_text_:p in 3097) [ClassicSimilarity], result of:
0.05199144 = score(doc=3097,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.31780142 = fieldWeight in 3097, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0625 = fieldNorm(doc=3097)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Schaer, P.; Mayr, P.; Sünkler, S.; Lewandowski, D.: How relevant is the long tail? : a relevance assessment study on million short (2016)
0.01
0.011488594 = product of:
0.022977188 = sum of:
0.022977188 = product of:
0.045954376 = sum of:
0.045954376 = weight(_text_:p in 3144) [ClassicSimilarity], result of:
0.045954376 = score(doc=3144,freq=4.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.28089944 = fieldWeight in 3144, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0390625 = fieldNorm(doc=3144)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Geist, K.: Qualität und Relevanz von bildungsbezogenen Suchergebnissen bei der Suche im Web (2012)
0.01
0.009748395 = product of:
0.01949679 = sum of:
0.01949679 = product of:
0.03899358 = sum of:
0.03899358 = weight(_text_:p in 570) [ClassicSimilarity], result of:
0.03899358 = score(doc=570,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.23835106 = fieldWeight in 570, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.046875 = fieldNorm(doc=570)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Content
- Vgl. auch unter: http://www.univie.ac.at/voeb/blog/?p=23628 bzw. http://eprints.rclis.org/handle/10760/17623.
-
Colace, F.; Santo, M. de; Greco, L.; Napoletano, P.: Improving relevance feedback-based query expansion by the use of a weighted word pairs approach (2015)
0.01
0.009748395 = product of:
0.01949679 = sum of:
0.01949679 = product of:
0.03899358 = sum of:
0.03899358 = weight(_text_:p in 2263) [ClassicSimilarity], result of:
0.03899358 = score(doc=2263,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.23835106 = fieldWeight in 2263, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.046875 = fieldNorm(doc=2263)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Li, J.; Zhang, P.; Song, D.; Wu, Y.: Understanding an enriched multidimensional user relevance model by analyzing query logs (2017)
0.01
0.009748395 = product of:
0.01949679 = sum of:
0.01949679 = product of:
0.03899358 = sum of:
0.03899358 = weight(_text_:p in 3961) [ClassicSimilarity], result of:
0.03899358 = score(doc=3961,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.23835106 = fieldWeight in 3961, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.046875 = fieldNorm(doc=3961)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Munkelt, J.; Schaer, P.; Lepsky, K.: Towards an IR test collection for the German National Library (2018)
0.01
0.009748395 = product of:
0.01949679 = sum of:
0.01949679 = product of:
0.03899358 = sum of:
0.03899358 = weight(_text_:p in 4311) [ClassicSimilarity], result of:
0.03899358 = score(doc=4311,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.23835106 = fieldWeight in 4311, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.046875 = fieldNorm(doc=4311)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Vakkari, P.; Huuskonen, S.: Search effort degrades search output but improves task outcome (2012)
0.01
0.008123662 = product of:
0.016247325 = sum of:
0.016247325 = product of:
0.03249465 = sum of:
0.03249465 = weight(_text_:p in 46) [ClassicSimilarity], result of:
0.03249465 = score(doc=46,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.19862589 = fieldWeight in 46, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0390625 = fieldNorm(doc=46)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Hider, P.: ¬The search value added by professional indexing to a bibliographic database (2017)
0.01
0.008123662 = product of:
0.016247325 = sum of:
0.016247325 = product of:
0.03249465 = sum of:
0.03249465 = weight(_text_:p in 3868) [ClassicSimilarity], result of:
0.03249465 = score(doc=3868,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.19862589 = fieldWeight in 3868, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0390625 = fieldNorm(doc=3868)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Hider, P.: ¬The search value added by professional indexing to a bibliographic database (2018)
0.01
0.008123662 = product of:
0.016247325 = sum of:
0.016247325 = product of:
0.03249465 = sum of:
0.03249465 = weight(_text_:p in 4300) [ClassicSimilarity], result of:
0.03249465 = score(doc=4300,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.19862589 = fieldWeight in 4300, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0390625 = fieldNorm(doc=4300)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019)
0.01
0.008123662 = product of:
0.016247325 = sum of:
0.016247325 = product of:
0.03249465 = sum of:
0.03249465 = weight(_text_:p in 4674) [ClassicSimilarity], result of:
0.03249465 = score(doc=4674,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.19862589 = fieldWeight in 4674, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.0390625 = fieldNorm(doc=4674)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- In information retrieval evaluation, pooling is a well-known technique to extract a sample of documents to be assessed for relevance. Given the pooled documents, a number of studies have proposed different prioritization methods to adjudicate documents for judgment. These methods follow different strategies to reduce the assessment effort. However, there is no clear guidance on how many relevance judgments are required for creating a reliable test collection. In this article we investigate and further develop methods to determine when to stop making relevance judgments. We propose a highly diversified set of stopping methods and provide a comprehensive analysis of the usefulness of the resulting test collections. Some of the stopping methods introduced here combine innovative estimates of recall with time series models used in Financial Trading. Experimental results on several representative collections show that some stopping methods can reduce up to 95% of the assessment effort and still produce a robust test collection. We demonstrate that the reduced set of judgments can be reliably employed to compare search systems using disparate effectiveness metrics such as Average Precision, NDCG, P@100, and Rank Biased Precision. With all these measures, the correlations found between full pool rankings and reduced pool rankings is very high.
-
Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011)
0.01
0.007705845 = product of:
0.01541169 = sum of:
0.01541169 = product of:
0.03082338 = sum of:
0.03082338 = weight(_text_:22 in 4197) [ClassicSimilarity], result of:
0.03082338 = score(doc=4197,freq=2.0), product of:
0.15933464 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.045500398 = queryNorm
0.19345059 = fieldWeight in 4197, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=4197)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 22. 1.2011 14:20:56
-
Chu, H.: Factors affecting relevance judgment : a report from TREC Legal track (2011)
0.01
0.007705845 = product of:
0.01541169 = sum of:
0.01541169 = product of:
0.03082338 = sum of:
0.03082338 = weight(_text_:22 in 4540) [ClassicSimilarity], result of:
0.03082338 = score(doc=4540,freq=2.0), product of:
0.15933464 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.045500398 = queryNorm
0.19345059 = fieldWeight in 4540, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=4540)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 12. 7.2011 18:29:22
-
Wildemuth, B.; Freund, L.; Toms, E.G.: Untangling search task complexity and difficulty in the context of interactive information retrieval studies (2014)
0.01
0.007705845 = product of:
0.01541169 = sum of:
0.01541169 = product of:
0.03082338 = sum of:
0.03082338 = weight(_text_:22 in 1786) [ClassicSimilarity], result of:
0.03082338 = score(doc=1786,freq=2.0), product of:
0.15933464 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.045500398 = queryNorm
0.19345059 = fieldWeight in 1786, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=1786)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 6. 4.2015 19:31:22
-
Borlund, P.: ¬A study of the use of simulated work task situations in interactive information retrieval evaluations : a meta-evaluation (2016)
0.01
0.00649893 = product of:
0.01299786 = sum of:
0.01299786 = product of:
0.02599572 = sum of:
0.02599572 = weight(_text_:p in 2880) [ClassicSimilarity], result of:
0.02599572 = score(doc=2880,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.15890071 = fieldWeight in 2880, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.03125 = fieldNorm(doc=2880)
0.5 = coord(1/2)
0.5 = coord(1/2)
-
Leiva-Mederos, A.; Senso, J.A.; Hidalgo-Delgado, Y.; Hipola, P.: Working framework of semantic interoperability for CRIS with heterogeneous data sources (2017)
0.01
0.00649893 = product of:
0.01299786 = sum of:
0.01299786 = product of:
0.02599572 = sum of:
0.02599572 = weight(_text_:p in 3706) [ClassicSimilarity], result of:
0.02599572 = score(doc=3706,freq=2.0), product of:
0.16359726 = queryWeight, product of:
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.045500398 = queryNorm
0.15890071 = fieldWeight in 3706, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5955126 = idf(docFreq=3298, maxDocs=44218)
0.03125 = fieldNorm(doc=3706)
0.5 = coord(1/2)
0.5 = coord(1/2)
