Document (#41675)

Author

Losada, D.E.
Parapar, J.
Barreiro, A.

Title

When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections

Source

Journal of the Association for Information Science and Technology. 70(2019) no.1, S.49-60

Year

2019

Abstract

In information retrieval evaluation, pooling is a well-known technique to extract a sample of documents to be assessed for relevance. Given the pooled documents, a number of studies have proposed different prioritization methods to adjudicate documents for judgment. These methods follow different strategies to reduce the assessment effort. However, there is no clear guidance on how many relevance judgments are required for creating a reliable test collection. In this article we investigate and further develop methods to determine when to stop making relevance judgments. We propose a highly diversified set of stopping methods and provide a comprehensive analysis of the usefulness of the resulting test collections. Some of the stopping methods introduced here combine innovative estimates of recall with time series models used in Financial Trading. Experimental results on several representative collections show that some stopping methods can reduce up to 95% of the assessment effort and still produce a robust test collection. We demonstrate that the reduced set of judgments can be reliably employed to compare search systems using disparate effectiveness metrics such as Average Precision, NDCG, P@100, and Rank Biased Precision. With all these measures, the correlations found between full pool rankings and reduced pool rankings is very high.

Content

Vgl.: https://onlinelibrary.wiley.com/doi/10.1002/asi.24077.

Theme

Retrievalstudien

Similar documents (author)

1. Losada, D.E.; Barreiro, A.: Emebedding term similarity and inverse document frequency into a logical model of information retrieval (2003) 4.88

   4.8754888 = sum of:
     4.8754888 = weight(author_txt:losada in 1422) [ClassicSimilarity], result of:
       4.8754888 = fieldWeight in 1422, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.5 = fieldNorm(doc=1422)
   

2. Fernández, R.T.; Losada, D.E.: Effective sentence retrieval based on query-independent evidence (2012) 4.88

   4.8754888 = sum of:
     4.8754888 = weight(author_txt:losada in 2728) [ClassicSimilarity], result of:
       4.8754888 = fieldWeight in 2728, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.5 = fieldNorm(doc=2728)
   

3. Sweeney, S.; Crestani, F.; Losada, D.E.: 'Show me more' : incremental length summarisation using novelty detection (2008) 3.66

   3.6566167 = sum of:
     3.6566167 = weight(author_txt:losada in 2054) [ClassicSimilarity], result of:
       3.6566167 = fieldWeight in 2054, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.375 = fieldNorm(doc=2054)
   

4. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 3.66

   3.6566167 = sum of:
     3.6566167 = weight(author_txt:losada in 5098) [ClassicSimilarity], result of:
       3.6566167 = fieldWeight in 5098, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.375 = fieldNorm(doc=5098)
   

5. Parapar, J.; Losada, D.E.; Presedo-Quindimil, M.A.; Barreiro, A.: Using score distributions to compare statistical significance tests for information retrieval evaluation (2020) 3.05

   3.0471804 = sum of:
     3.0471804 = weight(author_txt:losada in 5506) [ClassicSimilarity], result of:
       3.0471804 = fieldWeight in 5506, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.3125 = fieldNorm(doc=5506)
   

Similar documents (content)

1. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.46

   0.45634177 = sum of:
     0.45634177 = product of:
       1.0371404 = sum of:
         0.03693998 = weight(abstract_txt:precision in 2591) [ClassicSimilarity], result of:
           0.03693998 = score(doc=2591,freq=1.0), product of:
             0.1069719 = queryWeight, product of:
               1.4336419 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.013504611 = queryNorm
             0.34532416 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.04105299 = weight(abstract_txt:effort in 2591) [ClassicSimilarity], result of:
           0.04105299 = score(doc=2591,freq=1.0), product of:
             0.11477178 = queryWeight, product of:
               1.4849895 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.013504611 = queryNorm
             0.35769236 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.039830085 = weight(abstract_txt:documents in 2591) [ClassicSimilarity], result of:
           0.039830085 = score(doc=2591,freq=3.0), product of:
             0.089276284 = queryWeight, product of:
               1.6040559 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.013504611 = queryNorm
             0.44614407 = fieldWeight in 2591, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.077066176 = weight(abstract_txt:reduce in 2591) [ClassicSimilarity], result of:
           0.077066176 = score(doc=2591,freq=2.0), product of:
             0.1386238 = queryWeight, product of:
               1.6320171 = boost
               6.2897153 = idf(docFreq=222, maxDocs=44218)
               0.013504611 = queryNorm
             0.5559375 = fieldWeight in 2591, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.2897153 = idf(docFreq=222, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.07146797 = weight(abstract_txt:rankings in 2591) [ClassicSimilarity], result of:
           0.07146797 = score(doc=2591,freq=1.0), product of:
             0.16609105 = queryWeight, product of:
               1.7864009 = boost
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.013504611 = queryNorm
             0.43029392 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.033972457 = weight(abstract_txt:collections in 2591) [ClassicSimilarity], result of:
           0.033972457 = score(doc=2591,freq=1.0), product of:
             0.11580314 = queryWeight, product of:
               1.8268867 = boost
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.013504611 = queryNorm
             0.29336387 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.11143089 = weight(abstract_txt:pool in 2591) [ClassicSimilarity], result of:
           0.11143089 = score(doc=2591,freq=1.0), product of:
             0.22332756 = queryWeight, product of:
               2.0714622 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.013504611 = queryNorm
             0.4989572 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.1390157 = weight(abstract_txt:relevance in 2591) [ClassicSimilarity], result of:
           0.1390157 = score(doc=2591,freq=7.0), product of:
             0.17046106 = queryWeight, product of:
               2.559372 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013504611 = queryNorm
             0.8155277 = fieldWeight in 2591, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.056296907 = weight(abstract_txt:test in 2591) [ClassicSimilarity], result of:
           0.056296907 = score(doc=2591,freq=1.0), product of:
             0.17848633 = queryWeight, product of:
               2.6189265 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.013504611 = queryNorm
             0.315413 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.3207527 = weight(abstract_txt:judgments in 2591) [ClassicSimilarity], result of:
           0.3207527 = score(doc=2591,freq=5.0), product of:
             0.3329703 = queryWeight, product of:
               3.577038 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.013504611 = queryNorm
             0.9633073 = fieldWeight in 2591, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.109314576 = weight(abstract_txt:methods in 2591) [ClassicSimilarity], result of:
           0.109314576 = score(doc=2591,freq=4.0), product of:
             0.21089195 = queryWeight, product of:
               3.7659092 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.013504611 = queryNorm
             0.518344 = fieldWeight in 2591, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
       0.44 = coord(11/25)
   

2. Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011) 0.42

   0.4239487 = sum of:
     0.4239487 = product of:
       0.8832265 = sum of:
         0.030589275 = weight(abstract_txt:when in 4197) [ClassicSimilarity], result of:
           0.030589275 = score(doc=4197,freq=5.0), product of:
             0.060300745 = queryWeight, product of:
               1.0763838 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.013504611 = queryNorm
             0.50727856 = fieldWeight in 4197, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.059234437 = weight(abstract_txt:reliably in 4197) [ClassicSimilarity], result of:
           0.059234437 = score(doc=4197,freq=1.0), product of:
             0.12714699 = queryWeight, product of:
               1.1052076 = boost
               8.518833 = idf(docFreq=23, maxDocs=44218)
               0.013504611 = queryNorm
             0.4658737 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.518833 = idf(docFreq=23, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.13344496 = weight(abstract_txt:pooling in 4197) [ClassicSimilarity], result of:
           0.13344496 = score(doc=4197,freq=3.0), product of:
             0.15150173 = queryWeight, product of:
               1.206423 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.013504611 = queryNorm
             0.8808148 = fieldWeight in 4197, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.055984184 = weight(abstract_txt:precision in 4197) [ClassicSimilarity], result of:
           0.055984184 = score(doc=4197,freq=3.0), product of:
             0.1069719 = queryWeight, product of:
               1.4336419 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.013504611 = queryNorm
             0.5233541 = fieldWeight in 4197, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.07184274 = weight(abstract_txt:effort in 4197) [ClassicSimilarity], result of:
           0.07184274 = score(doc=4197,freq=4.0), product of:
             0.11477178 = queryWeight, product of:
               1.4849895 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.013504611 = queryNorm
             0.62596166 = fieldWeight in 4197, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.051701512 = weight(abstract_txt:assessment in 4197) [ClassicSimilarity], result of:
           0.051701512 = score(doc=4197,freq=2.0), product of:
             0.11612491 = queryWeight, product of:
               1.4937177 = boost
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.013504611 = queryNorm
             0.44522327 = fieldWeight in 4197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.062534474 = weight(abstract_txt:rankings in 4197) [ClassicSimilarity], result of:
           0.062534474 = score(doc=4197,freq=1.0), product of:
             0.16609105 = queryWeight, product of:
               1.7864009 = boost
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.013504611 = queryNorm
             0.37650716 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.06462797 = weight(abstract_txt:reduced in 4197) [ClassicSimilarity], result of:
           0.06462797 = score(doc=4197,freq=1.0), product of:
             0.16977754 = queryWeight, product of:
               1.8061173 = boost
               6.9606886 = idf(docFreq=113, maxDocs=44218)
               0.013504611 = queryNorm
             0.38066265 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9606886 = idf(docFreq=113, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.04203877 = weight(abstract_txt:collections in 4197) [ClassicSimilarity], result of:
           0.04203877 = score(doc=4197,freq=2.0), product of:
             0.11580314 = queryWeight, product of:
               1.8268867 = boost
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.013504611 = queryNorm
             0.36301926 = fieldWeight in 4197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.13788868 = weight(abstract_txt:pool in 4197) [ClassicSimilarity], result of:
           0.13788868 = score(doc=4197,freq=2.0), product of:
             0.22332756 = queryWeight, product of:
               2.0714622 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.013504611 = queryNorm
             0.617428 = fieldWeight in 4197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.12551434 = weight(abstract_txt:judgments in 4197) [ClassicSimilarity], result of:
           0.12551434 = score(doc=4197,freq=1.0), product of:
             0.3329703 = queryWeight, product of:
               3.577038 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.013504611 = queryNorm
             0.3769536 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.047825128 = weight(abstract_txt:methods in 4197) [ClassicSimilarity], result of:
           0.047825128 = score(doc=4197,freq=1.0), product of:
             0.21089195 = queryWeight, product of:
               3.7659092 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.013504611 = queryNorm
             0.2267755 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
       0.48 = coord(12/25)
   

3. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.36

   0.35749277 = sum of:
     0.35749277 = product of:
       0.89373195 = sum of:
         0.088050835 = weight(abstract_txt:pooling in 5098) [ClassicSimilarity], result of:
           0.088050835 = score(doc=5098,freq=1.0), product of:
             0.15150173 = queryWeight, product of:
               1.206423 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.013504611 = queryNorm
             0.581187 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.027454719 = weight(abstract_txt:making in 5098) [ClassicSimilarity], result of:
           0.027454719 = score(doc=5098,freq=1.0), product of:
             0.087770775 = queryWeight, product of:
               1.298616 = boost
               5.0048037 = idf(docFreq=805, maxDocs=44218)
               0.013504611 = queryNorm
             0.31280023 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0048037 = idf(docFreq=805, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.04105299 = weight(abstract_txt:effort in 5098) [ClassicSimilarity], result of:
           0.04105299 = score(doc=5098,freq=1.0), product of:
             0.11477178 = queryWeight, product of:
               1.4849895 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.013504611 = queryNorm
             0.35769236 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.04178113 = weight(abstract_txt:assessment in 5098) [ClassicSimilarity], result of:
           0.04178113 = score(doc=5098,freq=1.0), product of:
             0.11612491 = queryWeight, product of:
               1.4937177 = boost
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.013504611 = queryNorm
             0.35979474 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.045991823 = weight(abstract_txt:documents in 5098) [ClassicSimilarity], result of:
           0.045991823 = score(doc=5098,freq=4.0), product of:
             0.089276284 = queryWeight, product of:
               1.6040559 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.013504611 = queryNorm
             0.5151628 = fieldWeight in 5098, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.033972457 = weight(abstract_txt:collections in 5098) [ClassicSimilarity], result of:
           0.033972457 = score(doc=5098,freq=1.0), product of:
             0.11580314 = queryWeight, product of:
               1.8268867 = boost
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.013504611 = queryNorm
             0.29336387 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.15758708 = weight(abstract_txt:pool in 5098) [ClassicSimilarity], result of:
           0.15758708 = score(doc=5098,freq=2.0), product of:
             0.22332756 = queryWeight, product of:
               2.0714622 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.013504611 = queryNorm
             0.70563203 = fieldWeight in 5098, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.09100715 = weight(abstract_txt:relevance in 5098) [ClassicSimilarity], result of:
           0.09100715 = score(doc=5098,freq=3.0), product of:
             0.17046106 = queryWeight, product of:
               2.559372 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013504611 = queryNorm
             0.5338882 = fieldWeight in 5098, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.20286183 = weight(abstract_txt:judgments in 5098) [ClassicSimilarity], result of:
           0.20286183 = score(doc=5098,freq=2.0), product of:
             0.3329703 = queryWeight, product of:
               3.577038 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.013504611 = queryNorm
             0.60924906 = fieldWeight in 5098, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.16397187 = weight(abstract_txt:methods in 5098) [ClassicSimilarity], result of:
           0.16397187 = score(doc=5098,freq=9.0), product of:
             0.21089195 = queryWeight, product of:
               3.7659092 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.013504611 = queryNorm
             0.777516 = fieldWeight in 5098, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
       0.4 = coord(10/25)
   

4. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.22

   0.21605907 = sum of:
     0.21605907 = product of:
       0.60016406 = sum of:
         0.01658259 = weight(abstract_txt:when in 5092) [ClassicSimilarity], result of:
           0.01658259 = score(doc=5092,freq=2.0), product of:
             0.060300745 = queryWeight, product of:
               1.0763838 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.013504611 = queryNorm
             0.2749981 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.050772373 = weight(abstract_txt:reliably in 5092) [ClassicSimilarity], result of:
           0.050772373 = score(doc=5092,freq=1.0), product of:
             0.12714699 = queryWeight, product of:
               1.1052076 = boost
               8.518833 = idf(docFreq=23, maxDocs=44218)
               0.013504611 = queryNorm
             0.3993203 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.518833 = idf(docFreq=23, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.06603813 = weight(abstract_txt:pooling in 5092) [ClassicSimilarity], result of:
           0.06603813 = score(doc=5092,freq=1.0), product of:
             0.15150173 = queryWeight, product of:
               1.206423 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.013504611 = queryNorm
             0.43589026 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.030789742 = weight(abstract_txt:effort in 5092) [ClassicSimilarity], result of:
           0.030789742 = score(doc=5092,freq=1.0), product of:
             0.11477178 = queryWeight, product of:
               1.4849895 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.013504611 = queryNorm
             0.26826927 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.050958686 = weight(abstract_txt:collections in 5092) [ClassicSimilarity], result of:
           0.050958686 = score(doc=5092,freq=4.0), product of:
             0.11580314 = queryWeight, product of:
               1.8268867 = boost
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.013504611 = queryNorm
             0.4400458 = fieldWeight in 5092, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.05573027 = weight(abstract_txt:relevance in 5092) [ClassicSimilarity], result of:
           0.05573027 = score(doc=5092,freq=2.0), product of:
             0.17046106 = queryWeight, product of:
               2.559372 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013504611 = queryNorm
             0.32693842 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.07313182 = weight(abstract_txt:test in 5092) [ClassicSimilarity], result of:
           0.07313182 = score(doc=5092,freq=3.0), product of:
             0.17848633 = queryWeight, product of:
               2.6189265 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.013504611 = queryNorm
             0.40973347 = fieldWeight in 5092, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.21516746 = weight(abstract_txt:judgments in 5092) [ClassicSimilarity], result of:
           0.21516746 = score(doc=5092,freq=4.0), product of:
             0.3329703 = queryWeight, product of:
               3.577038 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.013504611 = queryNorm
             0.6462062 = fieldWeight in 5092, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.040992968 = weight(abstract_txt:methods in 5092) [ClassicSimilarity], result of:
           0.040992968 = score(doc=5092,freq=1.0), product of:
             0.21089195 = queryWeight, product of:
               3.7659092 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.013504611 = queryNorm
             0.194379 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
       0.36 = coord(9/25)
   

5. Yang, Y.; Wilbur, J.: Using corpus statistics to remove redundant words in text categorization (1996) 0.18

   0.1834437 = sum of:
     0.1834437 = product of:
       0.57326156 = sum of:
         0.046174977 = weight(abstract_txt:precision in 4199) [ClassicSimilarity], result of:
           0.046174977 = score(doc=4199,freq=1.0), product of:
             0.1069719 = queryWeight, product of:
               1.4336419 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.013504611 = queryNorm
             0.4316552 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.04065141 = weight(abstract_txt:documents in 4199) [ClassicSimilarity], result of:
           0.04065141 = score(doc=4199,freq=2.0), product of:
             0.089276284 = queryWeight, product of:
               1.6040559 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.013504611 = queryNorm
             0.4553439 = fieldWeight in 4199, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.06811752 = weight(abstract_txt:reduce in 4199) [ClassicSimilarity], result of:
           0.06811752 = score(doc=4199,freq=1.0), product of:
             0.1386238 = queryWeight, product of:
               1.6320171 = boost
               6.2897153 = idf(docFreq=222, maxDocs=44218)
               0.013504611 = queryNorm
             0.491384 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.2897153 = idf(docFreq=222, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.09232567 = weight(abstract_txt:reduced in 4199) [ClassicSimilarity], result of:
           0.09232567 = score(doc=4199,freq=1.0), product of:
             0.16977754 = queryWeight, product of:
               1.8061173 = boost
               6.9606886 = idf(docFreq=113, maxDocs=44218)
               0.013504611 = queryNorm
             0.5438038 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9606886 = idf(docFreq=113, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.042465568 = weight(abstract_txt:collections in 4199) [ClassicSimilarity], result of:
           0.042465568 = score(doc=4199,freq=1.0), product of:
             0.11580314 = queryWeight, product of:
               1.8268867 = boost
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.013504611 = queryNorm
             0.36670482 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.693822 = idf(docFreq=1099, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.116533935 = weight(abstract_txt:stop in 4199) [ClassicSimilarity], result of:
           0.116533935 = score(doc=4199,freq=1.0), product of:
             0.19828972 = queryWeight, product of:
               1.9518926 = boost
               7.5225 = idf(docFreq=64, maxDocs=44218)
               0.013504611 = queryNorm
             0.5876953 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5225 = idf(docFreq=64, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.070371136 = weight(abstract_txt:test in 4199) [ClassicSimilarity], result of:
           0.070371136 = score(doc=4199,freq=1.0), product of:
             0.17848633 = queryWeight, product of:
               2.6189265 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.013504611 = queryNorm
             0.39426625 = fieldWeight in 4199, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
         0.09662135 = weight(abstract_txt:methods in 4199) [ClassicSimilarity], result of:
           0.09662135 = score(doc=4199,freq=2.0), product of:
             0.21089195 = queryWeight, product of:
               3.7659092 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.013504611 = queryNorm
             0.4581557 = fieldWeight in 4199, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.078125 = fieldNorm(doc=4199)
       0.32 = coord(8/25)