Document (#41676)

Author
Losada, D.E.
Parapar, J.
Barreiro, A.
Title
When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections
Source
Journal of the Association for Information Science and Technology. 70(2019) no.1, S.49-60
Year
2019
Abstract
In information retrieval evaluation, pooling is a well-known technique to extract a sample of documents to be assessed for relevance. Given the pooled documents, a number of studies have proposed different prioritization methods to adjudicate documents for judgment. These methods follow different strategies to reduce the assessment effort. However, there is no clear guidance on how many relevance judgments are required for creating a reliable test collection. In this article we investigate and further develop methods to determine when to stop making relevance judgments. We propose a highly diversified set of stopping methods and provide a comprehensive analysis of the usefulness of the resulting test collections. Some of the stopping methods introduced here combine innovative estimates of recall with time series models used in Financial Trading. Experimental results on several representative collections show that some stopping methods can reduce up to 95% of the assessment effort and still produce a robust test collection. We demonstrate that the reduced set of judgments can be reliably employed to compare search systems using disparate effectiveness metrics such as Average Precision, NDCG, P@100, and Rank Biased Precision. With all these measures, the correlations found between full pool rankings and reduced pool rankings is very high.
Content
Vgl.: https://onlinelibrary.wiley.com/doi/10.1002/asi.24077.
Theme
Retrievalstudien

Similar documents (author)

  1. Losada, D.E.; Barreiro, A.: Emebedding term similarity and inverse document frequency into a logical model of information retrieval (2003) 4.93
    4.9304624 = sum of:
      4.9304624 = weight(author_txt:losada in 2423) [ClassicSimilarity], result of:
        4.9304624 = fieldWeight in 2423, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.860925 = idf(docFreq=5, maxDocs=42306)
          0.5 = fieldNorm(doc=2423)
    
  2. Fernández, R.T.; Losada, D.E.: Effective sentence retrieval based on query-independent evidence (2012) 4.93
    4.9304624 = sum of:
      4.9304624 = weight(author_txt:losada in 4729) [ClassicSimilarity], result of:
        4.9304624 = fieldWeight in 4729, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.860925 = idf(docFreq=5, maxDocs=42306)
          0.5 = fieldNorm(doc=4729)
    
  3. Sweeney, S.; Crestani, F.; Losada, D.E.: 'Show me more' : incremental length summarisation using novelty detection (2008) 3.70
    3.697847 = sum of:
      3.697847 = weight(author_txt:losada in 4055) [ClassicSimilarity], result of:
        3.697847 = fieldWeight in 4055, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.860925 = idf(docFreq=5, maxDocs=42306)
          0.375 = fieldNorm(doc=4055)
    
  4. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 3.70
    3.697847 = sum of:
      3.697847 = weight(author_txt:losada in 2016) [ClassicSimilarity], result of:
        3.697847 = fieldWeight in 2016, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.860925 = idf(docFreq=5, maxDocs=42306)
          0.375 = fieldNorm(doc=2016)
    

Similar documents (content)

  1. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.46
    0.45690793 = sum of:
      0.45690793 = product of:
        1.0384271 = sum of:
          0.036310103 = weight(abstract_txt:precision in 4592) [ClassicSimilarity], result of:
            0.036310103 = score(doc=4592,freq=1.0), product of:
              0.105384134 = queryWeight, product of:
                1.4206666 = boost
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.013455841 = queryNorm
              0.34454998 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.040854152 = weight(abstract_txt:effort in 4592) [ClassicSimilarity], result of:
            0.040854152 = score(doc=4592,freq=1.0), product of:
              0.1140025 = queryWeight, product of:
                1.4776165 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.013455841 = queryNorm
              0.3583619 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.039257385 = weight(abstract_txt:documents in 4592) [ClassicSimilarity], result of:
            0.039257385 = score(doc=4592,freq=3.0), product of:
              0.08811053 = queryWeight, product of:
                1.5909777 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.013455841 = queryNorm
              0.445547 = fieldWeight in 4592, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.07772664 = weight(abstract_txt:reduce in 4592) [ClassicSimilarity], result of:
            0.07772664 = score(doc=4592,freq=2.0), product of:
              0.13892919 = queryWeight, product of:
                1.6311778 = boost
                6.3296742 = idf(docFreq=204, maxDocs=42306)
                0.013455841 = queryNorm
              0.55946946 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.3296742 = idf(docFreq=204, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.070124365 = weight(abstract_txt:rankings in 4592) [ClassicSimilarity], result of:
            0.070124365 = score(doc=4592,freq=1.0), product of:
              0.16343166 = queryWeight, product of:
                1.7691826 = boost
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.013455841 = queryNorm
              0.42907453 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.03387482 = weight(abstract_txt:collections in 4592) [ClassicSimilarity], result of:
            0.03387482 = score(doc=4592,freq=1.0), product of:
              0.11517865 = queryWeight, product of:
                1.8190145 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.013455841 = queryNorm
              0.29410675 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.11271068 = weight(abstract_txt:pool in 4592) [ClassicSimilarity], result of:
            0.11271068 = score(doc=4592,freq=1.0), product of:
              0.2242506 = queryWeight, product of:
                2.0723894 = boost
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.013455841 = queryNorm
              0.5026104 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.138331 = weight(abstract_txt:relevance in 4592) [ClassicSimilarity], result of:
            0.138331 = score(doc=4592,freq=7.0), product of:
              0.1693092 = queryWeight, product of:
                2.5465946 = boost
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.013455841 = queryNorm
              0.81703174 = fieldWeight in 4592, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.056418706 = weight(abstract_txt:test in 4592) [ClassicSimilarity], result of:
            0.056418706 = score(doc=4592,freq=1.0), product of:
              0.17812122 = queryWeight, product of:
                2.6120253 = boost
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.013455841 = queryNorm
              0.3167433 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.32191476 = weight(abstract_txt:judgments in 4592) [ClassicSimilarity], result of:
            0.32191476 = score(doc=4592,freq=5.0), product of:
              0.33261168 = queryWeight, product of:
                3.569343 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.013455841 = queryNorm
              0.9678396 = fieldWeight in 4592, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.11090457 = weight(abstract_txt:methods in 4592) [ClassicSimilarity], result of:
            0.11090457 = score(doc=4592,freq=4.0), product of:
              0.21219048 = queryWeight, product of:
                3.7713907 = boost
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.013455841 = queryNorm
              0.52266514 = fieldWeight in 4592, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
        0.44 = coord(11/25)
    
  2. Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011) 0.42
    0.4231788 = sum of:
      0.4231788 = product of:
        0.8816225 = sum of:
          0.030973466 = weight(abstract_txt:when in 1198) [ClassicSimilarity], result of:
            0.030973466 = score(doc=1198,freq=5.0), product of:
              0.060592853 = queryWeight, product of:
                1.0772469 = boost
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.013455841 = queryNorm
              0.5111736 = fieldWeight in 1198, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.05858384 = weight(abstract_txt:reliably in 1198) [ClassicSimilarity], result of:
            0.05858384 = score(doc=1198,freq=1.0), product of:
              0.12577474 = queryWeight, product of:
                1.0974538 = boost
                8.51719 = idf(docFreq=22, maxDocs=42306)
                0.013455841 = queryNorm
              0.46578383 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.51719 = idf(docFreq=22, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.13018143 = weight(abstract_txt:pooling in 1198) [ClassicSimilarity], result of:
            0.13018143 = score(doc=1198,freq=3.0), product of:
              0.14850253 = queryWeight, product of:
                1.1924948 = boost
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.013455841 = queryNorm
              0.87662774 = fieldWeight in 1198, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.05502958 = weight(abstract_txt:precision in 1198) [ClassicSimilarity], result of:
            0.05502958 = score(doc=1198,freq=3.0), product of:
              0.105384134 = queryWeight, product of:
                1.4206666 = boost
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.013455841 = queryNorm
              0.52218086 = fieldWeight in 1198, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.071494766 = weight(abstract_txt:effort in 1198) [ClassicSimilarity], result of:
            0.071494766 = score(doc=1198,freq=4.0), product of:
              0.1140025 = queryWeight, product of:
                1.4776165 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.013455841 = queryNorm
              0.6271333 = fieldWeight in 1198, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.0525738 = weight(abstract_txt:assessment in 1198) [ClassicSimilarity], result of:
            0.0525738 = score(doc=1198,freq=2.0), product of:
              0.117018476 = queryWeight, product of:
                1.4970343 = boost
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.013455841 = queryNorm
              0.44927776 = fieldWeight in 1198, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.061358817 = weight(abstract_txt:rankings in 1198) [ClassicSimilarity], result of:
            0.061358817 = score(doc=1198,freq=1.0), product of:
              0.16343166 = queryWeight, product of:
                1.7691826 = boost
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.013455841 = queryNorm
              0.3754402 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.06554667 = weight(abstract_txt:reduced in 1198) [ClassicSimilarity], result of:
            0.06554667 = score(doc=1198,freq=1.0), product of:
              0.17078587 = queryWeight, product of:
                1.80855 = boost
                7.0179553 = idf(docFreq=102, maxDocs=42306)
                0.013455841 = queryNorm
              0.38379443 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.0179553 = idf(docFreq=102, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.04191795 = weight(abstract_txt:collections in 1198) [ClassicSimilarity], result of:
            0.04191795 = score(doc=1198,freq=2.0), product of:
              0.11517865 = queryWeight, product of:
                1.8190145 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.013455841 = queryNorm
              0.36393854 = fieldWeight in 1198, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.13947235 = weight(abstract_txt:pool in 1198) [ClassicSimilarity], result of:
            0.13947235 = score(doc=1198,freq=2.0), product of:
              0.2242506 = queryWeight, product of:
                2.0723894 = boost
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.013455841 = queryNorm
              0.6219486 = fieldWeight in 1198, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.12596907 = weight(abstract_txt:judgments in 1198) [ClassicSimilarity], result of:
            0.12596907 = score(doc=1198,freq=1.0), product of:
              0.33261168 = queryWeight, product of:
                3.569343 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.013455841 = queryNorm
              0.37872714 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.04852075 = weight(abstract_txt:methods in 1198) [ClassicSimilarity], result of:
            0.04852075 = score(doc=1198,freq=1.0), product of:
              0.21219048 = queryWeight, product of:
                3.7713907 = boost
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.013455841 = queryNorm
              0.22866601 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
        0.48 = coord(12/25)
    
  3. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.36
    0.35853335 = sum of:
      0.35853335 = product of:
        0.8963334 = sum of:
          0.085897475 = weight(abstract_txt:pooling in 2016) [ClassicSimilarity], result of:
            0.085897475 = score(doc=2016,freq=1.0), product of:
              0.14850253 = queryWeight, product of:
                1.1924948 = boost
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.013455841 = queryNorm
              0.57842433 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.027980903 = weight(abstract_txt:making in 2016) [ClassicSimilarity], result of:
            0.027980903 = score(doc=2016,freq=1.0), product of:
              0.088579126 = queryWeight, product of:
                1.3024775 = boost
                5.0541754 = idf(docFreq=733, maxDocs=42306)
                0.013455841 = queryNorm
              0.31588596 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.0541754 = idf(docFreq=733, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.040854152 = weight(abstract_txt:effort in 2016) [ClassicSimilarity], result of:
            0.040854152 = score(doc=2016,freq=1.0), product of:
              0.1140025 = queryWeight, product of:
                1.4776165 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.013455841 = queryNorm
              0.3583619 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.04248604 = weight(abstract_txt:assessment in 2016) [ClassicSimilarity], result of:
            0.04248604 = score(doc=2016,freq=1.0), product of:
              0.117018476 = queryWeight, product of:
                1.4970343 = boost
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.013455841 = queryNorm
              0.36307123 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.04533052 = weight(abstract_txt:documents in 2016) [ClassicSimilarity], result of:
            0.04533052 = score(doc=2016,freq=4.0), product of:
              0.08811053 = queryWeight, product of:
                1.5909777 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.013455841 = queryNorm
              0.5144734 = fieldWeight in 2016, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.03387482 = weight(abstract_txt:collections in 2016) [ClassicSimilarity], result of:
            0.03387482 = score(doc=2016,freq=1.0), product of:
              0.11517865 = queryWeight, product of:
                1.8190145 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.013455841 = queryNorm
              0.29410675 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.15939696 = weight(abstract_txt:pool in 2016) [ClassicSimilarity], result of:
            0.15939696 = score(doc=2016,freq=2.0), product of:
              0.2242506 = queryWeight, product of:
                2.0723894 = boost
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.013455841 = queryNorm
              0.7107984 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.090558894 = weight(abstract_txt:relevance in 2016) [ClassicSimilarity], result of:
            0.090558894 = score(doc=2016,freq=3.0), product of:
              0.1693092 = queryWeight, product of:
                2.5465946 = boost
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.013455841 = queryNorm
              0.53487283 = fieldWeight in 2016, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.20359677 = weight(abstract_txt:judgments in 2016) [ClassicSimilarity], result of:
            0.20359677 = score(doc=2016,freq=2.0), product of:
              0.33261168 = queryWeight, product of:
                3.569343 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.013455841 = queryNorm
              0.6121155 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.16635685 = weight(abstract_txt:methods in 2016) [ClassicSimilarity], result of:
            0.16635685 = score(doc=2016,freq=9.0), product of:
              0.21219048 = queryWeight, product of:
                3.7713907 = boost
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.013455841 = queryNorm
              0.7839977 = fieldWeight in 2016, product of:
                3.0 = tf(freq=9.0), with freq of:
                  9.0 = termFreq=9.0
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
        0.4 = coord(10/25)
    
  4. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.22
    0.21569888 = sum of:
      0.21569888 = product of:
        0.59916353 = sum of:
          0.016790863 = weight(abstract_txt:when in 2010) [ClassicSimilarity], result of:
            0.016790863 = score(doc=2010,freq=2.0), product of:
              0.060592853 = queryWeight, product of:
                1.0772469 = boost
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.013455841 = queryNorm
              0.27710962 = fieldWeight in 2010, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.050214723 = weight(abstract_txt:reliably in 2010) [ClassicSimilarity], result of:
            0.050214723 = score(doc=2010,freq=1.0), product of:
              0.12577474 = queryWeight, product of:
                1.0974538 = boost
                8.51719 = idf(docFreq=22, maxDocs=42306)
                0.013455841 = queryNorm
              0.3992433 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.51719 = idf(docFreq=22, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.06442311 = weight(abstract_txt:pooling in 2010) [ClassicSimilarity], result of:
            0.06442311 = score(doc=2010,freq=1.0), product of:
              0.14850253 = queryWeight, product of:
                1.1924948 = boost
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.013455841 = queryNorm
              0.43381825 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.030640613 = weight(abstract_txt:effort in 2010) [ClassicSimilarity], result of:
            0.030640613 = score(doc=2010,freq=1.0), product of:
              0.1140025 = queryWeight, product of:
                1.4776165 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.013455841 = queryNorm
              0.2687714 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.05081223 = weight(abstract_txt:collections in 2010) [ClassicSimilarity], result of:
            0.05081223 = score(doc=2010,freq=4.0), product of:
              0.11517865 = queryWeight, product of:
                1.8190145 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.013455841 = queryNorm
              0.44116014 = fieldWeight in 2010, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.05545577 = weight(abstract_txt:relevance in 2010) [ClassicSimilarity], result of:
            0.05545577 = score(doc=2010,freq=2.0), product of:
              0.1693092 = queryWeight, product of:
                2.5465946 = boost
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.013455841 = queryNorm
              0.32754138 = fieldWeight in 2010, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.07329005 = weight(abstract_txt:test in 2010) [ClassicSimilarity], result of:
            0.07329005 = score(doc=2010,freq=3.0), product of:
              0.17812122 = queryWeight, product of:
                2.6120253 = boost
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.013455841 = queryNorm
              0.41146165 = fieldWeight in 2010, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.21594697 = weight(abstract_txt:judgments in 2010) [ClassicSimilarity], result of:
            0.21594697 = score(doc=2010,freq=4.0), product of:
              0.33261168 = queryWeight, product of:
                3.569343 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.013455841 = queryNorm
              0.6492465 = fieldWeight in 2010, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.04158921 = weight(abstract_txt:methods in 2010) [ClassicSimilarity], result of:
            0.04158921 = score(doc=2010,freq=1.0), product of:
              0.21219048 = queryWeight, product of:
                3.7713907 = boost
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.013455841 = queryNorm
              0.19599943 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
        0.36 = coord(9/25)
    
  5. Yang, Y.; Wilbur, J.: Using corpus statistics to remove redundant words in text categorization (1996) 0.18
    0.18349238 = sum of:
      0.18349238 = product of:
        0.57341367 = sum of:
          0.04538763 = weight(abstract_txt:precision in 4268) [ClassicSimilarity], result of:
            0.04538763 = score(doc=4268,freq=1.0), product of:
              0.105384134 = queryWeight, product of:
                1.4206666 = boost
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.013455841 = queryNorm
              0.4306875 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.0400669 = weight(abstract_txt:documents in 4268) [ClassicSimilarity], result of:
            0.0400669 = score(doc=4268,freq=2.0), product of:
              0.08811053 = queryWeight, product of:
                1.5909777 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.013455841 = queryNorm
              0.45473453 = fieldWeight in 4268, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.06870129 = weight(abstract_txt:reduce in 4268) [ClassicSimilarity], result of:
            0.06870129 = score(doc=4268,freq=1.0), product of:
              0.13892919 = queryWeight, product of:
                1.6311778 = boost
                6.3296742 = idf(docFreq=204, maxDocs=42306)
                0.013455841 = queryNorm
              0.4945058 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.3296742 = idf(docFreq=204, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.09363809 = weight(abstract_txt:reduced in 4268) [ClassicSimilarity], result of:
            0.09363809 = score(doc=4268,freq=1.0), product of:
              0.17078587 = queryWeight, product of:
                1.80855 = boost
                7.0179553 = idf(docFreq=102, maxDocs=42306)
                0.013455841 = queryNorm
              0.54827774 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.0179553 = idf(docFreq=102, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.042343523 = weight(abstract_txt:collections in 4268) [ClassicSimilarity], result of:
            0.042343523 = score(doc=4268,freq=1.0), product of:
              0.11517865 = queryWeight, product of:
                1.8190145 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.013455841 = queryNorm
              0.36763343 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.11472617 = weight(abstract_txt:stop in 4268) [ClassicSimilarity], result of:
            0.11472617 = score(doc=4268,freq=1.0), product of:
              0.19555034 = queryWeight, product of:
                1.9352355 = boost
                7.5095496 = idf(docFreq=62, maxDocs=42306)
                0.013455841 = queryNorm
              0.5866836 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.5095496 = idf(docFreq=62, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.07052339 = weight(abstract_txt:test in 4268) [ClassicSimilarity], result of:
            0.07052339 = score(doc=4268,freq=1.0), product of:
              0.17812122 = queryWeight, product of:
                2.6120253 = boost
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.013455841 = queryNorm
              0.39592916 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.09802671 = weight(abstract_txt:methods in 4268) [ClassicSimilarity], result of:
            0.09802671 = score(doc=4268,freq=2.0), product of:
              0.21219048 = queryWeight, product of:
                3.7713907 = boost
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.013455841 = queryNorm
              0.46197507 = fieldWeight in 4268, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.181321 = idf(docFreq=1756, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
        0.32 = coord(8/25)