Document (#36199)

Author
Pal, S.
Mitra, M.
Kamps, J.
Title
Evaluation effort, reliability and reusability in XML retrieval
Source
Journal of the American Society for Information Science and Technology. 62(2011) no.2, S.375-394
Year
2011
Abstract
The Initiative for the Evaluation of XML retrieval (INEX) provides a TREC-like platform for evaluating content-oriented XML retrieval systems. Since 2007, INEX has been using a set of precision-recall based metrics for its ad hoc tasks. The authors investigate the reliability and robustness of these focused retrieval measures, and of the INEX pooling method. They explore four specific questions: How reliable are the metrics when assessments are incomplete, or when query sets are small? What is the minimum pool/query-set size that can be used to reliably evaluate systems? Can the INEX collections be used to fairly evaluate "new" systems that did not participate in the pooling process? And, for a fixed amount of assessment effort, would this effort be better spent in thoroughly judging a few queries, or in judging many queries relatively superficially? The authors' findings validate properties of precision-recall-based metrics observed in document retrieval settings. Early precision measures are found to be more error-prone and less stable under incomplete judgments and small topic-set sizes. They also find that system rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced, and confirm that the INEX collections remain usable when evaluating nonparticipating systems. Finally, they observe that for a fixed amount of effort, judging shallow pools for many queries is better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. This result confirms the effectiveness of pooling methods.
Theme
Retrievalstudien
Object
INEX

Similar documents (author)

  1. Kamps, T.: Diagram design : a constructive theory (1999) 2.27
    2.2684171 = sum of:
      2.2684171 = product of:
        4.5368342 = sum of:
          4.5368342 = weight(author_txt:kamps in 2973) [ClassicSimilarity], result of:
            4.5368342 = score(doc=2973,freq=1.0), product of:
              0.73613125 = queryWeight, product of:
                1.0428816 = boost
                9.860925 = idf(docFreq=5, maxDocs=42306)
                0.071581796 = queryNorm
              6.163078 = fieldWeight in 2973, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.860925 = idf(docFreq=5, maxDocs=42306)
                0.625 = fieldNorm(doc=2973)
        0.5 = coord(1/2)
    
  2. Mitra, S.: Who pays for the Internet? (1996) 2.00
    1.9999442 = sum of:
      1.9999442 = product of:
        3.9998884 = sum of:
          3.9998884 = weight(author_txt:mitra in 599) [ClassicSimilarity], result of:
            3.9998884 = score(doc=599,freq=1.0), product of:
              0.67683876 = queryWeight, product of:
                9.45546 = idf(docFreq=8, maxDocs=42306)
                0.071581796 = queryNorm
              5.9096622 = fieldWeight in 599, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.45546 = idf(docFreq=8, maxDocs=42306)
                0.625 = fieldNorm(doc=599)
        0.5 = coord(1/2)
    
  3. Kaptein, R.; Kamps, J.: Explicit extraction of topical context (2011) 1.81
    1.8147337 = sum of:
      1.8147337 = product of:
        3.6294675 = sum of:
          3.6294675 = weight(author_txt:kamps in 1631) [ClassicSimilarity], result of:
            3.6294675 = score(doc=1631,freq=1.0), product of:
              0.73613125 = queryWeight, product of:
                1.0428816 = boost
                9.860925 = idf(docFreq=5, maxDocs=42306)
                0.071581796 = queryNorm
              4.9304624 = fieldWeight in 1631, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.860925 = idf(docFreq=5, maxDocs=42306)
                0.5 = fieldNorm(doc=1631)
        0.5 = coord(1/2)
    
  4. Rokach, L.; Mitra, P.: Parsimonious citer-based measures : the artificial intelligence domain as a case study (2013) 1.60
    1.5999553 = sum of:
      1.5999553 = product of:
        3.1999106 = sum of:
          3.1999106 = weight(author_txt:mitra in 2213) [ClassicSimilarity], result of:
            3.1999106 = score(doc=2213,freq=1.0), product of:
              0.67683876 = queryWeight, product of:
                9.45546 = idf(docFreq=8, maxDocs=42306)
                0.071581796 = queryNorm
              4.72773 = fieldWeight in 2213, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.45546 = idf(docFreq=8, maxDocs=42306)
                0.5 = fieldNorm(doc=2213)
        0.5 = coord(1/2)
    
  5. Kamps, J.; Rijke, M. de; Sigurbjörnsson, B.: Length normalization in XML retrieval (2004) 1.36
    1.3610504 = sum of:
      1.3610504 = product of:
        2.7221007 = sum of:
          2.7221007 = weight(author_txt:kamps in 107) [ClassicSimilarity], result of:
            2.7221007 = score(doc=107,freq=1.0), product of:
              0.73613125 = queryWeight, product of:
                1.0428816 = boost
                9.860925 = idf(docFreq=5, maxDocs=42306)
                0.071581796 = queryNorm
              3.697847 = fieldWeight in 107, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.860925 = idf(docFreq=5, maxDocs=42306)
                0.375 = fieldNorm(doc=107)
        0.5 = coord(1/2)
    

Similar documents (content)

  1. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.37
    0.36528188 = sum of:
      0.36528188 = product of:
        0.91320467 = sum of:
          0.02467993 = weight(abstract_txt:evaluating in 2010) [ClassicSimilarity], result of:
            0.02467993 = score(doc=2010,freq=1.0), product of:
              0.089852355 = queryWeight, product of:
                1.0303599 = boost
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.014882206 = queryNorm
              0.27467206 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.0085302405 = weight(abstract_txt:that in 2010) [ClassicSimilarity], result of:
            0.0085302405 = score(doc=2010,freq=4.0), product of:
              0.037835635 = queryWeight, product of:
                1.0571698 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.014882206 = queryNorm
              0.2254552 = fieldWeight in 2010, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.022099335 = weight(abstract_txt:many in 2010) [ClassicSimilarity], result of:
            0.022099335 = score(doc=2010,freq=3.0), product of:
              0.066253625 = queryWeight, product of:
                1.0836141 = boost
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.014882206 = queryNorm
              0.33355662 = fieldWeight in 2010, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.013846105 = weight(abstract_txt:systems in 2010) [ClassicSimilarity], result of:
            0.013846105 = score(doc=2010,freq=2.0), product of:
              0.061120268 = queryWeight, product of:
                1.201799 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014882206 = queryNorm
              0.22653869 = fieldWeight in 2010, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.057617497 = weight(abstract_txt:reliability in 2010) [ClassicSimilarity], result of:
            0.057617497 = score(doc=2010,freq=2.0), product of:
              0.12550463 = queryWeight, product of:
                1.2177386 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.014882206 = queryNorm
              0.45908663 = fieldWeight in 2010, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.012731336 = weight(abstract_txt:retrieval in 2010) [ClassicSimilarity], result of:
            0.012731336 = score(doc=2010,freq=1.0), product of:
              0.078438535 = queryWeight, product of:
                1.5221553 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014882206 = queryNorm
              0.1623097 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.031678554 = weight(abstract_txt:when in 2010) [ClassicSimilarity], result of:
            0.031678554 = score(doc=2010,freq=2.0), product of:
              0.11431777 = queryWeight, product of:
                1.8376004 = boost
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.014882206 = queryNorm
              0.27710962 = fieldWeight in 2010, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.14585294 = weight(abstract_txt:pooling in 2010) [ClassicSimilarity], result of:
            0.14585294 = score(doc=2010,freq=1.0), product of:
              0.33620748 = queryWeight, product of:
                2.4410324 = boost
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.014882206 = queryNorm
              0.43381825 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.057808243 = weight(abstract_txt:effort in 2010) [ClassicSimilarity], result of:
            0.057808243 = score(doc=2010,freq=1.0), product of:
              0.2150833 = queryWeight, product of:
                2.5205631 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014882206 = queryNorm
              0.2687714 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.53836054 = weight(abstract_txt:judging in 2010) [ClassicSimilarity], result of:
            0.53836054 = score(doc=2010,freq=10.0), product of:
              0.44190505 = queryWeight, product of:
                3.612923 = boost
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.014882206 = queryNorm
              1.218272 = fieldWeight in 2010, product of:
                3.1622777 = tf(freq=10.0), with freq of:
                  10.0 = termFreq=10.0
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
        0.4 = coord(10/25)
    
  2. Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019) 0.34
    0.34427357 = sum of:
      0.34427357 = product of:
        0.71723664 = sum of:
          0.030658858 = weight(abstract_txt:recall in 1593) [ClassicSimilarity], result of:
            0.030658858 = score(doc=1593,freq=1.0), product of:
              0.085712664 = queryWeight, product of:
                1.0063446 = boost
                5.723095 = idf(docFreq=375, maxDocs=42306)
                0.014882206 = queryNorm
              0.35769343 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.723095 = idf(docFreq=375, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.045343347 = weight(abstract_txt:assessment in 1593) [ClassicSimilarity], result of:
            0.045343347 = score(doc=1593,freq=2.0), product of:
              0.088309355 = queryWeight, product of:
                1.0214746 = boost
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.014882206 = queryNorm
              0.5134603 = fieldWeight in 1593, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.008042389 = weight(abstract_txt:that in 1593) [ClassicSimilarity], result of:
            0.008042389 = score(doc=1593,freq=2.0), product of:
              0.037835635 = queryWeight, product of:
                1.0571698 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.014882206 = queryNorm
              0.2125612 = fieldWeight in 1593, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.017012076 = weight(abstract_txt:many in 1593) [ClassicSimilarity], result of:
            0.017012076 = score(doc=1593,freq=1.0), product of:
              0.066253625 = queryWeight, product of:
                1.0836141 = boost
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.014882206 = queryNorm
              0.256772 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.013054233 = weight(abstract_txt:systems in 1593) [ClassicSimilarity], result of:
            0.013054233 = score(doc=1593,freq=1.0), product of:
              0.061120268 = queryWeight, product of:
                1.201799 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014882206 = queryNorm
              0.21358272 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.12029079 = weight(abstract_txt:pool in 1593) [ClassicSimilarity], result of:
            0.12029079 = score(doc=1593,freq=2.0), product of:
              0.16923335 = queryWeight, product of:
                1.4140579 = boost
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.014882206 = queryNorm
              0.7107984 = fieldWeight in 1593, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.058128074 = weight(abstract_txt:precision in 1593) [ClassicSimilarity], result of:
            0.058128074 = score(doc=1593,freq=2.0), product of:
              0.119294025 = queryWeight, product of:
                1.4540495 = boost
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.014882206 = queryNorm
              0.48726726 = fieldWeight in 1593, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.016975114 = weight(abstract_txt:retrieval in 1593) [ClassicSimilarity], result of:
            0.016975114 = score(doc=1593,freq=1.0), product of:
              0.078438535 = queryWeight, product of:
                1.5221553 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014882206 = queryNorm
              0.21641295 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.0743901 = weight(abstract_txt:metrics in 1593) [ClassicSimilarity], result of:
            0.0743901 = score(doc=1593,freq=1.0), product of:
              0.17716646 = queryWeight, product of:
                1.7719872 = boost
                6.71821 = idf(docFreq=138, maxDocs=42306)
                0.014882206 = queryNorm
              0.41988814 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.71821 = idf(docFreq=138, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.029866828 = weight(abstract_txt:when in 1593) [ClassicSimilarity], result of:
            0.029866828 = score(doc=1593,freq=1.0), product of:
              0.11431777 = queryWeight, product of:
                1.8376004 = boost
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.014882206 = queryNorm
              0.26126146 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.19447058 = weight(abstract_txt:pooling in 1593) [ClassicSimilarity], result of:
            0.19447058 = score(doc=1593,freq=1.0), product of:
              0.33620748 = queryWeight, product of:
                2.4410324 = boost
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.014882206 = queryNorm
              0.57842433 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.10900427 = weight(abstract_txt:effort in 1593) [ClassicSimilarity], result of:
            0.10900427 = score(doc=1593,freq=2.0), product of:
              0.2150833 = queryWeight, product of:
                2.5205631 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014882206 = queryNorm
              0.50680023 = fieldWeight in 1593, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
        0.48 = coord(12/25)
    
  3. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.34
    0.34365878 = sum of:
      0.34365878 = product of:
        0.85914695 = sum of:
          0.032062586 = weight(abstract_txt:assessment in 2016) [ClassicSimilarity], result of:
            0.032062586 = score(doc=2016,freq=1.0), product of:
              0.088309355 = queryWeight, product of:
                1.0214746 = boost
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.014882206 = queryNorm
              0.36307123 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.8091397 = idf(docFreq=344, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.032906577 = weight(abstract_txt:evaluating in 2016) [ClassicSimilarity], result of:
            0.032906577 = score(doc=2016,freq=1.0), product of:
              0.089852355 = queryWeight, product of:
                1.0303599 = boost
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.014882206 = queryNorm
              0.36622941 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.008042389 = weight(abstract_txt:that in 2016) [ClassicSimilarity], result of:
            0.008042389 = score(doc=2016,freq=2.0), product of:
              0.037835635 = queryWeight, product of:
                1.0571698 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.014882206 = queryNorm
              0.2125612 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.013054233 = weight(abstract_txt:systems in 2016) [ClassicSimilarity], result of:
            0.013054233 = score(doc=2016,freq=1.0), product of:
              0.061120268 = queryWeight, product of:
                1.201799 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014882206 = queryNorm
              0.21358272 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.12029079 = weight(abstract_txt:pool in 2016) [ClassicSimilarity], result of:
            0.12029079 = score(doc=2016,freq=2.0), product of:
              0.16923335 = queryWeight, product of:
                1.4140579 = boost
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.014882206 = queryNorm
              0.7107984 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.016975114 = weight(abstract_txt:retrieval in 2016) [ClassicSimilarity], result of:
            0.016975114 = score(doc=2016,freq=1.0), product of:
              0.078438535 = queryWeight, product of:
                1.5221553 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014882206 = queryNorm
              0.21641295 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.043250807 = weight(abstract_txt:queries in 2016) [ClassicSimilarity], result of:
            0.043250807 = score(doc=2016,freq=1.0), product of:
              0.13583566 = queryWeight, product of:
                1.7916209 = boost
                5.094486 = idf(docFreq=704, maxDocs=42306)
                0.014882206 = queryNorm
              0.3184054 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.094486 = idf(docFreq=704, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.19447058 = weight(abstract_txt:pooling in 2016) [ClassicSimilarity], result of:
            0.19447058 = score(doc=2016,freq=1.0), product of:
              0.33620748 = queryWeight, product of:
                2.4410324 = boost
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.014882206 = queryNorm
              0.57842433 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.254789 = idf(docFreq=10, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.07707766 = weight(abstract_txt:effort in 2016) [ClassicSimilarity], result of:
            0.07707766 = score(doc=2016,freq=1.0), product of:
              0.2150833 = queryWeight, product of:
                2.5205631 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014882206 = queryNorm
              0.3583619 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.32101616 = weight(abstract_txt:judging in 2016) [ClassicSimilarity], result of:
            0.32101616 = score(doc=2016,freq=2.0), product of:
              0.44190505 = queryWeight, product of:
                3.612923 = boost
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.014882206 = queryNorm
              0.726437 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
        0.4 = coord(10/25)
    
  4. Klein, S.T.: Processing queries with metrical constraints in XML-based IR systems (2008) 0.21
    0.20844126 = sum of:
      0.20844126 = product of:
        0.74443305 = sum of:
          0.045988284 = weight(abstract_txt:recall in 3343) [ClassicSimilarity], result of:
            0.045988284 = score(doc=3343,freq=1.0), product of:
              0.085712664 = queryWeight, product of:
                1.0063446 = boost
                5.723095 = idf(docFreq=375, maxDocs=42306)
                0.014882206 = queryNorm
              0.53654015 = fieldWeight in 3343, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.723095 = idf(docFreq=375, maxDocs=42306)
                0.09375 = fieldNorm(doc=3343)
          0.025518114 = weight(abstract_txt:many in 3343) [ClassicSimilarity], result of:
            0.025518114 = score(doc=3343,freq=1.0), product of:
              0.066253625 = queryWeight, product of:
                1.0836141 = boost
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.014882206 = queryNorm
              0.385158 = fieldWeight in 3343, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.09375 = fieldNorm(doc=3343)
          0.01958135 = weight(abstract_txt:systems in 3343) [ClassicSimilarity], result of:
            0.01958135 = score(doc=3343,freq=1.0), product of:
              0.061120268 = queryWeight, product of:
                1.201799 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014882206 = queryNorm
              0.32037407 = fieldWeight in 3343, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.09375 = fieldNorm(doc=3343)
          0.061654128 = weight(abstract_txt:precision in 3343) [ClassicSimilarity], result of:
            0.061654128 = score(doc=3343,freq=1.0), product of:
              0.119294025 = queryWeight, product of:
                1.4540495 = boost
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.014882206 = queryNorm
              0.51682496 = fieldWeight in 3343, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.5127997 = idf(docFreq=463, maxDocs=42306)
                0.09375 = fieldNorm(doc=3343)
          0.025462672 = weight(abstract_txt:retrieval in 3343) [ClassicSimilarity], result of:
            0.025462672 = score(doc=3343,freq=1.0), product of:
              0.078438535 = queryWeight, product of:
                1.5221553 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014882206 = queryNorm
              0.3246194 = fieldWeight in 3343, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.09375 = fieldNorm(doc=3343)
          0.06487621 = weight(abstract_txt:queries in 3343) [ClassicSimilarity], result of:
            0.06487621 = score(doc=3343,freq=1.0), product of:
              0.13583566 = queryWeight, product of:
                1.7916209 = boost
                5.094486 = idf(docFreq=704, maxDocs=42306)
                0.014882206 = queryNorm
              0.47760808 = fieldWeight in 3343, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.094486 = idf(docFreq=704, maxDocs=42306)
                0.09375 = fieldNorm(doc=3343)
          0.5013523 = weight(abstract_txt:inex in 3343) [ClassicSimilarity], result of:
            0.5013523 = score(doc=3343,freq=1.0), product of:
              0.5719466 = queryWeight, product of:
                4.1102853 = boost
                9.3501 = idf(docFreq=9, maxDocs=42306)
                0.014882206 = queryNorm
              0.87657183 = fieldWeight in 3343, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.3501 = idf(docFreq=9, maxDocs=42306)
                0.09375 = fieldNorm(doc=3343)
        0.28 = coord(7/25)
    
  5. White, H.D.: Combining bibliometrics, information retrieval, and relevance theory : part 2: some implications for information science (2007) 0.17
    0.17065862 = sum of:
      0.17065862 = product of:
        0.5333082 = sum of:
          0.011698803 = weight(abstract_txt:they in 2438) [ClassicSimilarity], result of:
            0.011698803 = score(doc=2438,freq=1.0), product of:
              0.05642354 = queryWeight, product of:
                3.7913425 = idf(docFreq=2594, maxDocs=42306)
                0.014882206 = queryNorm
              0.20733905 = fieldWeight in 2438, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.7913425 = idf(docFreq=2594, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
          0.0049759736 = weight(abstract_txt:that in 2438) [ClassicSimilarity], result of:
            0.0049759736 = score(doc=2438,freq=1.0), product of:
              0.037835635 = queryWeight, product of:
                1.0571698 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.014882206 = queryNorm
              0.13151553 = fieldWeight in 2438, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
          0.01615379 = weight(abstract_txt:systems in 2438) [ClassicSimilarity], result of:
            0.01615379 = score(doc=2438,freq=2.0), product of:
              0.061120268 = queryWeight, product of:
                1.201799 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014882206 = queryNorm
              0.26429513 = fieldWeight in 2438, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
          0.07161218 = weight(abstract_txt:judge in 2438) [ClassicSimilarity], result of:
            0.07161218 = score(doc=2438,freq=1.0), product of:
              0.16494034 = queryWeight, product of:
                1.3960072 = boost
                7.939112 = idf(docFreq=40, maxDocs=42306)
                0.014882206 = queryNorm
              0.4341702 = fieldWeight in 2438, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.939112 = idf(docFreq=40, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
          0.014853226 = weight(abstract_txt:retrieval in 2438) [ClassicSimilarity], result of:
            0.014853226 = score(doc=2438,freq=1.0), product of:
              0.078438535 = queryWeight, product of:
                1.5221553 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014882206 = queryNorm
              0.18936133 = fieldWeight in 2438, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
          0.036958314 = weight(abstract_txt:when in 2438) [ClassicSimilarity], result of:
            0.036958314 = score(doc=2438,freq=2.0), product of:
              0.11431777 = queryWeight, product of:
                1.8376004 = boost
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.014882206 = queryNorm
              0.32329458 = fieldWeight in 2438, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.1801834 = idf(docFreq=1758, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
          0.17843728 = weight(abstract_txt:effort in 2438) [ClassicSimilarity], result of:
            0.17843728 = score(doc=2438,freq=7.0), product of:
              0.2150833 = queryWeight, product of:
                2.5205631 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014882206 = queryNorm
              0.8296194 = fieldWeight in 2438, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
          0.19861864 = weight(abstract_txt:judging in 2438) [ClassicSimilarity], result of:
            0.19861864 = score(doc=2438,freq=1.0), product of:
              0.44190505 = queryWeight, product of:
                3.612923 = boost
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.014882206 = queryNorm
              0.44945997 = fieldWeight in 2438, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.0546875 = fieldNorm(doc=2438)
        0.32 = coord(8/25)