Document (#36199)

Author
Pal, S.
Mitra, M.
Kamps, J.
Title
Evaluation effort, reliability and reusability in XML retrieval
Source
Journal of the American Society for Information Science and Technology. 62(2011) no.2, S.375-394
Year
2011
Abstract
The Initiative for the Evaluation of XML retrieval (INEX) provides a TREC-like platform for evaluating content-oriented XML retrieval systems. Since 2007, INEX has been using a set of precision-recall based metrics for its ad hoc tasks. The authors investigate the reliability and robustness of these focused retrieval measures, and of the INEX pooling method. They explore four specific questions: How reliable are the metrics when assessments are incomplete, or when query sets are small? What is the minimum pool/query-set size that can be used to reliably evaluate systems? Can the INEX collections be used to fairly evaluate "new" systems that did not participate in the pooling process? And, for a fixed amount of assessment effort, would this effort be better spent in thoroughly judging a few queries, or in judging many queries relatively superficially? The authors' findings validate properties of precision-recall-based metrics observed in document retrieval settings. Early precision measures are found to be more error-prone and less stable under incomplete judgments and small topic-set sizes. They also find that system rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced, and confirm that the INEX collections remain usable when evaluating nonparticipating systems. Finally, they observe that for a fixed amount of effort, judging shallow pools for many queries is better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. This result confirms the effectiveness of pooling methods.
Theme
Retrievalstudien
Object
INEX

Similar documents (author)

  1. Kamps, T.: Diagram design : a constructive theory (1999) 2.27
    2.2699265 = sum of:
      2.2699265 = product of:
        4.539853 = sum of:
          4.539853 = weight(author_txt:kamps in 2973) [ClassicSimilarity], result of:
            4.539853 = score(doc=2973,freq=1.0), product of:
              0.73611116 = queryWeight, product of:
                1.0428506 = boost
                9.867756 = idf(docFreq=5, maxDocs=42596)
                0.07153242 = queryNorm
              6.1673474 = fieldWeight in 2973, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.867756 = idf(docFreq=5, maxDocs=42596)
                0.625 = fieldNorm(doc=2973)
        0.5 = coord(1/2)
    
  2. Mitra, S.: Who pays for the Internet? (1996) 2.00
    2.0014536 = sum of:
      2.0014536 = product of:
        4.0029073 = sum of:
          4.0029073 = weight(author_txt:mitra in 599) [ClassicSimilarity], result of:
            4.0029073 = score(doc=599,freq=1.0), product of:
              0.6768606 = queryWeight, product of:
                9.462291 = idf(docFreq=8, maxDocs=42596)
                0.07153242 = queryNorm
              5.913932 = fieldWeight in 599, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.462291 = idf(docFreq=8, maxDocs=42596)
                0.625 = fieldNorm(doc=599)
        0.5 = coord(1/2)
    
  3. Kaptein, R.; Kamps, J.: Explicit extraction of topical context (2011) 1.82
    1.8159413 = sum of:
      1.8159413 = product of:
        3.6318827 = sum of:
          3.6318827 = weight(author_txt:kamps in 631) [ClassicSimilarity], result of:
            3.6318827 = score(doc=631,freq=1.0), product of:
              0.73611116 = queryWeight, product of:
                1.0428506 = boost
                9.867756 = idf(docFreq=5, maxDocs=42596)
                0.07153242 = queryNorm
              4.933878 = fieldWeight in 631, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.867756 = idf(docFreq=5, maxDocs=42596)
                0.5 = fieldNorm(doc=631)
        0.5 = coord(1/2)
    
  4. Rokach, L.; Mitra, P.: Parsimonious citer-based measures : the artificial intelligence domain as a case study (2013) 1.60
    1.6011629 = sum of:
      1.6011629 = product of:
        3.2023258 = sum of:
          3.2023258 = weight(author_txt:mitra in 1213) [ClassicSimilarity], result of:
            3.2023258 = score(doc=1213,freq=1.0), product of:
              0.6768606 = queryWeight, product of:
                9.462291 = idf(docFreq=8, maxDocs=42596)
                0.07153242 = queryNorm
              4.7311454 = fieldWeight in 1213, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.462291 = idf(docFreq=8, maxDocs=42596)
                0.5 = fieldNorm(doc=1213)
        0.5 = coord(1/2)
    
  5. Kamps, J.; Rijke, M. de; Sigurbjörnsson, B.: Length normalization in XML retrieval (2004) 1.36
    1.361956 = sum of:
      1.361956 = product of:
        2.723912 = sum of:
          2.723912 = weight(author_txt:kamps in 5107) [ClassicSimilarity], result of:
            2.723912 = score(doc=5107,freq=1.0), product of:
              0.73611116 = queryWeight, product of:
                1.0428506 = boost
                9.867756 = idf(docFreq=5, maxDocs=42596)
                0.07153242 = queryNorm
              3.7004085 = fieldWeight in 5107, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.867756 = idf(docFreq=5, maxDocs=42596)
                0.375 = fieldNorm(doc=5107)
        0.5 = coord(1/2)
    

Similar documents (content)

  1. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.37
    0.3654526 = sum of:
      0.3654526 = product of:
        0.91363144 = sum of:
          0.02460514 = weight(abstract_txt:evaluating in 690) [ClassicSimilarity], result of:
            0.02460514 = score(doc=690,freq=1.0), product of:
              0.08966101 = queryWeight, product of:
                1.0316298 = boost
                5.8543806 = idf(docFreq=331, maxDocs=42596)
                0.014845635 = queryNorm
              0.27442408 = fieldWeight in 690, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.8543806 = idf(docFreq=331, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.00843075 = weight(abstract_txt:that in 690) [ClassicSimilarity], result of:
            0.00843075 = score(doc=690,freq=4.0), product of:
              0.037536796 = queryWeight, product of:
                1.0554087 = boost
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.014845635 = queryNorm
              0.22459963 = fieldWeight in 690, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.02200732 = weight(abstract_txt:many in 690) [ClassicSimilarity], result of:
            0.02200732 = score(doc=690,freq=3.0), product of:
              0.06606242 = queryWeight, product of:
                1.0845385 = boost
                4.1030874 = idf(docFreq=1912, maxDocs=42596)
                0.014845635 = queryNorm
              0.33312917 = fieldWeight in 690, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.1030874 = idf(docFreq=1912, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.013799659 = weight(abstract_txt:systems in 690) [ClassicSimilarity], result of:
            0.013799659 = score(doc=690,freq=2.0), product of:
              0.06097689 = queryWeight, product of:
                1.2031498 = boost
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.014845635 = queryNorm
              0.22630966 = fieldWeight in 690, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.057331868 = weight(abstract_txt:reliability in 690) [ClassicSimilarity], result of:
            0.057331868 = score(doc=690,freq=2.0), product of:
              0.12507594 = queryWeight, product of:
                1.2184535 = boost
                6.9145837 = idf(docFreq=114, maxDocs=42596)
                0.014845635 = queryNorm
              0.45837647 = fieldWeight in 690, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.9145837 = idf(docFreq=114, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.012729355 = weight(abstract_txt:retrieval in 690) [ClassicSimilarity], result of:
            0.012729355 = score(doc=690,freq=1.0), product of:
              0.07842189 = queryWeight, product of:
                1.5254948 = boost
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.014845635 = queryNorm
              0.1623189 = fieldWeight in 690, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.031477682 = weight(abstract_txt:when in 690) [ClassicSimilarity], result of:
            0.031477682 = score(doc=690,freq=2.0), product of:
              0.113821656 = queryWeight, product of:
                1.8378265 = boost
                4.171782 = idf(docFreq=1785, maxDocs=42596)
                0.014845635 = queryNorm
              0.27655268 = fieldWeight in 690, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.171782 = idf(docFreq=1785, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.1461286 = weight(abstract_txt:pooling in 690) [ClassicSimilarity], result of:
            0.1461286 = score(doc=690,freq=1.0), product of:
              0.33659443 = queryWeight, product of:
                2.448055 = boost
                9.2616205 = idf(docFreq=10, maxDocs=42596)
                0.014845635 = queryNorm
              0.43413848 = fieldWeight in 690, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.2616205 = idf(docFreq=10, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.057592522 = weight(abstract_txt:effort in 690) [ClassicSimilarity], result of:
            0.057592522 = score(doc=690,freq=1.0), product of:
              0.2145246 = queryWeight, product of:
                2.5230777 = boost
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.014845635 = queryNorm
              0.26846582 = fieldWeight in 690, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
          0.53952855 = weight(abstract_txt:judging in 690) [ClassicSimilarity], result of:
            0.53952855 = score(doc=690,freq=10.0), product of:
              0.44249597 = queryWeight, product of:
                3.6236541 = boost
                8.225529 = idf(docFreq=30, maxDocs=42596)
                0.014845635 = queryNorm
              1.2192847 = fieldWeight in 690, product of:
                3.1622777 = tf(freq=10.0), with freq of:
                  10.0 = termFreq=10.0
                8.225529 = idf(docFreq=30, maxDocs=42596)
                0.046875 = fieldNorm(doc=690)
        0.4 = coord(10/25)
    
  2. Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019) 0.34
    0.34309605 = sum of:
      0.34309605 = product of:
        0.71478343 = sum of:
          0.030758766 = weight(abstract_txt:recall in 272) [ClassicSimilarity], result of:
            0.030758766 = score(doc=272,freq=1.0), product of:
              0.08588945 = queryWeight, product of:
                1.0096991 = boost
                5.7299266 = idf(docFreq=375, maxDocs=42596)
                0.014845635 = queryNorm
              0.3581204 = fieldWeight in 272, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7299266 = idf(docFreq=375, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.04521875 = weight(abstract_txt:assessment in 272) [ClassicSimilarity], result of:
            0.04521875 = score(doc=272,freq=2.0), product of:
              0.08813795 = queryWeight, product of:
                1.0228301 = boost
                5.804444 = idf(docFreq=348, maxDocs=42596)
                0.014845635 = queryNorm
              0.5130452 = fieldWeight in 272, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.804444 = idf(docFreq=348, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.007948588 = weight(abstract_txt:that in 272) [ClassicSimilarity], result of:
            0.007948588 = score(doc=272,freq=2.0), product of:
              0.037536796 = queryWeight, product of:
                1.0554087 = boost
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.014845635 = queryNorm
              0.21175455 = fieldWeight in 272, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.016941244 = weight(abstract_txt:many in 272) [ClassicSimilarity], result of:
            0.016941244 = score(doc=272,freq=1.0), product of:
              0.06606242 = queryWeight, product of:
                1.0845385 = boost
                4.1030874 = idf(docFreq=1912, maxDocs=42596)
                0.014845635 = queryNorm
              0.25644296 = fieldWeight in 272, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.1030874 = idf(docFreq=1912, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.013010443 = weight(abstract_txt:systems in 272) [ClassicSimilarity], result of:
            0.013010443 = score(doc=272,freq=1.0), product of:
              0.06097689 = queryWeight, product of:
                1.2031498 = boost
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.014845635 = queryNorm
              0.21336679 = fieldWeight in 272, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.11820819 = weight(abstract_txt:pool in 272) [ClassicSimilarity], result of:
            0.11820819 = score(doc=272,freq=2.0), product of:
              0.16725622 = queryWeight, product of:
                1.4090072 = boost
                7.995954 = idf(docFreq=38, maxDocs=42596)
                0.014845635 = queryNorm
              0.70674914 = fieldWeight in 272, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.995954 = idf(docFreq=38, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.058257222 = weight(abstract_txt:precision in 272) [ClassicSimilarity], result of:
            0.058257222 = score(doc=272,freq=2.0), product of:
              0.1194577 = queryWeight, product of:
                1.4583936 = boost
                5.517478 = idf(docFreq=464, maxDocs=42596)
                0.014845635 = queryNorm
              0.48768076 = fieldWeight in 272, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.517478 = idf(docFreq=464, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.016972473 = weight(abstract_txt:retrieval in 272) [ClassicSimilarity], result of:
            0.016972473 = score(doc=272,freq=1.0), product of:
              0.07842189 = queryWeight, product of:
                1.5254948 = boost
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.014845635 = queryNorm
              0.2164252 = fieldWeight in 272, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.07435469 = weight(abstract_txt:metrics in 272) [ClassicSimilarity], result of:
            0.07435469 = score(doc=272,freq=1.0), product of:
              0.17709103 = queryWeight, product of:
                1.7756852 = boost
                6.717873 = idf(docFreq=139, maxDocs=42596)
                0.014845635 = queryNorm
              0.41986707 = fieldWeight in 272, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.717873 = idf(docFreq=139, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.029677445 = weight(abstract_txt:when in 272) [ClassicSimilarity], result of:
            0.029677445 = score(doc=272,freq=1.0), product of:
              0.113821656 = queryWeight, product of:
                1.8378265 = boost
                4.171782 = idf(docFreq=1785, maxDocs=42596)
                0.014845635 = queryNorm
              0.26073638 = fieldWeight in 272, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.171782 = idf(docFreq=1785, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.19483812 = weight(abstract_txt:pooling in 272) [ClassicSimilarity], result of:
            0.19483812 = score(doc=272,freq=1.0), product of:
              0.33659443 = queryWeight, product of:
                2.448055 = boost
                9.2616205 = idf(docFreq=10, maxDocs=42596)
                0.014845635 = queryNorm
              0.5788513 = fieldWeight in 272, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.2616205 = idf(docFreq=10, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
          0.108597495 = weight(abstract_txt:effort in 272) [ClassicSimilarity], result of:
            0.108597495 = score(doc=272,freq=2.0), product of:
              0.2145246 = queryWeight, product of:
                2.5230777 = boost
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.014845635 = queryNorm
              0.506224 = fieldWeight in 272, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.0625 = fieldNorm(doc=272)
        0.48 = coord(12/25)
    
  3. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.34
    0.3429827 = sum of:
      0.3429827 = product of:
        0.85745674 = sum of:
          0.031974487 = weight(abstract_txt:assessment in 696) [ClassicSimilarity], result of:
            0.031974487 = score(doc=696,freq=1.0), product of:
              0.08813795 = queryWeight, product of:
                1.0228301 = boost
                5.804444 = idf(docFreq=348, maxDocs=42596)
                0.014845635 = queryNorm
              0.36277774 = fieldWeight in 696, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.804444 = idf(docFreq=348, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.032806855 = weight(abstract_txt:evaluating in 696) [ClassicSimilarity], result of:
            0.032806855 = score(doc=696,freq=1.0), product of:
              0.08966101 = queryWeight, product of:
                1.0316298 = boost
                5.8543806 = idf(docFreq=331, maxDocs=42596)
                0.014845635 = queryNorm
              0.3658988 = fieldWeight in 696, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.8543806 = idf(docFreq=331, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.007948588 = weight(abstract_txt:that in 696) [ClassicSimilarity], result of:
            0.007948588 = score(doc=696,freq=2.0), product of:
              0.037536796 = queryWeight, product of:
                1.0554087 = boost
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.014845635 = queryNorm
              0.21175455 = fieldWeight in 696, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.013010443 = weight(abstract_txt:systems in 696) [ClassicSimilarity], result of:
            0.013010443 = score(doc=696,freq=1.0), product of:
              0.06097689 = queryWeight, product of:
                1.2031498 = boost
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.014845635 = queryNorm
              0.21336679 = fieldWeight in 696, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.11820819 = weight(abstract_txt:pool in 696) [ClassicSimilarity], result of:
            0.11820819 = score(doc=696,freq=2.0), product of:
              0.16725622 = queryWeight, product of:
                1.4090072 = boost
                7.995954 = idf(docFreq=38, maxDocs=42596)
                0.014845635 = queryNorm
              0.70674914 = fieldWeight in 696, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.995954 = idf(docFreq=38, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.016972473 = weight(abstract_txt:retrieval in 696) [ClassicSimilarity], result of:
            0.016972473 = score(doc=696,freq=1.0), product of:
              0.07842189 = queryWeight, product of:
                1.5254948 = boost
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.014845635 = queryNorm
              0.2164252 = fieldWeight in 696, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.043194912 = weight(abstract_txt:queries in 696) [ClassicSimilarity], result of:
            0.043194912 = score(doc=696,freq=1.0), product of:
              0.13570389 = queryWeight, product of:
                1.7948709 = boost
                5.092843 = idf(docFreq=710, maxDocs=42596)
                0.014845635 = queryNorm
              0.3183027 = fieldWeight in 696, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.092843 = idf(docFreq=710, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.19483812 = weight(abstract_txt:pooling in 696) [ClassicSimilarity], result of:
            0.19483812 = score(doc=696,freq=1.0), product of:
              0.33659443 = queryWeight, product of:
                2.448055 = boost
                9.2616205 = idf(docFreq=10, maxDocs=42596)
                0.014845635 = queryNorm
              0.5788513 = fieldWeight in 696, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.2616205 = idf(docFreq=10, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.07679003 = weight(abstract_txt:effort in 696) [ClassicSimilarity], result of:
            0.07679003 = score(doc=696,freq=1.0), product of:
              0.2145246 = queryWeight, product of:
                2.5230777 = boost
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.014845635 = queryNorm
              0.3579544 = fieldWeight in 696, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
          0.32171267 = weight(abstract_txt:judging in 696) [ClassicSimilarity], result of:
            0.32171267 = score(doc=696,freq=2.0), product of:
              0.44249597 = queryWeight, product of:
                3.6236541 = boost
                8.225529 = idf(docFreq=30, maxDocs=42596)
                0.014845635 = queryNorm
              0.7270409 = fieldWeight in 696, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.225529 = idf(docFreq=30, maxDocs=42596)
                0.0625 = fieldNorm(doc=696)
        0.4 = coord(10/25)
    
  4. Klein, S.T.: Processing queries with metrical constraints in XML-based IR systems (2008) 0.21
    0.20871098 = sum of:
      0.20871098 = product of:
        0.7453964 = sum of:
          0.04613815 = weight(abstract_txt:recall in 2522) [ClassicSimilarity], result of:
            0.04613815 = score(doc=2522,freq=1.0), product of:
              0.08588945 = queryWeight, product of:
                1.0096991 = boost
                5.7299266 = idf(docFreq=375, maxDocs=42596)
                0.014845635 = queryNorm
              0.5371806 = fieldWeight in 2522, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7299266 = idf(docFreq=375, maxDocs=42596)
                0.09375 = fieldNorm(doc=2522)
          0.025411865 = weight(abstract_txt:many in 2522) [ClassicSimilarity], result of:
            0.025411865 = score(doc=2522,freq=1.0), product of:
              0.06606242 = queryWeight, product of:
                1.0845385 = boost
                4.1030874 = idf(docFreq=1912, maxDocs=42596)
                0.014845635 = queryNorm
              0.38466445 = fieldWeight in 2522, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.1030874 = idf(docFreq=1912, maxDocs=42596)
                0.09375 = fieldNorm(doc=2522)
          0.019515663 = weight(abstract_txt:systems in 2522) [ClassicSimilarity], result of:
            0.019515663 = score(doc=2522,freq=1.0), product of:
              0.06097689 = queryWeight, product of:
                1.2031498 = boost
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.014845635 = queryNorm
              0.32005018 = fieldWeight in 2522, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.09375 = fieldNorm(doc=2522)
          0.06179111 = weight(abstract_txt:precision in 2522) [ClassicSimilarity], result of:
            0.06179111 = score(doc=2522,freq=1.0), product of:
              0.1194577 = queryWeight, product of:
                1.4583936 = boost
                5.517478 = idf(docFreq=464, maxDocs=42596)
                0.014845635 = queryNorm
              0.51726353 = fieldWeight in 2522, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.517478 = idf(docFreq=464, maxDocs=42596)
                0.09375 = fieldNorm(doc=2522)
          0.02545871 = weight(abstract_txt:retrieval in 2522) [ClassicSimilarity], result of:
            0.02545871 = score(doc=2522,freq=1.0), product of:
              0.07842189 = queryWeight, product of:
                1.5254948 = boost
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.014845635 = queryNorm
              0.3246378 = fieldWeight in 2522, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.09375 = fieldNorm(doc=2522)
          0.06479237 = weight(abstract_txt:queries in 2522) [ClassicSimilarity], result of:
            0.06479237 = score(doc=2522,freq=1.0), product of:
              0.13570389 = queryWeight, product of:
                1.7948709 = boost
                5.092843 = idf(docFreq=710, maxDocs=42596)
                0.014845635 = queryNorm
              0.47745404 = fieldWeight in 2522, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.092843 = idf(docFreq=710, maxDocs=42596)
                0.09375 = fieldNorm(doc=2522)
          0.5022885 = weight(abstract_txt:inex in 2522) [ClassicSimilarity], result of:
            0.5022885 = score(doc=2522,freq=1.0), product of:
              0.5725963 = queryWeight, product of:
                4.1220794 = boost
                9.356931 = idf(docFreq=9, maxDocs=42596)
                0.014845635 = queryNorm
              0.8772123 = fieldWeight in 2522, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.356931 = idf(docFreq=9, maxDocs=42596)
                0.09375 = fieldNorm(doc=2522)
        0.28 = coord(7/25)
    
  5. White, H.D.: Combining bibliometrics, information retrieval, and relevance theory : part 2: some implications for information science (2007) 0.17
    0.1704984 = sum of:
      0.1704984 = product of:
        0.5328075 = sum of:
          0.011620331 = weight(abstract_txt:they in 1617) [ClassicSimilarity], result of:
            0.011620331 = score(doc=1617,freq=1.0), product of:
              0.05616485 = queryWeight, product of:
                3.7832568 = idf(docFreq=2633, maxDocs=42596)
                0.014845635 = queryNorm
              0.20689686 = fieldWeight in 1617, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.7832568 = idf(docFreq=2633, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
          0.004917938 = weight(abstract_txt:that in 1617) [ClassicSimilarity], result of:
            0.004917938 = score(doc=1617,freq=1.0), product of:
              0.037536796 = queryWeight, product of:
                1.0554087 = boost
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.014845635 = queryNorm
              0.13101645 = fieldWeight in 1617, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.3957293 = idf(docFreq=10548, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
          0.016099604 = weight(abstract_txt:systems in 1617) [ClassicSimilarity], result of:
            0.016099604 = score(doc=1617,freq=2.0), product of:
              0.06097689 = queryWeight, product of:
                1.2031498 = boost
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.014845635 = queryNorm
              0.26402795 = fieldWeight in 1617, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4138687 = idf(docFreq=3810, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
          0.07177383 = weight(abstract_txt:judge in 1617) [ClassicSimilarity], result of:
            0.07177383 = score(doc=1617,freq=1.0), product of:
              0.16517054 = queryWeight, product of:
                1.4001945 = boost
                7.9459434 = idf(docFreq=40, maxDocs=42596)
                0.014845635 = queryNorm
              0.4345438 = fieldWeight in 1617, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.9459434 = idf(docFreq=40, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
          0.0148509145 = weight(abstract_txt:retrieval in 1617) [ClassicSimilarity], result of:
            0.0148509145 = score(doc=1617,freq=1.0), product of:
              0.07842189 = queryWeight, product of:
                1.5254948 = boost
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.014845635 = queryNorm
              0.18937205 = fieldWeight in 1617, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4628031 = idf(docFreq=3628, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
          0.036723964 = weight(abstract_txt:when in 1617) [ClassicSimilarity], result of:
            0.036723964 = score(doc=1617,freq=2.0), product of:
              0.113821656 = queryWeight, product of:
                1.8378265 = boost
                4.171782 = idf(docFreq=1785, maxDocs=42596)
                0.014845635 = queryNorm
              0.3226448 = fieldWeight in 1617, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.171782 = idf(docFreq=1785, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
          0.17777139 = weight(abstract_txt:effort in 1617) [ClassicSimilarity], result of:
            0.17777139 = score(doc=1617,freq=7.0), product of:
              0.2145246 = queryWeight, product of:
                2.5230777 = boost
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.014845635 = queryNorm
              0.82867604 = fieldWeight in 1617, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                5.7272706 = idf(docFreq=376, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
          0.19904956 = weight(abstract_txt:judging in 1617) [ClassicSimilarity], result of:
            0.19904956 = score(doc=1617,freq=1.0), product of:
              0.44249597 = queryWeight, product of:
                3.6236541 = boost
                8.225529 = idf(docFreq=30, maxDocs=42596)
                0.014845635 = queryNorm
              0.4498336 = fieldWeight in 1617, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.225529 = idf(docFreq=30, maxDocs=42596)
                0.0546875 = fieldNorm(doc=1617)
        0.32 = coord(8/25)