Document (#42288)

Author
Rajagopal, P.
Ravana, S.D.
Koh, Y.S.
Balakrishnan, V.
Title
Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
Source
Aslib journal of information management. 71(2019) no.1, S.2-17
Year
2019
Abstract
Purpose The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges' involvement. Then the study determines the variation in system rankings due to low effort relevance judgment in evaluating retrieval systems at different depth of evaluation. Design/methodology/approach Effort-based relevance judgments are generated using a proposed boxplot approach for simple document features, HTML features and readability features. The boxplot approach is a simple yet repeatable approach in classifying documents' effort while ensuring outlier scores do not skew the grading of the entire set of documents. Findings The retrieval systems evaluation using low effort relevance judgments has a stronger influence on shallow depth of evaluation compared to deeper depth. It is proved that difference in the system rankings is due to low effort documents and not the number of relevant documents. Originality/value Hence, it is crucial to evaluate retrieval systems at shallow depth using low effort relevance judgments.
Content
Vgl.: https://doi.org/10.1108/AJIM-04-2018-0086.
Theme
Retrievalstudien

Similar documents (author)

  1. Balakrishnan, U.: DFG-Projekt: Coli-conc : das Mapping Tool "Cocoda" (2016) 5.91
    5.9096622 = sum of:
      5.9096622 = weight(author_txt:balakrishnan in 37) [ClassicSimilarity], result of:
        5.9096622 = fieldWeight in 37, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.45546 = idf(docFreq=8, maxDocs=42306)
          0.625 = fieldNorm(doc=37)
    
  2. Balakrishnan, U.; Voß, J.: ¬The Cocoda mapping tool (2015) 4.73
    4.72773 = sum of:
      4.72773 = weight(author_txt:balakrishnan in 1124) [ClassicSimilarity], result of:
        4.72773 = fieldWeight in 1124, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.45546 = idf(docFreq=8, maxDocs=42306)
          0.5 = fieldNorm(doc=1124)
    
  3. Balakrishnan, U.; Krausz, A,; Voss, J.: Cocoda - ein Konkordanztool für bibliothekarische Klassifikationssysteme (2015) 3.55
    3.5457973 = sum of:
      3.5457973 = weight(author_txt:balakrishnan in 4031) [ClassicSimilarity], result of:
        3.5457973 = fieldWeight in 4031, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.45546 = idf(docFreq=8, maxDocs=42306)
          0.375 = fieldNorm(doc=4031)
    
  4. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 3.55
    3.5457973 = sum of:
      3.5457973 = weight(author_txt:balakrishnan in 4592) [ClassicSimilarity], result of:
        3.5457973 = fieldWeight in 4592, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.45546 = idf(docFreq=8, maxDocs=42306)
          0.375 = fieldNorm(doc=4592)
    
  5. Balakrishnan, V.; Ahmadi, K.; Ravana, S.D.: Improving retrieval relevance using users' explicit feedback (2016) 3.55
    3.5457973 = sum of:
      3.5457973 = weight(author_txt:balakrishnan in 4922) [ClassicSimilarity], result of:
        3.5457973 = fieldWeight in 4922, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          9.45546 = idf(docFreq=8, maxDocs=42306)
          0.375 = fieldNorm(doc=4922)
    

Similar documents (content)

  1. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.89
    0.88557523 = sum of:
      0.88557523 = product of:
        1.4759587 = sum of:
          0.036358293 = weight(abstract_txt:document in 4592) [ClassicSimilarity], result of:
            0.036358293 = score(doc=4592,freq=4.0), product of:
              0.06797039 = queryWeight, product of:
                1.1298289 = boost
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.014058317 = queryNorm
              0.5349137 = fieldWeight in 4592, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.021631608 = weight(abstract_txt:purpose in 4592) [ClassicSimilarity], result of:
            0.021631608 = score(doc=4592,freq=1.0), product of:
              0.07632438 = queryWeight, product of:
                1.1972489 = boost
                4.534668 = idf(docFreq=1233, maxDocs=42306)
                0.014058317 = queryNorm
              0.28341675 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.534668 = idf(docFreq=1233, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.035235662 = weight(abstract_txt:simple in 4592) [ClassicSimilarity], result of:
            0.035235662 = score(doc=4592,freq=1.0), product of:
              0.10566367 = queryWeight, product of:
                1.4086907 = boost
                5.3355193 = idf(docFreq=553, maxDocs=42306)
                0.014058317 = queryNorm
              0.33346996 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.3355193 = idf(docFreq=553, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.031569418 = weight(abstract_txt:evaluation in 4592) [ClassicSimilarity], result of:
            0.031569418 = score(doc=4592,freq=1.0), product of:
              0.11241188 = queryWeight, product of:
                1.7795268 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.014058317 = queryNorm
              0.28083703 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.018515721 = weight(abstract_txt:systems in 4592) [ClassicSimilarity], result of:
            0.018515721 = score(doc=4592,freq=1.0), product of:
              0.0866911 = queryWeight, product of:
                1.8044924 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014058317 = queryNorm
              0.21358272 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.07506032 = weight(abstract_txt:rankings in 4592) [ClassicSimilarity], result of:
            0.07506032 = score(doc=4592,freq=1.0), product of:
              0.1749354 = queryWeight, product of:
                1.812557 = boost
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.014058317 = queryNorm
              0.42907453 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.027239991 = weight(abstract_txt:retrieval in 4592) [ClassicSimilarity], result of:
            0.027239991 = score(doc=4592,freq=2.0), product of:
              0.08900384 = queryWeight, product of:
                1.8284041 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014058317 = queryNorm
              0.30605412 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.02781381 = weight(abstract_txt:using in 4592) [ClassicSimilarity], result of:
            0.02781381 = score(doc=4592,freq=2.0), product of:
              0.09024942 = queryWeight, product of:
                1.8411535 = boost
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.014058317 = queryNorm
              0.30818826 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.2069972 = weight(abstract_txt:judgment in 4592) [ClassicSimilarity], result of:
            0.2069972 = score(doc=4592,freq=5.0), product of:
              0.20118259 = queryWeight, product of:
                1.943784 = boost
                7.3622246 = idf(docFreq=72, maxDocs=42306)
                0.014058317 = queryNorm
              1.0289022 = fieldWeight in 4592, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                7.3622246 = idf(docFreq=72, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.035692953 = weight(abstract_txt:approach in 4592) [ClassicSimilarity], result of:
            0.035692953 = score(doc=4592,freq=2.0), product of:
              0.106575914 = queryWeight, product of:
                2.0007708 = boost
                3.789033 = idf(docFreq=2600, maxDocs=42306)
                0.014058317 = queryNorm
              0.33490637 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.789033 = idf(docFreq=2600, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.056027547 = weight(abstract_txt:documents in 4592) [ClassicSimilarity], result of:
            0.056027547 = score(doc=4592,freq=3.0), product of:
              0.12575002 = queryWeight, product of:
                2.1733108 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.014058317 = queryNorm
              0.445547 = fieldWeight in 4592, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.10333917 = weight(abstract_txt:depth in 4592) [ClassicSimilarity], result of:
            0.10333917 = score(doc=4592,freq=1.0), product of:
              0.27276632 = queryWeight, product of:
                3.200834 = boost
                6.061697 = idf(docFreq=267, maxDocs=42306)
                0.014058317 = queryNorm
              0.37885606 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.061697 = idf(docFreq=267, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.34457392 = weight(abstract_txt:judgments in 4592) [ClassicSimilarity], result of:
            0.34457392 = score(doc=4592,freq=5.0), product of:
              0.3560238 = queryWeight, product of:
                3.6568513 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.014058317 = queryNorm
              0.9678396 = fieldWeight in 4592, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.25911888 = weight(abstract_txt:relevance in 4592) [ClassicSimilarity], result of:
            0.25911888 = score(doc=4592,freq=7.0), product of:
              0.31714666 = queryWeight, product of:
                4.5658 = boost
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.014058317 = queryNorm
              0.81703174 = fieldWeight in 4592, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.19678423 = weight(abstract_txt:effort in 4592) [ClassicSimilarity], result of:
            0.19678423 = score(doc=4592,freq=1.0), product of:
              0.5491215 = queryWeight, product of:
                6.8122926 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014058317 = queryNorm
              0.3583619 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
        0.6 = coord(15/25)
    
  2. Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019) 0.42
    0.4234395 = sum of:
      0.4234395 = product of:
        1.0585988 = sum of:
          0.031569418 = weight(abstract_txt:evaluation in 1593) [ClassicSimilarity], result of:
            0.031569418 = score(doc=1593,freq=1.0), product of:
              0.11241188 = queryWeight, product of:
                1.7795268 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.014058317 = queryNorm
              0.28083703 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.018515721 = weight(abstract_txt:systems in 1593) [ClassicSimilarity], result of:
            0.018515721 = score(doc=1593,freq=1.0), product of:
              0.0866911 = queryWeight, product of:
                1.8044924 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014058317 = queryNorm
              0.21358272 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.10615133 = weight(abstract_txt:rankings in 1593) [ClassicSimilarity], result of:
            0.10615133 = score(doc=1593,freq=2.0), product of:
              0.1749354 = queryWeight, product of:
                1.812557 = boost
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.014058317 = queryNorm
              0.606803 = fieldWeight in 1593, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.019261584 = weight(abstract_txt:retrieval in 1593) [ClassicSimilarity], result of:
            0.019261584 = score(doc=1593,freq=1.0), product of:
              0.08900384 = queryWeight, product of:
                1.8284041 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014058317 = queryNorm
              0.21641295 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.019667335 = weight(abstract_txt:using in 1593) [ClassicSimilarity], result of:
            0.019667335 = score(doc=1593,freq=1.0), product of:
              0.09024942 = queryWeight, product of:
                1.8411535 = boost
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.014058317 = queryNorm
              0.217922 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.09257196 = weight(abstract_txt:judgment in 1593) [ClassicSimilarity], result of:
            0.09257196 = score(doc=1593,freq=1.0), product of:
              0.20118259 = queryWeight, product of:
                1.943784 = boost
                7.3622246 = idf(docFreq=72, maxDocs=42306)
                0.014058317 = queryNorm
              0.46013904 = fieldWeight in 1593, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.3622246 = idf(docFreq=72, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.056027547 = weight(abstract_txt:documents in 1593) [ClassicSimilarity], result of:
            0.056027547 = score(doc=1593,freq=3.0), product of:
              0.12575002 = queryWeight, product of:
                2.1733108 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.014058317 = queryNorm
              0.445547 = fieldWeight in 1593, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.26690578 = weight(abstract_txt:judgments in 1593) [ClassicSimilarity], result of:
            0.26690578 = score(doc=1593,freq=3.0), product of:
              0.3560238 = queryWeight, product of:
                3.6568513 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.014058317 = queryNorm
              0.7496853 = fieldWeight in 1593, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.16963314 = weight(abstract_txt:relevance in 1593) [ClassicSimilarity], result of:
            0.16963314 = score(doc=1593,freq=3.0), product of:
              0.31714666 = queryWeight, product of:
                4.5658 = boost
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.014058317 = queryNorm
              0.53487283 = fieldWeight in 1593, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
          0.2782949 = weight(abstract_txt:effort in 1593) [ClassicSimilarity], result of:
            0.2782949 = score(doc=1593,freq=2.0), product of:
              0.5491215 = queryWeight, product of:
                6.8122926 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014058317 = queryNorm
              0.50680023 = fieldWeight in 1593, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0625 = fieldNorm(doc=1593)
        0.4 = coord(10/25)
    
  3. Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011) 0.34
    0.33918214 = sum of:
      0.33918214 = product of:
        0.84795535 = sum of:
          0.015906753 = weight(abstract_txt:document in 1198) [ClassicSimilarity], result of:
            0.015906753 = score(doc=1198,freq=1.0), product of:
              0.06797039 = queryWeight, product of:
                1.1298289 = boost
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.014058317 = queryNorm
              0.23402475 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.05775571 = weight(abstract_txt:evaluating in 1198) [ClassicSimilarity], result of:
            0.05775571 = score(doc=1198,freq=2.0), product of:
              0.12744379 = queryWeight, product of:
                1.5470778 = boost
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.014058317 = queryNorm
              0.45318574 = fieldWeight in 1198, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.02762324 = weight(abstract_txt:evaluation in 1198) [ClassicSimilarity], result of:
            0.02762324 = score(doc=1198,freq=1.0), product of:
              0.11241188 = queryWeight, product of:
                1.7795268 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.014058317 = queryNorm
              0.2457324 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.03240251 = weight(abstract_txt:systems in 1198) [ClassicSimilarity], result of:
            0.03240251 = score(doc=1198,freq=4.0), product of:
              0.0866911 = queryWeight, product of:
                1.8044924 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014058317 = queryNorm
              0.37376976 = fieldWeight in 1198, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.065677784 = weight(abstract_txt:rankings in 1198) [ClassicSimilarity], result of:
            0.065677784 = score(doc=1198,freq=1.0), product of:
              0.1749354 = queryWeight, product of:
                1.812557 = boost
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.014058317 = queryNorm
              0.3754402 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.8651924 = idf(docFreq=119, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.03370777 = weight(abstract_txt:retrieval in 1198) [ClassicSimilarity], result of:
            0.03370777 = score(doc=1198,freq=4.0), product of:
              0.08900384 = queryWeight, product of:
                1.8284041 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014058317 = queryNorm
              0.37872267 = fieldWeight in 1198, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.017208917 = weight(abstract_txt:using in 1198) [ClassicSimilarity], result of:
            0.017208917 = score(doc=1198,freq=1.0), product of:
              0.09024942 = queryWeight, product of:
                1.8411535 = boost
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.014058317 = queryNorm
              0.19068176 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.11846438 = weight(abstract_txt:shallow in 1198) [ClassicSimilarity], result of:
            0.11846438 = score(doc=1198,freq=1.0), product of:
              0.25921327 = queryWeight, product of:
                2.2063856 = boost
                8.356848 = idf(docFreq=26, maxDocs=42306)
                0.014058317 = queryNorm
              0.4570151 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.356848 = idf(docFreq=26, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.13483587 = weight(abstract_txt:judgments in 1198) [ClassicSimilarity], result of:
            0.13483587 = score(doc=1198,freq=1.0), product of:
              0.3560238 = queryWeight, product of:
                3.6568513 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.014058317 = queryNorm
              0.37872714 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.3443724 = weight(abstract_txt:effort in 1198) [ClassicSimilarity], result of:
            0.3443724 = score(doc=1198,freq=4.0), product of:
              0.5491215 = queryWeight, product of:
                6.8122926 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014058317 = queryNorm
              0.6271333 = fieldWeight in 1198, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
        0.4 = coord(10/25)
    
  4. White, H.D.: Relevance theory and distributions of judgments in document retrieval (2017) 0.32
    0.32154006 = sum of:
      0.32154006 = product of:
        1.0048127 = sum of:
          0.12064426 = weight(abstract_txt:judges in 2017) [ClassicSimilarity], result of:
            0.12064426 = score(doc=2017,freq=4.0), product of:
              0.12001767 = queryWeight, product of:
                1.0615988 = boost
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.014058317 = queryNorm
              1.0052208 = fieldWeight in 2017, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                8.041766 = idf(docFreq=36, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
          0.031569418 = weight(abstract_txt:evaluation in 2017) [ClassicSimilarity], result of:
            0.031569418 = score(doc=2017,freq=1.0), product of:
              0.11241188 = queryWeight, product of:
                1.7795268 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.014058317 = queryNorm
              0.28083703 = fieldWeight in 2017, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
          0.03336204 = weight(abstract_txt:retrieval in 2017) [ClassicSimilarity], result of:
            0.03336204 = score(doc=2017,freq=3.0), product of:
              0.08900384 = queryWeight, product of:
                1.8284041 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014058317 = queryNorm
              0.3748382 = fieldWeight in 2017, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
          0.019667335 = weight(abstract_txt:using in 2017) [ClassicSimilarity], result of:
            0.019667335 = score(doc=2017,freq=1.0), product of:
              0.09024942 = queryWeight, product of:
                1.8411535 = boost
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.014058317 = queryNorm
              0.217922 = fieldWeight in 2017, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.486752 = idf(docFreq=3518, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
          0.056027547 = weight(abstract_txt:documents in 2017) [ClassicSimilarity], result of:
            0.056027547 = score(doc=2017,freq=3.0), product of:
              0.12575002 = queryWeight, product of:
                2.1733108 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.014058317 = queryNorm
              0.445547 = fieldWeight in 2017, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
          0.15409814 = weight(abstract_txt:judgments in 2017) [ClassicSimilarity], result of:
            0.15409814 = score(doc=2017,freq=1.0), product of:
              0.3560238 = queryWeight, product of:
                3.6568513 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.014058317 = queryNorm
              0.43283102 = fieldWeight in 2017, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
          0.19587548 = weight(abstract_txt:relevance in 2017) [ClassicSimilarity], result of:
            0.19587548 = score(doc=2017,freq=4.0), product of:
              0.31714666 = queryWeight, product of:
                4.5658 = boost
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.014058317 = queryNorm
              0.61761796 = fieldWeight in 2017, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
          0.39356846 = weight(abstract_txt:effort in 2017) [ClassicSimilarity], result of:
            0.39356846 = score(doc=2017,freq=4.0), product of:
              0.5491215 = queryWeight, product of:
                6.8122926 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014058317 = queryNorm
              0.7167238 = fieldWeight in 2017, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.0625 = fieldNorm(doc=2017)
        0.32 = coord(8/25)
    
  5. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.31
    0.31212208 = sum of:
      0.31212208 = product of:
        0.8670057 = sum of:
          0.013634359 = weight(abstract_txt:document in 2010) [ClassicSimilarity], result of:
            0.013634359 = score(doc=2010,freq=1.0), product of:
              0.06797039 = queryWeight, product of:
                1.1298289 = boost
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.014058317 = queryNorm
              0.20059264 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.03500525 = weight(abstract_txt:evaluating in 2010) [ClassicSimilarity], result of:
            0.03500525 = score(doc=2010,freq=1.0), product of:
              0.12744379 = queryWeight, product of:
                1.5470778 = boost
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.014058317 = queryNorm
              0.27467206 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.8596706 = idf(docFreq=327, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.052943524 = weight(abstract_txt:evaluation in 2010) [ClassicSimilarity], result of:
            0.052943524 = score(doc=2010,freq=5.0), product of:
              0.11241188 = queryWeight, product of:
                1.7795268 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.014058317 = queryNorm
              0.47097802 = fieldWeight in 2010, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.019638887 = weight(abstract_txt:systems in 2010) [ClassicSimilarity], result of:
            0.019638887 = score(doc=2010,freq=2.0), product of:
              0.0866911 = queryWeight, product of:
                1.8044924 = boost
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.014058317 = queryNorm
              0.22653869 = fieldWeight in 2010, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4173236 = idf(docFreq=3771, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.014446187 = weight(abstract_txt:retrieval in 2010) [ClassicSimilarity], result of:
            0.014446187 = score(doc=2010,freq=1.0), product of:
              0.08900384 = queryWeight, product of:
                1.8284041 = boost
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.014058317 = queryNorm
              0.1623097 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4626071 = idf(docFreq=3604, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.2487234 = weight(abstract_txt:shallow in 2010) [ClassicSimilarity], result of:
            0.2487234 = score(doc=2010,freq=6.0), product of:
              0.25921327 = queryWeight, product of:
                2.2063856 = boost
                8.356848 = idf(docFreq=26, maxDocs=42306)
                0.014058317 = queryNorm
              0.9595319 = fieldWeight in 2010, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                8.356848 = idf(docFreq=26, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.2311472 = weight(abstract_txt:judgments in 2010) [ClassicSimilarity], result of:
            0.2311472 = score(doc=2010,freq=4.0), product of:
              0.3560238 = queryWeight, product of:
                3.6568513 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.014058317 = queryNorm
              0.6492465 = fieldWeight in 2010, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.103878655 = weight(abstract_txt:relevance in 2010) [ClassicSimilarity], result of:
            0.103878655 = score(doc=2010,freq=2.0), product of:
              0.31714666 = queryWeight, product of:
                4.5658 = boost
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.014058317 = queryNorm
              0.32754138 = fieldWeight in 2010, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.9409437 = idf(docFreq=821, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
          0.14758816 = weight(abstract_txt:effort in 2010) [ClassicSimilarity], result of:
            0.14758816 = score(doc=2010,freq=1.0), product of:
              0.5491215 = queryWeight, product of:
                6.8122926 = boost
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.014058317 = queryNorm
              0.2687714 = fieldWeight in 2010, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7337904 = idf(docFreq=371, maxDocs=42306)
                0.046875 = fieldNorm(doc=2010)
        0.36 = coord(9/25)