Document (#21061)

Author

Tang, R.
Shaw, W.M.
Vevea, J.L.

Title

Towards the identification of the optimal number of relevance categories

Source

Journal of the American Society for Information Science. 50(1999) no.3, S.254-264

Year

1999

Abstract

In this paper, we are concerned with participants' confidence in their judgements of the relevance of bibliographic records to particular research questions. We describe an empirical investigation of the association between judges' confidence and the number of categories for a relevance rating scale. Participants rated the relevance of bibliographic records, and recorded their confidence in the relevance ratings. We hypothesize that confidence in relevance judgements is a function of the number of relevance categories that are available in the rating scale. We consider scales ranging from 2 to 11 points, and define the optimal scale as the one for which participants express a maximum level of confidence. A pilot study finds no optimal number of points (because confidence continues to improve slightly through the 11-point scale); nevertheless, the study shows little added benefit associated with scales that have more than six points. On the basis of the findings in that study, we adjusted our experimental procedures and found, in our principal study, that the optimal scale for maximizing confidence in relevance judgements has approximately seven points. We also present exploratory results involving gender effects, and the comparison of scales that have an odd number of points (for which a neutral judgement is possible) with scales that have an even number of points

Similar documents (author)

1. Shaw, R.R.: Classification systems (1962/63) 1.79

   1.7861812 = sum of:
     1.7861812 = product of:
       3.5723624 = sum of:
         3.5723624 = weight(author_txt:shaw in 603) [ClassicSimilarity], result of:
           3.5723624 = score(doc=603,freq=1.0), product of:
             0.7115086 = queryWeight, product of:
               1.0062643 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.088018246 = queryNorm
             5.0208282 = fieldWeight in 603, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.625 = fieldNorm(doc=603)
       0.5 = coord(1/2)
   

2. Shaw, W.M.: Subject and citation indexing : pt.1: the clustering structure of composite representations in the cystic fibrosis document collection (1991) 1.79

   1.7861812 = sum of:
     1.7861812 = product of:
       3.5723624 = sum of:
         3.5723624 = weight(author_txt:shaw in 4841) [ClassicSimilarity], result of:
           3.5723624 = score(doc=4841,freq=1.0), product of:
             0.7115086 = queryWeight, product of:
               1.0062643 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.088018246 = queryNorm
             5.0208282 = fieldWeight in 4841, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.625 = fieldNorm(doc=4841)
       0.5 = coord(1/2)
   

3. Shaw, W.M.: Subject and citation indexing : pt.2: the optimal, cluster-based retrieval performance of composite representations (1991) 1.79

   1.7861812 = sum of:
     1.7861812 = product of:
       3.5723624 = sum of:
         3.5723624 = weight(author_txt:shaw in 4842) [ClassicSimilarity], result of:
           3.5723624 = score(doc=4842,freq=1.0), product of:
             0.7115086 = queryWeight, product of:
               1.0062643 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.088018246 = queryNorm
             5.0208282 = fieldWeight in 4842, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.625 = fieldNorm(doc=4842)
       0.5 = coord(1/2)
   

4. Shaw, S.: ¬The Internet as an entertainment system (1994) 1.79

   1.7861812 = sum of:
     1.7861812 = product of:
       3.5723624 = sum of:
         3.5723624 = weight(author_txt:shaw in 198) [ClassicSimilarity], result of:
           3.5723624 = score(doc=198,freq=1.0), product of:
             0.7115086 = queryWeight, product of:
               1.0062643 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.088018246 = queryNorm
             5.0208282 = fieldWeight in 198, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.625 = fieldNorm(doc=198)
       0.5 = coord(1/2)
   

5. Shaw, D.J.: Cataloguing rare books online (1993) 1.79

   1.7861812 = sum of:
     1.7861812 = product of:
       3.5723624 = sum of:
         3.5723624 = weight(author_txt:shaw in 576) [ClassicSimilarity], result of:
           3.5723624 = score(doc=576,freq=1.0), product of:
             0.7115086 = queryWeight, product of:
               1.0062643 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.088018246 = queryNorm
             5.0208282 = fieldWeight in 576, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.625 = fieldNorm(doc=576)
       0.5 = coord(1/2)
   

Similar documents (content)

1. White, H.D.: Relevance theory and distributions of judgments in document retrieval (2017) 0.28

   0.2825403 = sum of:
     0.2825403 = product of:
       1.0090724 = sum of:
         0.008887759 = weight(abstract_txt:with in 5099) [ClassicSimilarity], result of:
           0.008887759 = score(doc=5099,freq=4.0), product of:
             0.028443862 = queryWeight, product of:
               1.0009351 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.011368128 = queryNorm
             0.31246668 = fieldWeight in 5099, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=5099)
         0.10027348 = weight(abstract_txt:judges in 5099) [ClassicSimilarity], result of:
           0.10027348 = score(doc=5099,freq=4.0), product of:
             0.09920739 = queryWeight, product of:
               1.0792524 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.011368128 = queryNorm
             1.0107461 = fieldWeight in 5099, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.0625 = fieldNorm(doc=5099)
         0.019746924 = weight(abstract_txt:that in 5099) [ClassicSimilarity], result of:
           0.019746924 = score(doc=5099,freq=5.0), product of:
             0.0596324 = queryWeight, product of:
               2.2138143 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011368128 = queryNorm
             0.3311442 = fieldWeight in 5099, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=5099)
         0.15574741 = weight(abstract_txt:scale in 5099) [ClassicSimilarity], result of:
           0.15574741 = score(doc=5099,freq=4.0), product of:
             0.22752224 = queryWeight, product of:
               3.654668 = boost
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.011368128 = queryNorm
             0.6845371 = fieldWeight in 5099, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.0625 = fieldNorm(doc=5099)
         0.32273692 = weight(abstract_txt:scales in 5099) [ClassicSimilarity], result of:
           0.32273692 = score(doc=5099,freq=3.0), product of:
             0.3778493 = queryWeight, product of:
               4.212504 = boost
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.011368128 = queryNorm
             0.8541419 = fieldWeight in 5099, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.0625 = fieldNorm(doc=5099)
         0.21966414 = weight(abstract_txt:points in 5099) [ClassicSimilarity], result of:
           0.21966414 = score(doc=5099,freq=6.0), product of:
             0.26563138 = queryWeight, product of:
               4.325799 = boost
               5.4016213 = idf(docFreq=541, maxDocs=44218)
               0.011368128 = queryNorm
             0.826951 = fieldWeight in 5099, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.4016213 = idf(docFreq=541, maxDocs=44218)
               0.0625 = fieldNorm(doc=5099)
         0.18201575 = weight(abstract_txt:relevance in 5099) [ClassicSimilarity], result of:
           0.18201575 = score(doc=5099,freq=4.0), product of:
             0.29524955 = queryWeight, product of:
               5.2661185 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011368128 = queryNorm
             0.616481 = fieldWeight in 5099, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=5099)
       0.28 = coord(7/25)
   

2. Janes, J.W.; McKinney, R.: Relevance judgements of actual users and secondary judges : a comparative study (1992) 0.21

   0.20625716 = sum of:
     0.20625716 = product of:
       0.64455366 = sum of:
         0.0066658193 = weight(abstract_txt:with in 4276) [ClassicSimilarity], result of:
           0.0066658193 = score(doc=4276,freq=1.0), product of:
             0.028443862 = queryWeight, product of:
               1.0009351 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.011368128 = queryNorm
             0.23435001 = fieldWeight in 4276, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
         0.15041022 = weight(abstract_txt:judges in 4276) [ClassicSimilarity], result of:
           0.15041022 = score(doc=4276,freq=4.0), product of:
             0.09920739 = queryWeight, product of:
               1.0792524 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.011368128 = queryNorm
             1.5161192 = fieldWeight in 4276, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
         0.024560755 = weight(abstract_txt:records in 4276) [ClassicSimilarity], result of:
           0.024560755 = score(doc=4276,freq=1.0), product of:
             0.05927664 = queryWeight, product of:
               1.1797984 = boost
               4.4196396 = idf(docFreq=1446, maxDocs=44218)
               0.011368128 = queryNorm
             0.4143412 = fieldWeight in 4276, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4196396 = idf(docFreq=1446, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
         0.014044125 = weight(abstract_txt:have in 4276) [ClassicSimilarity], result of:
           0.014044125 = score(doc=4276,freq=1.0), product of:
             0.046746522 = queryWeight, product of:
               1.2831762 = boost
               3.2046018 = idf(docFreq=4876, maxDocs=44218)
               0.011368128 = queryNorm
             0.30043143 = fieldWeight in 4276, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.2046018 = idf(docFreq=4876, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
         0.022836974 = weight(abstract_txt:study in 4276) [ClassicSimilarity], result of:
           0.022836974 = score(doc=4276,freq=1.0), product of:
             0.071147256 = queryWeight, product of:
               1.8279322 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.011368128 = queryNorm
             0.3209818 = fieldWeight in 4276, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
         0.013246639 = weight(abstract_txt:that in 4276) [ClassicSimilarity], result of:
           0.013246639 = score(doc=4276,freq=1.0), product of:
             0.0596324 = queryWeight, product of:
               2.2138143 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011368128 = queryNorm
             0.22213829 = fieldWeight in 4276, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
         0.17634374 = weight(abstract_txt:judgements in 4276) [ClassicSimilarity], result of:
           0.17634374 = score(doc=4276,freq=1.0), product of:
             0.2525377 = queryWeight, product of:
               2.9824607 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.011368128 = queryNorm
             0.6982868 = fieldWeight in 4276, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
         0.23644538 = weight(abstract_txt:relevance in 4276) [ClassicSimilarity], result of:
           0.23644538 = score(doc=4276,freq=3.0), product of:
             0.29524955 = queryWeight, product of:
               5.2661185 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011368128 = queryNorm
             0.80083233 = fieldWeight in 4276, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=4276)
       0.32 = coord(8/25)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.19

   0.18738677 = sum of:
     0.18738677 = product of:
       0.66923845 = sum of:
         0.0062845945 = weight(abstract_txt:with in 3593) [ClassicSimilarity], result of:
           0.0062845945 = score(doc=3593,freq=2.0), product of:
             0.028443862 = queryWeight, product of:
               1.0009351 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.011368128 = queryNorm
             0.22094731 = fieldWeight in 3593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.0304493 = weight(abstract_txt:study in 3593) [ClassicSimilarity], result of:
           0.0304493 = score(doc=3593,freq=4.0), product of:
             0.071147256 = queryWeight, product of:
               1.8279322 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.011368128 = queryNorm
             0.42797574 = fieldWeight in 3593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.043898746 = weight(abstract_txt:participants in 3593) [ClassicSimilarity], result of:
           0.043898746 = score(doc=3593,freq=1.0), product of:
             0.1309529 = queryWeight, product of:
               2.1476781 = boost
               5.3636074 = idf(docFreq=562, maxDocs=44218)
               0.011368128 = queryNorm
             0.33522546 = fieldWeight in 3593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3636074 = idf(docFreq=562, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.008831093 = weight(abstract_txt:that in 3593) [ClassicSimilarity], result of:
           0.008831093 = score(doc=3593,freq=1.0), product of:
             0.0596324 = queryWeight, product of:
               2.2138143 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011368128 = queryNorm
             0.1480922 = fieldWeight in 3593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.2628777 = weight(abstract_txt:judgements in 3593) [ClassicSimilarity], result of:
           0.2628777 = score(doc=3593,freq=5.0), product of:
             0.2525377 = queryWeight, product of:
               2.9824607 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.011368128 = queryNorm
             1.0409445 = fieldWeight in 3593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.13488123 = weight(abstract_txt:scale in 3593) [ClassicSimilarity], result of:
           0.13488123 = score(doc=3593,freq=3.0), product of:
             0.22752224 = queryWeight, product of:
               3.654668 = boost
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.011368128 = queryNorm
             0.59282655 = fieldWeight in 3593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.18201575 = weight(abstract_txt:relevance in 3593) [ClassicSimilarity], result of:
           0.18201575 = score(doc=3593,freq=4.0), product of:
             0.29524955 = queryWeight, product of:
               5.2661185 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011368128 = queryNorm
             0.616481 = fieldWeight in 3593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
       0.28 = coord(7/25)
   

4. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings (2016) 0.18

   0.18215597 = sum of:
     0.18215597 = product of:
       0.65055704 = sum of:
         0.0054990207 = weight(abstract_txt:with in 3071) [ClassicSimilarity], result of:
           0.0054990207 = score(doc=3071,freq=2.0), product of:
             0.028443862 = queryWeight, product of:
               1.0009351 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.011368128 = queryNorm
             0.1933289 = fieldWeight in 3071, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0546875 = fieldNorm(doc=3071)
         0.026643138 = weight(abstract_txt:study in 3071) [ClassicSimilarity], result of:
           0.026643138 = score(doc=3071,freq=4.0), product of:
             0.071147256 = queryWeight, product of:
               1.8279322 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.011368128 = queryNorm
             0.3744788 = fieldWeight in 3071, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0546875 = fieldNorm(doc=3071)
         0.0384114 = weight(abstract_txt:participants in 3071) [ClassicSimilarity], result of:
           0.0384114 = score(doc=3071,freq=1.0), product of:
             0.1309529 = queryWeight, product of:
               2.1476781 = boost
               5.3636074 = idf(docFreq=562, maxDocs=44218)
               0.011368128 = queryNorm
             0.29332227 = fieldWeight in 3071, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3636074 = idf(docFreq=562, maxDocs=44218)
               0.0546875 = fieldNorm(doc=3071)
         0.015454413 = weight(abstract_txt:that in 3071) [ClassicSimilarity], result of:
           0.015454413 = score(doc=3071,freq=4.0), product of:
             0.0596324 = queryWeight, product of:
               2.2138143 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011368128 = queryNorm
             0.25916135 = fieldWeight in 3071, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0546875 = fieldNorm(doc=3071)
         0.30860153 = weight(abstract_txt:judgements in 3071) [ClassicSimilarity], result of:
           0.30860153 = score(doc=3071,freq=9.0), product of:
             0.2525377 = queryWeight, product of:
               2.9824607 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.011368128 = queryNorm
             1.2220018 = fieldWeight in 3071, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.0546875 = fieldNorm(doc=3071)
         0.11802108 = weight(abstract_txt:scale in 3071) [ClassicSimilarity], result of:
           0.11802108 = score(doc=3071,freq=3.0), product of:
             0.22752224 = queryWeight, product of:
               3.654668 = boost
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.011368128 = queryNorm
             0.51872325 = fieldWeight in 3071, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.0546875 = fieldNorm(doc=3071)
         0.13792647 = weight(abstract_txt:relevance in 3071) [ClassicSimilarity], result of:
           0.13792647 = score(doc=3071,freq=3.0), product of:
             0.29524955 = queryWeight, product of:
               5.2661185 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011368128 = queryNorm
             0.46715218 = fieldWeight in 3071, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0546875 = fieldNorm(doc=3071)
       0.28 = coord(7/25)
   

5. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.17

   0.17130455 = sum of:
     0.17130455 = product of:
       0.713769 = sum of:
         0.0066658193 = weight(abstract_txt:with in 3268) [ClassicSimilarity], result of:
           0.0066658193 = score(doc=3268,freq=1.0), product of:
             0.028443862 = queryWeight, product of:
               1.0009351 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.011368128 = queryNorm
             0.23435001 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.09966436 = weight(abstract_txt:judgement in 3268) [ClassicSimilarity], result of:
           0.09966436 = score(doc=3268,freq=2.0), product of:
             0.09500119 = queryWeight, product of:
               1.0561255 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.011368128 = queryNorm
             1.0490854 = fieldWeight in 3268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.0395548 = weight(abstract_txt:study in 3268) [ClassicSimilarity], result of:
           0.0395548 = score(doc=3268,freq=3.0), product of:
             0.071147256 = queryWeight, product of:
               1.8279322 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.011368128 = queryNorm
             0.5559568 = fieldWeight in 3268, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.013246639 = weight(abstract_txt:that in 3268) [ClassicSimilarity], result of:
           0.013246639 = score(doc=3268,freq=1.0), product of:
             0.0596324 = queryWeight, product of:
               2.2138143 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011368128 = queryNorm
             0.22213829 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.2493877 = weight(abstract_txt:judgements in 3268) [ClassicSimilarity], result of:
           0.2493877 = score(doc=3268,freq=2.0), product of:
             0.2525377 = queryWeight, product of:
               2.9824607 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.011368128 = queryNorm
             0.9875266 = fieldWeight in 3268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.3052497 = weight(abstract_txt:relevance in 3268) [ClassicSimilarity], result of:
           0.3052497 = score(doc=3268,freq=5.0), product of:
             0.29524955 = queryWeight, product of:
               5.2661185 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011368128 = queryNorm
             1.0338701 = fieldWeight in 3268, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
       0.24 = coord(6/25)