Document (#12074)

Author

Harter, S.P.

Title

Variations in relevance assessments and the measurement of retrieval effectiveness

Source

Journal of the American Society for Information Science. 47(1996) no.1, S.37-49

Year

1996

Abstract

The purpose of this article is to bring attention to the problem of variations in relevance assessments and the effects that these may have on measures of retrieval effectiveness. Through an analytical review of the literature, I show that despite known wide variations in relevance assessments in experimental test collections, their effects on the measurement of retrieval performance are almost completely unstudied. I will further argue that what we know about tha many variables that have been found to affect relevance assessments under experimental conditions, as well as our new understanding of psychological, situational, user-based relevance, point to a single conclusion. We can no longer rest the evaluation of information retrieval systems on the assumption that such variations do not significantly affect the measurement of information retrieval performance. A series of thourough, rigorous, and extensive tests is needed, of precisely how, and under what conditions, variations in relevance assessments do, and do not, affect measures of retrieval performance. We need to develop approaches to evaluation that are sensitive to these variations and to human factors and individual differences more generally. Our approaches to evaluation must reflect the real world of real users

Theme

Retrievalstudien

Similar documents (author)

1. Harter, S.P.: Search term combinations and retrieval overlap : a proposed methodology and case study (1990) 5.54

   5.5397964 = sum of:
     5.5397964 = weight(author_txt:harter in 339) [ClassicSimilarity], result of:
       5.5397964 = fieldWeight in 339, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.863674 = idf(docFreq=16, maxDocs=44218)
         0.625 = fieldNorm(doc=339)
   

2. Harter, S.P.: Detrimental effects of searching with precoordinated terms (1988) 5.54

   5.5397964 = sum of:
     5.5397964 = weight(author_txt:harter in 366) [ClassicSimilarity], result of:
       5.5397964 = fieldWeight in 366, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.863674 = idf(docFreq=16, maxDocs=44218)
         0.625 = fieldNorm(doc=366)
   

3. Harter, S.P.: Psychological relevance and information science (1992) 5.54

   5.5397964 = sum of:
     5.5397964 = weight(author_txt:harter in 5355) [ClassicSimilarity], result of:
       5.5397964 = fieldWeight in 5355, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.863674 = idf(docFreq=16, maxDocs=44218)
         0.625 = fieldNorm(doc=5355)
   

4. Harter, S.P.: ¬The Cranfield II relevance assessments : a critical evaluation (1971) 5.54

   5.5397964 = sum of:
     5.5397964 = weight(author_txt:harter in 5364) [ClassicSimilarity], result of:
       5.5397964 = fieldWeight in 5364, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.863674 = idf(docFreq=16, maxDocs=44218)
         0.625 = fieldNorm(doc=5364)
   

5. Harter, S.P.: ¬A probabilistic approach to keyword indexing (1975) 5.54

   5.5397964 = sum of:
     5.5397964 = weight(author_txt:harter in 5481) [ClassicSimilarity], result of:
       5.5397964 = fieldWeight in 5481, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.863674 = idf(docFreq=16, maxDocs=44218)
         0.625 = fieldNorm(doc=5481)
   

Similar documents (content)

1. Burgin, R.: Variations in relevance judgements and the evaluation of retrieval performance (1992) 0.54

   0.5372763 = sum of:
     0.5372763 = product of:
       1.4924341 = sum of:
         0.048815396 = weight(abstract_txt:effectiveness in 3308) [ClassicSimilarity], result of:
           0.048815396 = score(doc=3308,freq=1.0), product of:
             0.10213004 = queryWeight, product of:
               1.4378402 = boost
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.013931915 = queryNorm
             0.47797295 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.05048511 = weight(abstract_txt:under in 3308) [ClassicSimilarity], result of:
           0.05048511 = score(doc=3308,freq=1.0), product of:
             0.104445845 = queryWeight, product of:
               1.4540504 = boost
               5.155857 = idf(docFreq=692, maxDocs=44218)
               0.013931915 = queryNorm
             0.4833616 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.155857 = idf(docFreq=692, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.089382194 = weight(abstract_txt:conditions in 3308) [ClassicSimilarity], result of:
           0.089382194 = score(doc=3308,freq=1.0), product of:
             0.15285616 = queryWeight, product of:
               1.7590384 = boost
               6.237302 = idf(docFreq=234, maxDocs=44218)
               0.013931915 = queryNorm
             0.5847471 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.237302 = idf(docFreq=234, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.049882673 = weight(abstract_txt:evaluation in 3308) [ClassicSimilarity], result of:
           0.049882673 = score(doc=3308,freq=1.0), product of:
             0.118607596 = queryWeight, product of:
               1.8977358 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.013931915 = queryNorm
             0.42056894 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.09501097 = weight(abstract_txt:performance in 3308) [ClassicSimilarity], result of:
           0.09501097 = score(doc=3308,freq=3.0), product of:
             0.12636346 = queryWeight, product of:
               1.9588007 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.013931915 = queryNorm
             0.7518865 = fieldWeight in 3308, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.16231026 = weight(abstract_txt:affect in 3308) [ClassicSimilarity], result of:
           0.16231026 = score(doc=3308,freq=2.0), product of:
             0.20671247 = queryWeight, product of:
               2.505318 = boost
               5.9223356 = idf(docFreq=321, maxDocs=44218)
               0.013931915 = queryNorm
             0.7851982 = fieldWeight in 3308, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.9223356 = idf(docFreq=321, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.09275313 = weight(abstract_txt:retrieval in 3308) [ClassicSimilarity], result of:
           0.09275313 = score(doc=3308,freq=4.0), product of:
             0.14234921 = queryWeight, product of:
               2.9401674 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.013931915 = queryNorm
             0.6515886 = fieldWeight in 3308, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.32470325 = weight(abstract_txt:relevance in 3308) [ClassicSimilarity], result of:
           0.32470325 = score(doc=3308,freq=6.0), product of:
             0.28670153 = queryWeight, product of:
               4.1726265 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013931915 = queryNorm
             1.132548 = fieldWeight in 3308, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.5790912 = weight(abstract_txt:variations in 3308) [ClassicSimilarity], result of:
           0.5790912 = score(doc=3308,freq=3.0), product of:
             0.5312272 = queryWeight, product of:
               5.679821 = boost
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.013931915 = queryNorm
             1.0901009 = fieldWeight in 3308, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
       0.36 = coord(9/25)
   

2. Saracevic, T.: Effects of inconsistent relevance judgments on information retrieval test results : a historical perspective (2008) 0.27

   0.26637882 = sum of:
     0.26637882 = product of:
       0.7399411 = sum of:
         0.0325436 = weight(abstract_txt:effectiveness in 5585) [ClassicSimilarity], result of:
           0.0325436 = score(doc=5585,freq=1.0), product of:
             0.10213004 = queryWeight, product of:
               1.4378402 = boost
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.013931915 = queryNorm
             0.31864864 = fieldWeight in 5585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.038623627 = weight(abstract_txt:experimental in 5585) [ClassicSimilarity], result of:
           0.038623627 = score(doc=5585,freq=1.0), product of:
             0.1144841 = queryWeight, product of:
               1.5223218 = boost
               5.397938 = idf(docFreq=543, maxDocs=44218)
               0.013931915 = queryNorm
             0.3373711 = fieldWeight in 5585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.397938 = idf(docFreq=543, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.042336956 = weight(abstract_txt:effects in 5585) [ClassicSimilarity], result of:
           0.042336956 = score(doc=5585,freq=1.0), product of:
             0.12170905 = queryWeight, product of:
               1.5696229 = boost
               5.565661 = idf(docFreq=459, maxDocs=44218)
               0.013931915 = queryNorm
             0.3478538 = fieldWeight in 5585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.565661 = idf(docFreq=459, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.057599545 = weight(abstract_txt:evaluation in 5585) [ClassicSimilarity], result of:
           0.057599545 = score(doc=5585,freq=3.0), product of:
             0.118607596 = queryWeight, product of:
               1.8977358 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.013931915 = queryNorm
             0.48563117 = fieldWeight in 5585, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.036569744 = weight(abstract_txt:performance in 5585) [ClassicSimilarity], result of:
           0.036569744 = score(doc=5585,freq=1.0), product of:
             0.12636346 = queryWeight, product of:
               1.9588007 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.013931915 = queryNorm
             0.28940126 = fieldWeight in 5585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.009800474 = weight(abstract_txt:that in 5585) [ClassicSimilarity], result of:
           0.009800474 = score(doc=5585,freq=1.0), product of:
             0.066178195 = queryWeight, product of:
               2.0047119 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.013931915 = queryNorm
             0.1480922 = fieldWeight in 5585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.053551044 = weight(abstract_txt:retrieval in 5585) [ClassicSimilarity], result of:
           0.053551044 = score(doc=5585,freq=3.0), product of:
             0.14234921 = queryWeight, product of:
               2.9401674 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.013931915 = queryNorm
             0.37619486 = fieldWeight in 5585, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.24995665 = weight(abstract_txt:relevance in 5585) [ClassicSimilarity], result of:
           0.24995665 = score(doc=5585,freq=8.0), product of:
             0.28670153 = queryWeight, product of:
               4.1726265 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013931915 = queryNorm
             0.87183577 = fieldWeight in 5585, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
         0.21895945 = weight(abstract_txt:assessments in 5585) [ClassicSimilarity], result of:
           0.21895945 = score(doc=5585,freq=1.0), product of:
             0.49400604 = queryWeight, product of:
               5.0 = boost
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.013931915 = queryNorm
             0.44323233 = fieldWeight in 5585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.0625 = fieldNorm(doc=5585)
       0.36 = coord(9/25)
   

3. Golub, K.; Soergel, D.; Buchanan, G.; Tudhope, D.; Lykke, M.; Hiom, D.: ¬A framework for evaluating automatic indexing or classification in the context of retrieval (2016) 0.26

   0.25564325 = sum of:
     0.25564325 = product of:
       0.6391081 = sum of:
         0.033990465 = weight(abstract_txt:approaches in 3311) [ClassicSimilarity], result of:
           0.033990465 = score(doc=3311,freq=2.0), product of:
             0.08344581 = queryWeight, product of:
               1.2996792 = boost
               4.6084785 = idf(docFreq=1197, maxDocs=44218)
               0.013931915 = queryNorm
             0.4073358 = fieldWeight in 3311, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6084785 = idf(docFreq=1197, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.03673179 = weight(abstract_txt:real in 3311) [ClassicSimilarity], result of:
           0.03673179 = score(doc=3311,freq=1.0), product of:
             0.1107145 = queryWeight, product of:
               1.4970495 = boost
               5.308326 = idf(docFreq=594, maxDocs=44218)
               0.013931915 = queryNorm
             0.33177036 = fieldWeight in 3311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.308326 = idf(docFreq=594, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.038623627 = weight(abstract_txt:experimental in 3311) [ClassicSimilarity], result of:
           0.038623627 = score(doc=3311,freq=1.0), product of:
             0.1144841 = queryWeight, product of:
               1.5223218 = boost
               5.397938 = idf(docFreq=543, maxDocs=44218)
               0.013931915 = queryNorm
             0.3373711 = fieldWeight in 3311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.397938 = idf(docFreq=543, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.059588123 = weight(abstract_txt:conditions in 3311) [ClassicSimilarity], result of:
           0.059588123 = score(doc=3311,freq=1.0), product of:
             0.15285616 = queryWeight, product of:
               1.7590384 = boost
               6.237302 = idf(docFreq=234, maxDocs=44218)
               0.013931915 = queryNorm
             0.38983136 = fieldWeight in 3311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.237302 = idf(docFreq=234, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.057599545 = weight(abstract_txt:evaluation in 3311) [ClassicSimilarity], result of:
           0.057599545 = score(doc=3311,freq=3.0), product of:
             0.118607596 = queryWeight, product of:
               1.8977358 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.013931915 = queryNorm
             0.48563117 = fieldWeight in 3311, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.051717427 = weight(abstract_txt:performance in 3311) [ClassicSimilarity], result of:
           0.051717427 = score(doc=3311,freq=2.0), product of:
             0.12636346 = queryWeight, product of:
               1.9588007 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.013931915 = queryNorm
             0.40927517 = fieldWeight in 3311, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.009800474 = weight(abstract_txt:that in 3311) [ClassicSimilarity], result of:
           0.009800474 = score(doc=3311,freq=1.0), product of:
             0.066178195 = queryWeight, product of:
               2.0047119 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.013931915 = queryNorm
             0.1480922 = fieldWeight in 3311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.043724243 = weight(abstract_txt:retrieval in 3311) [ClassicSimilarity], result of:
           0.043724243 = score(doc=3311,freq=2.0), product of:
             0.14234921 = queryWeight, product of:
               2.9401674 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.013931915 = queryNorm
             0.3071618 = fieldWeight in 3311, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.08837302 = weight(abstract_txt:relevance in 3311) [ClassicSimilarity], result of:
           0.08837302 = score(doc=3311,freq=1.0), product of:
             0.28670153 = queryWeight, product of:
               4.1726265 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013931915 = queryNorm
             0.3082405 = fieldWeight in 3311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
         0.21895945 = weight(abstract_txt:assessments in 3311) [ClassicSimilarity], result of:
           0.21895945 = score(doc=3311,freq=1.0), product of:
             0.49400604 = queryWeight, product of:
               5.0 = boost
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.013931915 = queryNorm
             0.44323233 = fieldWeight in 3311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.0625 = fieldNorm(doc=3311)
       0.4 = coord(10/25)
   

4. Baillie, M.; Azzopardi, L.; Ruthven, I.: Evaluating epistemic uncertainty under incomplete assessments (2008) 0.24

   0.24442932 = sum of:
     0.24442932 = product of:
       0.7638416 = sum of:
         0.042070925 = weight(abstract_txt:under in 2065) [ClassicSimilarity], result of:
           0.042070925 = score(doc=2065,freq=1.0), product of:
             0.104445845 = queryWeight, product of:
               1.4540504 = boost
               5.155857 = idf(docFreq=692, maxDocs=44218)
               0.013931915 = queryNorm
             0.40280133 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.155857 = idf(docFreq=692, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.07199943 = weight(abstract_txt:evaluation in 2065) [ClassicSimilarity], result of:
           0.07199943 = score(doc=2065,freq=3.0), product of:
             0.118607596 = queryWeight, product of:
               1.8977358 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.013931915 = queryNorm
             0.607039 = fieldWeight in 2065, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.04571218 = weight(abstract_txt:performance in 2065) [ClassicSimilarity], result of:
           0.04571218 = score(doc=2065,freq=1.0), product of:
             0.12636346 = queryWeight, product of:
               1.9588007 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.013931915 = queryNorm
             0.3617516 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.012250593 = weight(abstract_txt:that in 2065) [ClassicSimilarity], result of:
           0.012250593 = score(doc=2065,freq=1.0), product of:
             0.066178195 = queryWeight, product of:
               2.0047119 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.013931915 = queryNorm
             0.18511525 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.15298761 = weight(abstract_txt:measurement in 2065) [ClassicSimilarity], result of:
           0.15298761 = score(doc=2065,freq=1.0), product of:
             0.28272945 = queryWeight, product of:
               2.9299827 = boost
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.013931915 = queryNorm
             0.54110956 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.054655302 = weight(abstract_txt:retrieval in 2065) [ClassicSimilarity], result of:
           0.054655302 = score(doc=2065,freq=2.0), product of:
             0.14234921 = queryWeight, product of:
               2.9401674 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.013931915 = queryNorm
             0.38395226 = fieldWeight in 2065, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.11046628 = weight(abstract_txt:relevance in 2065) [ClassicSimilarity], result of:
           0.11046628 = score(doc=2065,freq=1.0), product of:
             0.28670153 = queryWeight, product of:
               4.1726265 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013931915 = queryNorm
             0.38530064 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.2736993 = weight(abstract_txt:assessments in 2065) [ClassicSimilarity], result of:
           0.2736993 = score(doc=2065,freq=1.0), product of:
             0.49400604 = queryWeight, product of:
               5.0 = boost
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.013931915 = queryNorm
             0.55404043 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
       0.32 = coord(8/25)
   

5. Lesk, M.E.; Salton, G.: Relevance assements and retrieval system evaluation (1969) 0.24

   0.2394819 = sum of:
     0.2394819 = product of:
       0.8552925 = sum of:
         0.069035396 = weight(abstract_txt:effectiveness in 4151) [ClassicSimilarity], result of:
           0.069035396 = score(doc=4151,freq=2.0), product of:
             0.10213004 = queryWeight, product of:
               1.4378402 = boost
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.013931915 = queryNorm
             0.67595583 = fieldWeight in 4151, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.059149925 = weight(abstract_txt:measures in 4151) [ClassicSimilarity], result of:
           0.059149925 = score(doc=4151,freq=1.0), product of:
             0.116078496 = queryWeight, product of:
               1.5328857 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.013931915 = queryNorm
             0.50956833 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.049882673 = weight(abstract_txt:evaluation in 4151) [ClassicSimilarity], result of:
           0.049882673 = score(doc=4151,freq=1.0), product of:
             0.118607596 = queryWeight, product of:
               1.8977358 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.013931915 = queryNorm
             0.42056894 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.025462376 = weight(abstract_txt:that in 4151) [ClassicSimilarity], result of:
           0.025462376 = score(doc=4151,freq=3.0), product of:
             0.066178195 = queryWeight, product of:
               2.0047119 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.013931915 = queryNorm
             0.38475478 = fieldWeight in 4151, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.046376564 = weight(abstract_txt:retrieval in 4151) [ClassicSimilarity], result of:
           0.046376564 = score(doc=4151,freq=1.0), product of:
             0.14234921 = queryWeight, product of:
               2.9401674 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.013931915 = queryNorm
             0.3257943 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.13255954 = weight(abstract_txt:relevance in 4151) [ClassicSimilarity], result of:
           0.13255954 = score(doc=4151,freq=1.0), product of:
             0.28670153 = queryWeight, product of:
               4.1726265 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013931915 = queryNorm
             0.46236074 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.47282603 = weight(abstract_txt:variations in 4151) [ClassicSimilarity], result of:
           0.47282603 = score(doc=4151,freq=2.0), product of:
             0.5312272 = queryWeight, product of:
               5.679821 = boost
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.013931915 = queryNorm
             0.8900637 = fieldWeight in 4151, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
       0.28 = coord(7/25)