Document (#6934)

Author

Su, L.T.

Title

¬The relevance of recall and precision in user evaluation

Source

Journal of the American Society for Information Science. 45(1994) no.3, S.207-217

Year

1994

Abstract

The appropriateness of evaluation criteria and measures have been a subject of debate and a vital concern in the information retrieval evaluation literature. A study was conducted to investigate the appropriateness of 20 measures for evaluating interactive information retrieval performance, representing 4 major evaluation criteria. Among the 20 measures studied were the 2 most well-known relevance-based measures of effectiveness, recall and precision. The user's judgment of information success was used as the devised criterion measure with which all other 20 measures were to be correlated. A sample of 40 end-users with individual information problems from an academic environment were observed, interacting with 6 professional intermediaries searching on their behalf in large operational systems. Quantitative data consisting of values for all measures studies and verbal data containing users' reasons for assigning certain values to selected measures were collected. Statistical analysis of the quantitative data showed that precision, one of the most important traditional measures of effectiveness, is not sifnificantly correlated with the user's judgment of success. Users appear to be more concerned with absolute recall than with precision, although absolute recall was not directly tested in this study. 4 related measures of recall and precision are found to be significantly correlated with success. Among these are user's satisfaction with completeness of search results and user's satisfaction with precision of the search. This article explores the possible explanations for this outcome through content analysis of users' verbal data. The analysis shows that high precision does not always mean high quality (relevancy, completeness, etc.) to users because of different users' expectations. The user's purpose in obtaining information is suggested to be the primary cause for the high concern for recall. Implications for research and practice are discussed

Similar documents (content)

1. Su, L.T.: ¬A comprehensive and systematic model of user evaluation of Web search engines : Il. An evaluation by undergraduates (2003) 0.51

   0.5143218 = sum of:
     0.5143218 = product of:
       0.91843176 = sum of:
         0.024571594 = weight(abstract_txt:analysis in 3118) [ClassicSimilarity], result of:
           0.024571594 = score(doc=3118,freq=3.0), product of:
             0.070109166 = queryWeight, product of:
               1.0839305 = boost
               3.7000692 = idf(docFreq=2842, maxDocs=42306)
               0.01748089 = queryNorm
             0.3504762 = fieldWeight in 3118, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.7000692 = idf(docFreq=2842, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.056453567 = weight(abstract_txt:criteria in 3118) [ClassicSimilarity], result of:
           0.056453567 = score(doc=3118,freq=3.0), product of:
             0.106639095 = queryWeight, product of:
               1.0915073 = boost
               5.588899 = idf(docFreq=429, maxDocs=42306)
               0.01748089 = queryNorm
             0.529389 = fieldWeight in 3118, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.588899 = idf(docFreq=429, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.041280646 = weight(abstract_txt:quantitative in 3118) [ClassicSimilarity], result of:
           0.041280646 = score(doc=3118,freq=1.0), product of:
             0.12483228 = queryWeight, product of:
               1.180951 = boost
               6.046882 = idf(docFreq=271, maxDocs=42306)
               0.01748089 = queryNorm
             0.33068886 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.046882 = idf(docFreq=271, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.0095404 = weight(abstract_txt:information in 3118) [ClassicSimilarity], result of:
           0.0095404 = score(doc=3118,freq=2.0), product of:
             0.05064194 = queryWeight, product of:
               1.1893059 = boost
               2.435865 = idf(docFreq=10064, maxDocs=42306)
               0.01748089 = queryNorm
             0.1883893 = fieldWeight in 3118, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.435865 = idf(docFreq=10064, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.11990882 = weight(abstract_txt:satisfaction in 3118) [ClassicSimilarity], result of:
           0.11990882 = score(doc=3118,freq=6.0), product of:
             0.1398555 = queryWeight, product of:
               1.2499948 = boost
               6.4004107 = idf(docFreq=190, maxDocs=42306)
               0.01748089 = queryNorm
             0.85737646 = fieldWeight in 3118, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.4004107 = idf(docFreq=190, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.014453108 = weight(abstract_txt:data in 3118) [ClassicSimilarity], result of:
           0.014453108 = score(doc=3118,freq=1.0), product of:
             0.07812921 = queryWeight, product of:
               1.3212656 = boost
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.01748089 = queryNorm
             0.18498982 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.0637445 = weight(abstract_txt:verbal in 3118) [ClassicSimilarity], result of:
           0.0637445 = score(doc=3118,freq=1.0), product of:
             0.16677248 = queryWeight, product of:
               1.3649936 = boost
               6.989245 = idf(docFreq=105, maxDocs=42306)
               0.01748089 = queryNorm
             0.38222432 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.989245 = idf(docFreq=105, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.018942233 = weight(abstract_txt:were in 3118) [ClassicSimilarity], result of:
           0.018942233 = score(doc=3118,freq=1.0), product of:
             0.09356786 = queryWeight, product of:
               1.4459283 = boost
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.01748089 = queryNorm
             0.2024438 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.06775401 = weight(abstract_txt:evaluation in 3118) [ClassicSimilarity], result of:
           0.06775401 = score(doc=3118,freq=4.0), product of:
             0.13786137 = queryWeight, product of:
               1.7551115 = boost
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.01748089 = queryNorm
             0.4914648 = fieldWeight in 3118, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.025663769 = weight(abstract_txt:users in 3118) [ClassicSimilarity], result of:
           0.025663769 = score(doc=3118,freq=1.0), product of:
             0.13114446 = queryWeight, product of:
               2.0965443 = boost
               3.5783465 = idf(docFreq=3210, maxDocs=42306)
               0.01748089 = queryNorm
             0.19569083 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.5783465 = idf(docFreq=3210, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.027102012 = weight(abstract_txt:with in 3118) [ClassicSimilarity], result of:
           0.027102012 = score(doc=3118,freq=4.0), product of:
             0.098072626 = queryWeight, product of:
               2.2204888 = boost
               2.5265954 = idf(docFreq=9191, maxDocs=42306)
               0.01748089 = queryNorm
             0.27634636 = fieldWeight in 3118, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.5265954 = idf(docFreq=9191, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.10499438 = weight(abstract_txt:recall in 3118) [ClassicSimilarity], result of:
           0.10499438 = score(doc=3118,freq=1.0), product of:
             0.33546486 = queryWeight, product of:
               3.3531468 = boost
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.01748089 = queryNorm
             0.31298175 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.10948048 = weight(abstract_txt:precision in 3118) [ClassicSimilarity], result of:
           0.10948048 = score(doc=3118,freq=1.0), product of:
             0.36314192 = queryWeight, product of:
               3.768258 = boost
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.01748089 = queryNorm
             0.30148125 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
         0.23454216 = weight(abstract_txt:measures in 3118) [ClassicSimilarity], result of:
           0.23454216 = score(doc=3118,freq=3.0), product of:
             0.45499536 = queryWeight, product of:
               4.7827554 = boost
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.01748089 = queryNorm
             0.51548254 = fieldWeight in 3118, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3118)
       0.56 = coord(14/25)
   

2. Lesk, M.E.; Salton, G.: Relevance assements and retrieval system evaluation (1969) 0.38

   0.37713093 = sum of:
     0.37713093 = product of:
       1.1785342 = sum of:
         0.060764384 = weight(abstract_txt:effectiveness in 4220) [ClassicSimilarity], result of:
           0.060764384 = score(doc=4220,freq=2.0), product of:
             0.08950827 = queryWeight, product of:
               5.12035 = idf(docFreq=686, maxDocs=42306)
               0.01748089 = queryNorm
             0.6788689 = fieldWeight in 4220, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.12035 = idf(docFreq=686, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
         0.05587454 = weight(abstract_txt:criteria in 4220) [ClassicSimilarity], result of:
           0.05587454 = score(doc=4220,freq=1.0), product of:
             0.106639095 = queryWeight, product of:
               1.0915073 = boost
               5.588899 = idf(docFreq=429, maxDocs=42306)
               0.01748089 = queryNorm
             0.5239593 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.588899 = idf(docFreq=429, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
         0.011564711 = weight(abstract_txt:information in 4220) [ClassicSimilarity], result of:
           0.011564711 = score(doc=4220,freq=1.0), product of:
             0.05064194 = queryWeight, product of:
               1.1893059 = boost
               2.435865 = idf(docFreq=10064, maxDocs=42306)
               0.01748089 = queryNorm
             0.22836234 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.435865 = idf(docFreq=10064, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
         0.024776757 = weight(abstract_txt:data in 4220) [ClassicSimilarity], result of:
           0.024776757 = score(doc=4220,freq=1.0), product of:
             0.07812921 = queryWeight, product of:
               1.3212656 = boost
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.01748089 = queryNorm
             0.3171254 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
         0.058074865 = weight(abstract_txt:evaluation in 4220) [ClassicSimilarity], result of:
           0.058074865 = score(doc=4220,freq=1.0), product of:
             0.13786137 = queryWeight, product of:
               1.7551115 = boost
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.01748089 = queryNorm
             0.42125553 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
         0.35998073 = weight(abstract_txt:recall in 4220) [ClassicSimilarity], result of:
           0.35998073 = score(doc=4220,freq=4.0), product of:
             0.33546486 = queryWeight, product of:
               3.3531468 = boost
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.01748089 = queryNorm
             1.0730803 = fieldWeight in 4220, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
         0.37536162 = weight(abstract_txt:precision in 4220) [ClassicSimilarity], result of:
           0.37536162 = score(doc=4220,freq=4.0), product of:
             0.36314192 = queryWeight, product of:
               3.768258 = boost
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.01748089 = queryNorm
             1.0336499 = fieldWeight in 4220, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
         0.23213653 = weight(abstract_txt:measures in 4220) [ClassicSimilarity], result of:
           0.23213653 = score(doc=4220,freq=1.0), product of:
             0.45499536 = queryWeight, product of:
               4.7827554 = boost
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.01748089 = queryNorm
             0.5101954 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.09375 = fieldNorm(doc=4220)
       0.32 = coord(8/25)
   

3. Wildemuth, B.M.: Measures of success in searching a full-text fact base (1990) 0.33

   0.327471 = sum of:
     0.327471 = product of:
       1.1695392 = sum of:
         0.042914603 = weight(abstract_txt:data in 2119) [ClassicSimilarity], result of:
           0.042914603 = score(doc=2119,freq=3.0), product of:
             0.07812921 = queryWeight, product of:
               1.3212656 = boost
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.01748089 = queryNorm
             0.5492773 = fieldWeight in 2119, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.09375 = fieldNorm(doc=2119)
         0.032472398 = weight(abstract_txt:were in 2119) [ClassicSimilarity], result of:
           0.032472398 = score(doc=2119,freq=1.0), product of:
             0.09356786 = queryWeight, product of:
               1.4459283 = boost
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.01748089 = queryNorm
             0.3470465 = fieldWeight in 2119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.09375 = fieldNorm(doc=2119)
         0.12639311 = weight(abstract_txt:success in 2119) [ClassicSimilarity], result of:
           0.12639311 = score(doc=2119,freq=2.0), product of:
             0.166959 = queryWeight, product of:
               1.6727035 = boost
               5.7098846 = idf(docFreq=380, maxDocs=42306)
               0.01748089 = queryNorm
             0.75703084 = fieldWeight in 2119, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7098846 = idf(docFreq=380, maxDocs=42306)
               0.09375 = fieldNorm(doc=2119)
         0.058074865 = weight(abstract_txt:evaluation in 2119) [ClassicSimilarity], result of:
           0.058074865 = score(doc=2119,freq=1.0), product of:
             0.13786137 = queryWeight, product of:
               1.7551115 = boost
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.01748089 = queryNorm
             0.42125553 = fieldWeight in 2119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.09375 = fieldNorm(doc=2119)
         0.17999037 = weight(abstract_txt:recall in 2119) [ClassicSimilarity], result of:
           0.17999037 = score(doc=2119,freq=1.0), product of:
             0.33546486 = queryWeight, product of:
               3.3531468 = boost
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.01748089 = queryNorm
             0.53654015 = fieldWeight in 2119, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.09375 = fieldNorm(doc=2119)
         0.26542076 = weight(abstract_txt:precision in 2119) [ClassicSimilarity], result of:
           0.26542076 = score(doc=2119,freq=2.0), product of:
             0.36314192 = queryWeight, product of:
               3.768258 = boost
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.01748089 = queryNorm
             0.7309009 = fieldWeight in 2119, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.09375 = fieldNorm(doc=2119)
         0.46427307 = weight(abstract_txt:measures in 2119) [ClassicSimilarity], result of:
           0.46427307 = score(doc=2119,freq=4.0), product of:
             0.45499536 = queryWeight, product of:
               4.7827554 = boost
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.01748089 = queryNorm
             1.0203907 = fieldWeight in 2119, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.09375 = fieldNorm(doc=2119)
       0.28 = coord(7/25)
   

4. Huuskonen, S.; Vakkari, P.: Students' search process and outcome in Medline in writing an essay for a class on evidence-based medicine (2008) 0.32

   0.32292873 = sum of:
     0.32292873 = product of:
       0.8970242 = sum of:
         0.016524458 = weight(abstract_txt:information in 3892) [ClassicSimilarity], result of:
           0.016524458 = score(doc=3892,freq=6.0), product of:
             0.05064194 = queryWeight, product of:
               1.1893059 = boost
               2.435865 = idf(docFreq=10064, maxDocs=42306)
               0.01748089 = queryNorm
             0.32629988 = fieldWeight in 3892, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               2.435865 = idf(docFreq=10064, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.014453108 = weight(abstract_txt:data in 3892) [ClassicSimilarity], result of:
           0.014453108 = score(doc=3892,freq=1.0), product of:
             0.07812921 = queryWeight, product of:
               1.3212656 = boost
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.01748089 = queryNorm
             0.18498982 = fieldWeight in 3892, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.037884466 = weight(abstract_txt:were in 3892) [ClassicSimilarity], result of:
           0.037884466 = score(doc=3892,freq=4.0), product of:
             0.09356786 = queryWeight, product of:
               1.4459283 = boost
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.01748089 = queryNorm
             0.4048876 = fieldWeight in 3892, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.04790932 = weight(abstract_txt:evaluation in 3892) [ClassicSimilarity], result of:
           0.04790932 = score(doc=3892,freq=2.0), product of:
             0.13786137 = queryWeight, product of:
               1.7551115 = boost
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.01748089 = queryNorm
             0.3475181 = fieldWeight in 3892, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.036294047 = weight(abstract_txt:users in 3892) [ClassicSimilarity], result of:
           0.036294047 = score(doc=3892,freq=2.0), product of:
             0.13114446 = queryWeight, product of:
               2.0965443 = boost
               3.5783465 = idf(docFreq=3210, maxDocs=42306)
               0.01748089 = queryNorm
             0.2767486 = fieldWeight in 3892, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.5783465 = idf(docFreq=3210, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.027102012 = weight(abstract_txt:with in 3892) [ClassicSimilarity], result of:
           0.027102012 = score(doc=3892,freq=4.0), product of:
             0.098072626 = queryWeight, product of:
               2.2204888 = boost
               2.5265954 = idf(docFreq=9191, maxDocs=42306)
               0.01748089 = queryNorm
             0.27634636 = fieldWeight in 3892, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.5265954 = idf(docFreq=9191, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.25718266 = weight(abstract_txt:recall in 3892) [ClassicSimilarity], result of:
           0.25718266 = score(doc=3892,freq=6.0), product of:
             0.33546486 = queryWeight, product of:
               3.3531468 = boost
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.01748089 = queryNorm
             0.7666456 = fieldWeight in 3892, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.2681713 = weight(abstract_txt:precision in 3892) [ClassicSimilarity], result of:
           0.2681713 = score(doc=3892,freq=6.0), product of:
             0.36314192 = queryWeight, product of:
               3.768258 = boost
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.01748089 = queryNorm
             0.7384752 = fieldWeight in 3892, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
         0.19150288 = weight(abstract_txt:measures in 3892) [ClassicSimilarity], result of:
           0.19150288 = score(doc=3892,freq=2.0), product of:
             0.45499536 = queryWeight, product of:
               4.7827554 = boost
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.01748089 = queryNorm
             0.42088974 = fieldWeight in 3892, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.0546875 = fieldNorm(doc=3892)
       0.36 = coord(9/25)
   

5. Rowley, J.E.: CD-ROM versus online : an evaluation of the effects of the user interface on search effectiveness; a pilot study (1989) 0.31

   0.31435314 = sum of:
     0.31435314 = product of:
       0.98235357 = sum of:
         0.050128058 = weight(abstract_txt:effectiveness in 2942) [ClassicSimilarity], result of:
           0.050128058 = score(doc=2942,freq=1.0), product of:
             0.08950827 = queryWeight, product of:
               5.12035 = idf(docFreq=686, maxDocs=42306)
               0.01748089 = queryNorm
             0.56003827 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.12035 = idf(docFreq=686, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
         0.09790514 = weight(abstract_txt:satisfaction in 2942) [ClassicSimilarity], result of:
           0.09790514 = score(doc=2942,freq=1.0), product of:
             0.1398555 = queryWeight, product of:
               1.2499948 = boost
               6.4004107 = idf(docFreq=190, maxDocs=42306)
               0.01748089 = queryNorm
             0.70004493 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.4004107 = idf(docFreq=190, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
         0.028906217 = weight(abstract_txt:data in 2942) [ClassicSimilarity], result of:
           0.028906217 = score(doc=2942,freq=1.0), product of:
             0.07812921 = queryWeight, product of:
               1.3212656 = boost
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.01748089 = queryNorm
             0.36997965 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.382671 = idf(docFreq=3904, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
         0.037884466 = weight(abstract_txt:were in 2942) [ClassicSimilarity], result of:
           0.037884466 = score(doc=2942,freq=1.0), product of:
             0.09356786 = queryWeight, product of:
               1.4459283 = boost
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.01748089 = queryNorm
             0.4048876 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7018294 = idf(docFreq=2837, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
         0.06775401 = weight(abstract_txt:evaluation in 2942) [ClassicSimilarity], result of:
           0.06775401 = score(doc=2942,freq=1.0), product of:
             0.13786137 = queryWeight, product of:
               1.7551115 = boost
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.01748089 = queryNorm
             0.4914648 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4933925 = idf(docFreq=1285, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
         0.20998876 = weight(abstract_txt:recall in 2942) [ClassicSimilarity], result of:
           0.20998876 = score(doc=2942,freq=1.0), product of:
             0.33546486 = queryWeight, product of:
               3.3531468 = boost
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.01748089 = queryNorm
             0.6259635 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723095 = idf(docFreq=375, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
         0.21896096 = weight(abstract_txt:precision in 2942) [ClassicSimilarity], result of:
           0.21896096 = score(doc=2942,freq=1.0), product of:
             0.36314192 = queryWeight, product of:
               3.768258 = boost
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.01748089 = queryNorm
             0.6029625 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5127997 = idf(docFreq=463, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
         0.27082595 = weight(abstract_txt:measures in 2942) [ClassicSimilarity], result of:
           0.27082595 = score(doc=2942,freq=1.0), product of:
             0.45499536 = queryWeight, product of:
               4.7827554 = boost
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.01748089 = queryNorm
             0.59522796 = fieldWeight in 2942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4420843 = idf(docFreq=497, maxDocs=42306)
               0.109375 = fieldNorm(doc=2942)
       0.32 = coord(8/25)