Document (#27116)

Author

Su, L.T.

Title

¬A comprehensive and systematic model of user evaluation of Web search engines : Il. An evaluation by undergraduates

Source

Journal of the American Society for Information Science and technology. 54(2003) no.13, S.1193-1222

Year

2003

Abstract

This paper presents an application of the model described in Part I to the evaluation of Web search engines by undergraduates. The study observed how 36 undergraduate used four major search engines to find information for their own individual problems and how they evaluated these engines based an actual interaction with the search engines. User evaluation was based an 16 performance measures representing five evaluation criteria: relevance, efficiency, utility, user satisfaction, and connectivity. Non-performance (user-related) measures were also applied. Each participant searched his/ her own topic an all four engines and provided satisfaction ratings for system features and interaction and reasons for satisfaction. Each also made relevance judgements of retrieved items in relation to his/her own information need and participated in post-search Interviews to provide reactions to the search results and overall performance. The study found significant differences in precision PR1 relative recall, user satisfaction with output display, time saving, value of search results, and overall performance among the four engines and also significant engine by discipline interactions an all these measures. In addition, the study found significant differences in user satisfaction with response time among four engines, and significant engine by discipline interaction in user satisfaction with search interface. None of the four search engines dominated in every aspect of the multidimensional evaluation. Content analysis of verbal data identified a number of user criteria and users evaluative comments based an these criteria. Results from both quantitative analysis and content analysis provide insight for system design and development, and useful feedback an strengths and weaknesses of search engines for system improvement

Footnote

Teil I im selben Heft

Theme

Suchmaschinen
Benutzerstudien

Similar documents (content)

1. Su, L.T.; Chen, H.L.: Evaluation of Web search engines by undergraduate students (1999) 1.25

   1.2466737 = sum of:
     1.2466737 = product of:
       1.5583421 = sum of:
         0.014314278 = weight(abstract_txt:based in 544) [ClassicSimilarity], result of:
           0.014314278 = score(doc=544,freq=2.0), product of:
             0.057841625 = queryWeight, product of:
               3.1998224 = idf(docFreq=4826, maxDocs=43556)
               0.018076511 = queryNorm
             0.24747364 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1998224 = idf(docFreq=4826, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.014344918 = weight(abstract_txt:these in 544) [ClassicSimilarity], result of:
           0.014344918 = score(doc=544,freq=2.0), product of:
             0.057924137 = queryWeight, product of:
               1.000713 = boost
               3.2021039 = idf(docFreq=4815, maxDocs=43556)
               0.018076511 = queryNorm
             0.24765009 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.2021039 = idf(docFreq=4815, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.042854913 = weight(abstract_txt:relevance in 544) [ClassicSimilarity], result of:
           0.042854913 = score(doc=544,freq=3.0), product of:
             0.09169249 = queryWeight, product of:
               1.0280187 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.018076511 = queryNorm
             0.46737647 = fieldWeight in 544, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.025359994 = weight(abstract_txt:differences in 544) [ClassicSimilarity], result of:
           0.025359994 = score(doc=544,freq=1.0), product of:
             0.0932123 = queryWeight, product of:
               1.0365034 = boost
               4.9749403 = idf(docFreq=817, maxDocs=43556)
               0.018076511 = queryNorm
             0.27206704 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9749403 = idf(docFreq=817, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.016665775 = weight(abstract_txt:system in 544) [ClassicSimilarity], result of:
           0.016665775 = score(doc=544,freq=2.0), product of:
             0.064014435 = queryWeight, product of:
               1.0520072 = boost
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.018076511 = queryNorm
             0.26034403 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.017823933 = weight(abstract_txt:study in 544) [ClassicSimilarity], result of:
           0.017823933 = score(doc=544,freq=2.0), product of:
             0.06694681 = queryWeight, product of:
               1.0758326 = boost
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.018076511 = queryNorm
             0.2662402 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.018666673 = weight(abstract_txt:results in 544) [ClassicSimilarity], result of:
           0.018666673 = score(doc=544,freq=2.0), product of:
             0.06904075 = queryWeight, product of:
               1.0925279 = boost
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.018076511 = queryNorm
             0.27037182 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.033956062 = weight(abstract_txt:overall in 544) [ClassicSimilarity], result of:
           0.033956062 = score(doc=544,freq=1.0), product of:
             0.113236204 = queryWeight, product of:
               1.1424226 = boost
               5.483324 = idf(docFreq=491, maxDocs=43556)
               0.018076511 = queryNorm
             0.2998693 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.483324 = idf(docFreq=491, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.049007025 = weight(abstract_txt:engine in 544) [ClassicSimilarity], result of:
           0.049007025 = score(doc=544,freq=2.0), product of:
             0.11478082 = queryWeight, product of:
               1.1501878 = boost
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.018076511 = queryNorm
             0.42696178 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.044923775 = weight(abstract_txt:interaction in 544) [ClassicSimilarity], result of:
           0.044923775 = score(doc=544,freq=1.0), product of:
             0.15621462 = queryWeight, product of:
               1.6433898 = boost
               5.258556 = idf(docFreq=615, maxDocs=43556)
               0.018076511 = queryNorm
             0.28757727 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.258556 = idf(docFreq=615, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.085758135 = weight(abstract_txt:measures in 544) [ClassicSimilarity], result of:
           0.085758135 = score(doc=544,freq=3.0), product of:
             0.16667892 = queryWeight, product of:
               1.6975403 = boost
               5.4318275 = idf(docFreq=517, maxDocs=43556)
               0.018076511 = queryNorm
             0.514511 = fieldWeight in 544, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4318275 = idf(docFreq=517, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.053783573 = weight(abstract_txt:criteria in 544) [ClassicSimilarity], result of:
           0.053783573 = score(doc=544,freq=1.0), product of:
             0.1761316 = queryWeight, product of:
               1.7450119 = boost
               5.5837283 = idf(docFreq=444, maxDocs=43556)
               0.018076511 = queryNorm
             0.30536014 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5837283 = idf(docFreq=444, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.05807818 = weight(abstract_txt:performance in 544) [ClassicSimilarity], result of:
           0.05807818 = score(doc=544,freq=2.0), product of:
             0.16195056 = queryWeight, product of:
               1.9321479 = boost
               4.6368976 = idf(docFreq=1146, maxDocs=43556)
               0.018076511 = queryNorm
             0.35861674 = fieldWeight in 544, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6368976 = idf(docFreq=1146, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.045104068 = weight(abstract_txt:significant in 544) [ClassicSimilarity], result of:
           0.045104068 = score(doc=544,freq=1.0), product of:
             0.17239615 = queryWeight, product of:
               1.9934846 = boost
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.018076511 = queryNorm
             0.26163036 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.14533179 = weight(abstract_txt:four in 544) [ClassicSimilarity], result of:
           0.14533179 = score(doc=544,freq=4.0), product of:
             0.2552158 = queryWeight, product of:
               2.7118015 = boost
               5.20637 = idf(docFreq=648, maxDocs=43556)
               0.018076511 = queryNorm
             0.5694467 = fieldWeight in 544, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.20637 = idf(docFreq=648, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.09709341 = weight(abstract_txt:evaluation in 544) [ClassicSimilarity], result of:
           0.09709341 = score(doc=544,freq=3.0), product of:
             0.22812189 = queryWeight, product of:
               2.8085246 = boost
               4.49339 = idf(docFreq=1323, maxDocs=43556)
               0.018076511 = queryNorm
             0.42562076 = fieldWeight in 544, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.49339 = idf(docFreq=1323, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.041083373 = weight(abstract_txt:user in 544) [ClassicSimilarity], result of:
           0.041083373 = score(doc=544,freq=1.0), product of:
             0.20409757 = queryWeight, product of:
               3.0674891 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.018076511 = queryNorm
             0.20129281 = fieldWeight in 544, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.08706718 = weight(abstract_txt:search in 544) [ClassicSimilarity], result of:
           0.08706718 = score(doc=544,freq=3.0), product of:
             0.25151256 = queryWeight, product of:
               3.8071408 = boost
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.018076511 = queryNorm
             0.34617427 = fieldWeight in 544, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.32528764 = weight(abstract_txt:satisfaction in 544) [ClassicSimilarity], result of:
           0.32528764 = score(doc=544,freq=4.0), product of:
             0.46405834 = queryWeight, product of:
               4.0057244 = boost
               6.408803 = idf(docFreq=194, maxDocs=43556)
               0.018076511 = queryNorm
             0.70096284 = fieldWeight in 544, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.408803 = idf(docFreq=194, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
         0.34183744 = weight(abstract_txt:engines in 544) [ClassicSimilarity], result of:
           0.34183744 = score(doc=544,freq=5.0), product of:
             0.5279428 = queryWeight, product of:
               5.5158515 = boost
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.018076511 = queryNorm
             0.64748955 = fieldWeight in 544, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.0546875 = fieldNorm(doc=544)
       0.8 = coord(20/25)
   

2. Gluck, M.: Exploring the relationship between user satisfaction and relevance in information systems (1996) 0.66

   0.6575226 = sum of:
     0.6575226 = product of:
       1.095871 = sum of:
         0.016359175 = weight(abstract_txt:based in 4148) [ClassicSimilarity], result of:
           0.016359175 = score(doc=4148,freq=2.0), product of:
             0.057841625 = queryWeight, product of:
               3.1998224 = idf(docFreq=4826, maxDocs=43556)
               0.018076511 = queryNorm
             0.28282702 = fieldWeight in 4148, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1998224 = idf(docFreq=4826, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.011592444 = weight(abstract_txt:these in 4148) [ClassicSimilarity], result of:
           0.011592444 = score(doc=4148,freq=1.0), product of:
             0.057924137 = queryWeight, product of:
               1.000713 = boost
               3.2021039 = idf(docFreq=4815, maxDocs=43556)
               0.018076511 = queryNorm
             0.20013149 = fieldWeight in 4148, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.2021039 = idf(docFreq=4815, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.07481367 = weight(abstract_txt:relevance in 4148) [ClassicSimilarity], result of:
           0.07481367 = score(doc=4148,freq=7.0), product of:
             0.09169249 = queryWeight, product of:
               1.0280187 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.018076511 = queryNorm
             0.8159193 = fieldWeight in 4148, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.010510906 = weight(abstract_txt:with in 4148) [ClassicSimilarity], result of:
           0.010510906 = score(doc=4148,freq=2.0), product of:
             0.04740301 = queryWeight, product of:
               1.0453268 = boost
               2.508645 = idf(docFreq=9634, maxDocs=43556)
               0.018076511 = queryNorm
             0.22173499 = fieldWeight in 4148, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.508645 = idf(docFreq=9634, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.032989684 = weight(abstract_txt:system in 4148) [ClassicSimilarity], result of:
           0.032989684 = score(doc=4148,freq=6.0), product of:
             0.064014435 = queryWeight, product of:
               1.0520072 = boost
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.018076511 = queryNorm
             0.5153476 = fieldWeight in 4148, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.014044312 = weight(abstract_txt:also in 4148) [ClassicSimilarity], result of:
           0.014044312 = score(doc=4148,freq=1.0), product of:
             0.065827884 = queryWeight, product of:
               1.0668042 = boost
               3.4135838 = idf(docFreq=3897, maxDocs=43556)
               0.018076511 = queryNorm
             0.21334898 = fieldWeight in 4148, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4135838 = idf(docFreq=3897, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.014403914 = weight(abstract_txt:study in 4148) [ClassicSimilarity], result of:
           0.014403914 = score(doc=4148,freq=1.0), product of:
             0.06694681 = queryWeight, product of:
               1.0758326 = boost
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.018076511 = queryNorm
             0.21515459 = fieldWeight in 4148, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.015084952 = weight(abstract_txt:results in 4148) [ClassicSimilarity], result of:
           0.015084952 = score(doc=4148,freq=1.0), product of:
             0.06904075 = queryWeight, product of:
               1.0925279 = boost
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.018076511 = queryNorm
             0.21849345 = fieldWeight in 4148, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.05488128 = weight(abstract_txt:overall in 4148) [ClassicSimilarity], result of:
           0.05488128 = score(doc=4148,freq=2.0), product of:
             0.113236204 = queryWeight, product of:
               1.1424226 = boost
               5.483324 = idf(docFreq=491, maxDocs=43556)
               0.018076511 = queryNorm
             0.48466194 = fieldWeight in 4148, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.483324 = idf(docFreq=491, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.03486181 = weight(abstract_txt:analysis in 4148) [ClassicSimilarity], result of:
           0.03486181 = score(doc=4148,freq=4.0), product of:
             0.07602496 = queryWeight, product of:
               1.1464572 = boost
               3.6684597 = idf(docFreq=3020, maxDocs=43556)
               0.018076511 = queryNorm
             0.45855746 = fieldWeight in 4148, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.6684597 = idf(docFreq=3020, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.11317139 = weight(abstract_txt:measures in 4148) [ClassicSimilarity], result of:
           0.11317139 = score(doc=4148,freq=4.0), product of:
             0.16667892 = queryWeight, product of:
               1.6975403 = boost
               5.4318275 = idf(docFreq=517, maxDocs=43556)
               0.018076511 = queryNorm
             0.67897844 = fieldWeight in 4148, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.4318275 = idf(docFreq=517, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.08129252 = weight(abstract_txt:performance in 4148) [ClassicSimilarity], result of:
           0.08129252 = score(doc=4148,freq=3.0), product of:
             0.16195056 = queryWeight, product of:
               1.9321479 = boost
               4.6368976 = idf(docFreq=1146, maxDocs=43556)
               0.018076511 = queryNorm
             0.50195885 = fieldWeight in 4148, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.6368976 = idf(docFreq=1146, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.0515475 = weight(abstract_txt:significant in 4148) [ClassicSimilarity], result of:
           0.0515475 = score(doc=4148,freq=1.0), product of:
             0.17239615 = queryWeight, product of:
               1.9934846 = boost
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.018076511 = queryNorm
             0.2990061 = fieldWeight in 4148, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.11500949 = weight(abstract_txt:user in 4148) [ClassicSimilarity], result of:
           0.11500949 = score(doc=4148,freq=6.0), product of:
             0.20409757 = queryWeight, product of:
               3.0674891 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.018076511 = queryNorm
             0.5635025 = fieldWeight in 4148, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
         0.45530787 = weight(abstract_txt:satisfaction in 4148) [ClassicSimilarity], result of:
           0.45530787 = score(doc=4148,freq=6.0), product of:
             0.46405834 = queryWeight, product of:
               4.0057244 = boost
               6.408803 = idf(docFreq=194, maxDocs=43556)
               0.018076511 = queryNorm
             0.9811436 = fieldWeight in 4148, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.408803 = idf(docFreq=194, maxDocs=43556)
               0.0625 = fieldNorm(doc=4148)
       0.6 = coord(15/25)
   

3. Su, L.T.: ¬A comprehensive and systematic model of user evaluation of Web search engines : I. Theory and background (2003) 0.63

   0.6301015 = sum of:
     0.6301015 = product of:
       1.1251812 = sum of:
         0.028276911 = weight(abstract_txt:relevance in 162) [ClassicSimilarity], result of:
           0.028276911 = score(doc=162,freq=1.0), product of:
             0.09169249 = queryWeight, product of:
               1.0280187 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.018076511 = queryNorm
             0.3083885 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.0190466 = weight(abstract_txt:system in 162) [ClassicSimilarity], result of:
           0.0190466 = score(doc=162,freq=2.0), product of:
             0.064014435 = queryWeight, product of:
               1.0520072 = boost
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.018076511 = queryNorm
             0.29753605 = fieldWeight in 162, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.015084952 = weight(abstract_txt:results in 162) [ClassicSimilarity], result of:
           0.015084952 = score(doc=162,freq=1.0), product of:
             0.06904075 = queryWeight, product of:
               1.0925279 = boost
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.018076511 = queryNorm
             0.21849345 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.017430905 = weight(abstract_txt:analysis in 162) [ClassicSimilarity], result of:
           0.017430905 = score(doc=162,freq=1.0), product of:
             0.07602496 = queryWeight, product of:
               1.1464572 = boost
               3.6684597 = idf(docFreq=3020, maxDocs=43556)
               0.018076511 = queryNorm
             0.22927873 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6684597 = idf(docFreq=3020, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.039603654 = weight(abstract_txt:engine in 162) [ClassicSimilarity], result of:
           0.039603654 = score(doc=162,freq=1.0), product of:
             0.11478082 = queryWeight, product of:
               1.1501878 = boost
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.018076511 = queryNorm
             0.34503722 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.1130676 = weight(abstract_txt:undergraduates in 162) [ClassicSimilarity], result of:
           0.1130676 = score(doc=162,freq=1.0), product of:
             0.23099607 = queryWeight, product of:
               1.6316854 = boost
               7.831655 = idf(docFreq=46, maxDocs=43556)
               0.018076511 = queryNorm
             0.48947844 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.831655 = idf(docFreq=46, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.056585696 = weight(abstract_txt:measures in 162) [ClassicSimilarity], result of:
           0.056585696 = score(doc=162,freq=1.0), product of:
             0.16667892 = queryWeight, product of:
               1.6975403 = boost
               5.4318275 = idf(docFreq=517, maxDocs=43556)
               0.018076511 = queryNorm
             0.33948922 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4318275 = idf(docFreq=517, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.06146694 = weight(abstract_txt:criteria in 162) [ClassicSimilarity], result of:
           0.06146694 = score(doc=162,freq=1.0), product of:
             0.1761316 = queryWeight, product of:
               1.7450119 = boost
               5.5837283 = idf(docFreq=444, maxDocs=43556)
               0.018076511 = queryNorm
             0.34898302 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5837283 = idf(docFreq=444, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.08304674 = weight(abstract_txt:four in 162) [ClassicSimilarity], result of:
           0.08304674 = score(doc=162,freq=1.0), product of:
             0.2552158 = queryWeight, product of:
               2.7118015 = boost
               5.20637 = idf(docFreq=648, maxDocs=43556)
               0.018076511 = queryNorm
             0.32539812 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.20637 = idf(docFreq=648, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.1109639 = weight(abstract_txt:evaluation in 162) [ClassicSimilarity], result of:
           0.1109639 = score(doc=162,freq=3.0), product of:
             0.22812189 = queryWeight, product of:
               2.8085246 = boost
               4.49339 = idf(docFreq=1323, maxDocs=43556)
               0.018076511 = queryNorm
             0.48642373 = fieldWeight in 162, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.49339 = idf(docFreq=1323, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.06640076 = weight(abstract_txt:user in 162) [ClassicSimilarity], result of:
           0.06640076 = score(doc=162,freq=2.0), product of:
             0.20409757 = queryWeight, product of:
               3.0674891 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.018076511 = queryNorm
             0.3253383 = fieldWeight in 162, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.08124578 = weight(abstract_txt:search in 162) [ClassicSimilarity], result of:
           0.08124578 = score(doc=162,freq=2.0), product of:
             0.25151256 = queryWeight, product of:
               3.8071408 = boost
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.018076511 = queryNorm
             0.3230287 = fieldWeight in 162, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.18587865 = weight(abstract_txt:satisfaction in 162) [ClassicSimilarity], result of:
           0.18587865 = score(doc=162,freq=1.0), product of:
             0.46405834 = queryWeight, product of:
               4.0057244 = boost
               6.408803 = idf(docFreq=194, maxDocs=43556)
               0.018076511 = queryNorm
             0.4005502 = fieldWeight in 162, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.408803 = idf(docFreq=194, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
         0.24708223 = weight(abstract_txt:engines in 162) [ClassicSimilarity], result of:
           0.24708223 = score(doc=162,freq=2.0), product of:
             0.5279428 = queryWeight, product of:
               5.5158515 = boost
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.018076511 = queryNorm
             0.4680095 = fieldWeight in 162, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.0625 = fieldNorm(doc=162)
       0.56 = coord(14/25)
   

4. Spink, A.; Jansen, B.J.; Blakely, C.; Koshman, S.: ¬A study of results overlap and uniqueness among major Web search engines (2006) 0.53

   0.52974594 = sum of:
     0.52974594 = product of:
       1.203968 = sum of:
         0.010143388 = weight(abstract_txt:these in 2991) [ClassicSimilarity], result of:
           0.010143388 = score(doc=2991,freq=1.0), product of:
             0.057924137 = queryWeight, product of:
               1.000713 = boost
               3.2021039 = idf(docFreq=4815, maxDocs=43556)
               0.018076511 = queryNorm
             0.17511505 = fieldWeight in 2991, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.2021039 = idf(docFreq=4815, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.006503292 = weight(abstract_txt:with in 2991) [ClassicSimilarity], result of:
           0.006503292 = score(doc=2991,freq=1.0), product of:
             0.04740301 = queryWeight, product of:
               1.0453268 = boost
               2.508645 = idf(docFreq=9634, maxDocs=43556)
               0.018076511 = queryNorm
             0.13719153 = fieldWeight in 2991, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.508645 = idf(docFreq=9634, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.028182114 = weight(abstract_txt:study in 2991) [ClassicSimilarity], result of:
           0.028182114 = score(doc=2991,freq=5.0), product of:
             0.06694681 = queryWeight, product of:
               1.0758326 = boost
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.018076511 = queryNorm
             0.42096275 = fieldWeight in 2991, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.039598 = weight(abstract_txt:results in 2991) [ClassicSimilarity], result of:
           0.039598 = score(doc=2991,freq=9.0), product of:
             0.06904075 = queryWeight, product of:
               1.0925279 = boost
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.018076511 = queryNorm
             0.5735453 = fieldWeight in 2991, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.049007025 = weight(abstract_txt:engine in 2991) [ClassicSimilarity], result of:
           0.049007025 = score(doc=2991,freq=2.0), product of:
             0.11478082 = queryWeight, product of:
               1.1501878 = boost
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.018076511 = queryNorm
             0.42696178 = fieldWeight in 2991, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.041067477 = weight(abstract_txt:performance in 2991) [ClassicSimilarity], result of:
           0.041067477 = score(doc=2991,freq=1.0), product of:
             0.16195056 = queryWeight, product of:
               1.9321479 = boost
               4.6368976 = idf(docFreq=1146, maxDocs=43556)
               0.018076511 = queryNorm
             0.25358033 = fieldWeight in 2991, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6368976 = idf(docFreq=1146, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.063786775 = weight(abstract_txt:significant in 2991) [ClassicSimilarity], result of:
           0.063786775 = score(doc=2991,freq=2.0), product of:
             0.17239615 = queryWeight, product of:
               1.9934846 = boost
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.018076511 = queryNorm
             0.37000117 = fieldWeight in 2991, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.14533179 = weight(abstract_txt:four in 2991) [ClassicSimilarity], result of:
           0.14533179 = score(doc=2991,freq=4.0), product of:
             0.2552158 = queryWeight, product of:
               2.7118015 = boost
               5.20637 = idf(docFreq=648, maxDocs=43556)
               0.018076511 = queryNorm
             0.5694467 = fieldWeight in 2991, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.20637 = idf(docFreq=648, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.041083373 = weight(abstract_txt:user in 2991) [ClassicSimilarity], result of:
           0.041083373 = score(doc=2991,freq=1.0), product of:
             0.20409757 = queryWeight, product of:
               3.0674891 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.018076511 = queryNorm
             0.20129281 = fieldWeight in 2991, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.20726134 = weight(abstract_txt:search in 2991) [ClassicSimilarity], result of:
           0.20726134 = score(doc=2991,freq=17.0), product of:
             0.25151256 = queryWeight, product of:
               3.8071408 = boost
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.018076511 = queryNorm
             0.8240596 = fieldWeight in 2991, product of:
               4.1231055 = tf(freq=17.0), with freq of:
                 17.0 = termFreq=17.0
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
         0.5720034 = weight(abstract_txt:engines in 2991) [ClassicSimilarity], result of:
           0.5720034 = score(doc=2991,freq=14.0), product of:
             0.5279428 = queryWeight, product of:
               5.5158515 = boost
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.018076511 = queryNorm
             1.0834572 = fieldWeight in 2991, product of:
               3.7416575 = tf(freq=14.0), with freq of:
                 14.0 = termFreq=14.0
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.0546875 = fieldNorm(doc=2991)
       0.44 = coord(11/25)
   

5. Can, F.; Nuray, R.; Sevdik, A.B.: Automatic performance evaluation of Web search engines (2004) 0.47

   0.47104582 = sum of:
     0.47104582 = product of:
       1.0705587 = sum of:
         0.020448968 = weight(abstract_txt:based in 3568) [ClassicSimilarity], result of:
           0.020448968 = score(doc=3568,freq=2.0), product of:
             0.057841625 = queryWeight, product of:
               3.1998224 = idf(docFreq=4826, maxDocs=43556)
               0.018076511 = queryNorm
             0.35353377 = fieldWeight in 3568, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1998224 = idf(docFreq=4826, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.049986985 = weight(abstract_txt:relevance in 3568) [ClassicSimilarity], result of:
           0.049986985 = score(doc=3568,freq=2.0), product of:
             0.09169249 = queryWeight, product of:
               1.0280187 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.018076511 = queryNorm
             0.545159 = fieldWeight in 3568, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.013138632 = weight(abstract_txt:with in 3568) [ClassicSimilarity], result of:
           0.013138632 = score(doc=3568,freq=2.0), product of:
             0.04740301 = queryWeight, product of:
               1.0453268 = boost
               2.508645 = idf(docFreq=9634, maxDocs=43556)
               0.018076511 = queryNorm
             0.27716872 = fieldWeight in 3568, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.508645 = idf(docFreq=9634, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.018004892 = weight(abstract_txt:study in 3568) [ClassicSimilarity], result of:
           0.018004892 = score(doc=3568,freq=1.0), product of:
             0.06694681 = queryWeight, product of:
               1.0758326 = boost
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.018076511 = queryNorm
             0.26894325 = fieldWeight in 3568, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4424734 = idf(docFreq=3786, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.01885619 = weight(abstract_txt:results in 3568) [ClassicSimilarity], result of:
           0.01885619 = score(doc=3568,freq=1.0), product of:
             0.06904075 = queryWeight, product of:
               1.0925279 = boost
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.018076511 = queryNorm
             0.2731168 = fieldWeight in 3568, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4958951 = idf(docFreq=3589, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.04950457 = weight(abstract_txt:engine in 3568) [ClassicSimilarity], result of:
           0.04950457 = score(doc=3568,freq=1.0), product of:
             0.11478082 = queryWeight, product of:
               1.1501878 = boost
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.018076511 = queryNorm
             0.43129653 = fieldWeight in 3568, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5205956 = idf(docFreq=473, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.06443438 = weight(abstract_txt:significant in 3568) [ClassicSimilarity], result of:
           0.06443438 = score(doc=3568,freq=1.0), product of:
             0.17239615 = queryWeight, product of:
               1.9934846 = boost
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.018076511 = queryNorm
             0.37375763 = fieldWeight in 3568, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7840977 = idf(docFreq=989, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.113252066 = weight(abstract_txt:evaluation in 3568) [ClassicSimilarity], result of:
           0.113252066 = score(doc=3568,freq=2.0), product of:
             0.22812189 = queryWeight, product of:
               2.8085246 = boost
               4.49339 = idf(docFreq=1323, maxDocs=43556)
               0.018076511 = queryNorm
             0.49645418 = fieldWeight in 3568, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.49339 = idf(docFreq=1323, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.058690533 = weight(abstract_txt:user in 3568) [ClassicSimilarity], result of:
           0.058690533 = score(doc=3568,freq=1.0), product of:
             0.20409757 = queryWeight, product of:
               3.0674891 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.018076511 = queryNorm
             0.28756115 = fieldWeight in 3568, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.17590226 = weight(abstract_txt:search in 3568) [ClassicSimilarity], result of:
           0.17590226 = score(doc=3568,freq=6.0), product of:
             0.25151256 = queryWeight, product of:
               3.8071408 = boost
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.018076511 = queryNorm
             0.69937766 = fieldWeight in 3568, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.6546526 = idf(docFreq=3062, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
         0.4883392 = weight(abstract_txt:engines in 3568) [ClassicSimilarity], result of:
           0.4883392 = score(doc=3568,freq=5.0), product of:
             0.5279428 = queryWeight, product of:
               5.5158515 = boost
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.018076511 = queryNorm
             0.92498505 = fieldWeight in 3568, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.2949233 = idf(docFreq=593, maxDocs=43556)
               0.078125 = fieldNorm(doc=3568)
       0.44 = coord(11/25)