Document (#40072)

Author

Zhitomirsky-Geffet, M.
Bar-Ilan, J.
Levene, M.

Title

Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings

Source

Aslib journal of information management. 68(2016) no.4, S.407-427

Year

2016

Abstract

Purpose - One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the "wisdom of crowds". The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements. Design/methodology/approach - In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated "wisdom of crowds" judgements were used as a means to compare the differences between human assessments of search results and search engine's rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties. Findings - It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings. Practical implications - The proposed "wisdom of crowds"-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine's ranking over time to changes in users preferences. Originality/value - This is a first study that applies the notion of "wisdom of crowds" to examine an under-explored in the literature phenomenon of "change in time" in user evaluation of relevance.

Content

Vgl.: http://www.emeraldinsight.com/doi/abs/10.1108/AJIM-10-2015-0165.

Similar documents (author)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 5.50

   5.5016513 = sum of:
     5.5016513 = sum of:
       1.0574923 = weight(author_txt:ilan in 3593) [ClassicSimilarity], result of:
         1.0574923 = score(doc=3593,freq=1.0), product of:
           0.41849965 = queryWeight, product of:
             8.085969 = idf(docFreq=36, maxDocs=44218)
             0.051756278 = queryNorm
           2.5268652 = fieldWeight in 3593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.085969 = idf(docFreq=36, maxDocs=44218)
             0.3125 = fieldNorm(doc=3593)
       1.3929076 = weight(author_txt:geffet in 3593) [ClassicSimilarity], result of:
         1.3929076 = score(doc=3593,freq=1.0), product of:
           0.50287324 = queryWeight, product of:
             1.0961796 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.051756278 = queryNorm
           2.7698982 = fieldWeight in 3593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.3125 = fieldNorm(doc=3593)
       1.3929076 = weight(author_txt:zhitomirsky in 3593) [ClassicSimilarity], result of:
         1.3929076 = score(doc=3593,freq=1.0), product of:
           0.50287324 = queryWeight, product of:
             1.0961796 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.051756278 = queryNorm
           2.7698982 = fieldWeight in 3593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.3125 = fieldNorm(doc=3593)
       1.6583438 = weight(author_txt:levene in 3593) [ClassicSimilarity], result of:
         1.6583438 = score(doc=3593,freq=1.0), product of:
           0.56488496 = queryWeight, product of:
             1.1618029 = boost
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.051756278 = queryNorm
           2.9357195 = fieldWeight in 3593, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.3125 = fieldNorm(doc=3593)
   

2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Categorical relevance judgment (2018) 5.50

   5.5016513 = sum of:
     5.5016513 = sum of:
       1.0574923 = weight(author_txt:ilan in 4457) [ClassicSimilarity], result of:
         1.0574923 = score(doc=4457,freq=1.0), product of:
           0.41849965 = queryWeight, product of:
             8.085969 = idf(docFreq=36, maxDocs=44218)
             0.051756278 = queryNorm
           2.5268652 = fieldWeight in 4457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.085969 = idf(docFreq=36, maxDocs=44218)
             0.3125 = fieldNorm(doc=4457)
       1.3929076 = weight(author_txt:geffet in 4457) [ClassicSimilarity], result of:
         1.3929076 = score(doc=4457,freq=1.0), product of:
           0.50287324 = queryWeight, product of:
             1.0961796 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.051756278 = queryNorm
           2.7698982 = fieldWeight in 4457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.3125 = fieldNorm(doc=4457)
       1.3929076 = weight(author_txt:zhitomirsky in 4457) [ClassicSimilarity], result of:
         1.3929076 = score(doc=4457,freq=1.0), product of:
           0.50287324 = queryWeight, product of:
             1.0961796 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.051756278 = queryNorm
           2.7698982 = fieldWeight in 4457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.3125 = fieldNorm(doc=4457)
       1.6583438 = weight(author_txt:levene in 4457) [ClassicSimilarity], result of:
         1.6583438 = score(doc=4457,freq=1.0), product of:
           0.56488496 = queryWeight, product of:
             1.1618029 = boost
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.051756278 = queryNorm
           2.9357195 = fieldWeight in 4457, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.3125 = fieldNorm(doc=4457)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.: Towards maximal unification of semantically diverse ontologies for controversial domains (2014) 3.46

   3.4589767 = sum of:
     3.4589767 = product of:
       4.611969 = sum of:
         1.2689908 = weight(author_txt:ilan in 1634) [ClassicSimilarity], result of:
           1.2689908 = score(doc=1634,freq=1.0), product of:
             0.41849965 = queryWeight, product of:
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.051756278 = queryNorm
             3.0322385 = fieldWeight in 1634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.375 = fieldNorm(doc=1634)
         1.6714892 = weight(author_txt:geffet in 1634) [ClassicSimilarity], result of:
           1.6714892 = score(doc=1634,freq=1.0), product of:
             0.50287324 = queryWeight, product of:
               1.0961796 = boost
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.051756278 = queryNorm
             3.3238778 = fieldWeight in 1634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.375 = fieldNorm(doc=1634)
         1.6714892 = weight(author_txt:zhitomirsky in 1634) [ClassicSimilarity], result of:
           1.6714892 = score(doc=1634,freq=1.0), product of:
             0.50287324 = queryWeight, product of:
               1.0961796 = boost
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.051756278 = queryNorm
             3.3238778 = fieldWeight in 1634, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.375 = fieldNorm(doc=1634)
       0.75 = coord(3/4)
   

4. Bar-Ilan, J.; Zhitomirsky-Geffet, M.; Miller, Y.; Shoham, S.: ¬The effects of background information and social interaction on image tagging (2010) 2.88

   2.8824806 = sum of:
     2.8824806 = product of:
       3.8433075 = sum of:
         1.0574923 = weight(author_txt:ilan in 3453) [ClassicSimilarity], result of:
           1.0574923 = score(doc=3453,freq=1.0), product of:
             0.41849965 = queryWeight, product of:
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.051756278 = queryNorm
             2.5268652 = fieldWeight in 3453, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.3125 = fieldNorm(doc=3453)
         1.3929076 = weight(author_txt:geffet in 3453) [ClassicSimilarity], result of:
           1.3929076 = score(doc=3453,freq=1.0), product of:
             0.50287324 = queryWeight, product of:
               1.0961796 = boost
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.051756278 = queryNorm
             2.7698982 = fieldWeight in 3453, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.3125 = fieldNorm(doc=3453)
         1.3929076 = weight(author_txt:zhitomirsky in 3453) [ClassicSimilarity], result of:
           1.3929076 = score(doc=3453,freq=1.0), product of:
             0.50287324 = queryWeight, product of:
               1.0961796 = boost
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.051756278 = queryNorm
             2.7698982 = fieldWeight in 3453, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.3125 = fieldNorm(doc=3453)
       0.75 = coord(3/4)
   

5. Zhitomirsky-Geffet, M.; Erez, E.S.; Bar-Ilan, J.: Toward multiviewpoint ontology construction by collaboration of non-experts and crowdsourcing : the case of the effect of diet on health (2017) 2.88

   2.8824806 = sum of:
     2.8824806 = product of:
       3.8433075 = sum of:
         1.0574923 = weight(author_txt:ilan in 3439) [ClassicSimilarity], result of:
           1.0574923 = score(doc=3439,freq=1.0), product of:
             0.41849965 = queryWeight, product of:
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.051756278 = queryNorm
             2.5268652 = fieldWeight in 3439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.3125 = fieldNorm(doc=3439)
         1.3929076 = weight(author_txt:geffet in 3439) [ClassicSimilarity], result of:
           1.3929076 = score(doc=3439,freq=1.0), product of:
             0.50287324 = queryWeight, product of:
               1.0961796 = boost
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.051756278 = queryNorm
             2.7698982 = fieldWeight in 3439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.3125 = fieldNorm(doc=3439)
         1.3929076 = weight(author_txt:zhitomirsky in 3439) [ClassicSimilarity], result of:
           1.3929076 = score(doc=3439,freq=1.0), product of:
             0.50287324 = queryWeight, product of:
               1.0961796 = boost
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.051756278 = queryNorm
             2.7698982 = fieldWeight in 3439, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.3125 = fieldNorm(doc=3439)
       0.75 = coord(3/4)
   

Similar documents (content)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.60

   0.59519583 = sum of:
     0.59519583 = product of:
       1.0628496 = sum of:
         0.03420738 = weight(abstract_txt:ranking in 3593) [ClassicSimilarity], result of:
           0.03420738 = score(doc=3593,freq=2.0), product of:
             0.069124 = queryWeight, product of:
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.012346189 = queryNorm
             0.49486983 = fieldWeight in 3593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.018664056 = weight(abstract_txt:evaluation in 3593) [ClassicSimilarity], result of:
           0.018664056 = score(doc=3593,freq=1.0), product of:
             0.06656717 = queryWeight, product of:
               1.2018803 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.012346189 = queryNorm
             0.2803793 = fieldWeight in 3593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.022126073 = weight(abstract_txt:study in 3593) [ClassicSimilarity], result of:
           0.022126073 = score(doc=3593,freq=4.0), product of:
             0.051699363 = queryWeight, product of:
               1.223047 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.012346189 = queryNorm
             0.42797574 = fieldWeight in 3593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.026030121 = weight(abstract_txt:results in 3593) [ClassicSimilarity], result of:
           0.026030121 = score(doc=3593,freq=5.0), product of:
             0.053484727 = queryWeight, product of:
               1.2439859 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.012346189 = queryNorm
             0.48668328 = fieldWeight in 3593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.010061906 = weight(abstract_txt:this in 3593) [ClassicSimilarity], result of:
           0.010061906 = score(doc=3593,freq=3.0), product of:
             0.038519368 = queryWeight, product of:
               1.2929621 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.012346189 = queryNorm
             0.2612168 = fieldWeight in 3593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.049598355 = weight(abstract_txt:relevance in 3593) [ClassicSimilarity], result of:
           0.049598355 = score(doc=3593,freq=4.0), product of:
             0.080453984 = queryWeight, product of:
               1.3213108 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012346189 = queryNorm
             0.616481 = fieldWeight in 3593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.041218974 = weight(abstract_txt:point in 3593) [ClassicSimilarity], result of:
           0.041218974 = score(doc=3593,freq=2.0), product of:
             0.089600585 = queryWeight, product of:
               1.3943976 = boost
               5.2046475 = idf(docFreq=659, maxDocs=44218)
               0.012346189 = queryNorm
             0.4600302 = fieldWeight in 3593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2046475 = idf(docFreq=659, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.05880711 = weight(abstract_txt:scale in 3593) [ClassicSimilarity], result of:
           0.05880711 = score(doc=3593,freq=3.0), product of:
             0.099197835 = queryWeight, product of:
               1.4671762 = boost
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.012346189 = queryNorm
             0.59282655 = fieldWeight in 3593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.476297 = idf(docFreq=502, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.021077638 = weight(abstract_txt:over in 3593) [ClassicSimilarity], result of:
           0.021077638 = score(doc=3593,freq=1.0), product of:
             0.07945421 = queryWeight, product of:
               1.5162089 = boost
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.012346189 = queryNorm
             0.2652803 = fieldWeight in 3593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.067462705 = weight(abstract_txt:rankings in 3593) [ClassicSimilarity], result of:
           0.067462705 = score(doc=3593,freq=1.0), product of:
             0.15678285 = queryWeight, product of:
               1.8445077 = boost
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.012346189 = queryNorm
             0.43029392 = fieldWeight in 3593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.046208393 = weight(abstract_txt:user in 3593) [ClassicSimilarity], result of:
           0.046208393 = score(doc=3593,freq=5.0), product of:
             0.08976147 = queryWeight, product of:
               1.9737456 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.012346189 = queryNorm
             0.51479095 = fieldWeight in 3593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.04174202 = weight(abstract_txt:time in 3593) [ClassicSimilarity], result of:
           0.04174202 = score(doc=3593,freq=2.0), product of:
             0.11384266 = queryWeight, product of:
               2.2227914 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.012346189 = queryNorm
             0.36666414 = fieldWeight in 3593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.052582335 = weight(abstract_txt:search in 3593) [ClassicSimilarity], result of:
           0.052582335 = score(doc=3593,freq=3.0), product of:
             0.13278508 = queryWeight, product of:
               2.9401295 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.012346189 = queryNorm
             0.3959958 = fieldWeight in 3593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
         0.5730626 = weight(abstract_txt:judgements in 3593) [ClassicSimilarity], result of:
           0.5730626 = score(doc=3593,freq=5.0), product of:
             0.5505218 = queryWeight, product of:
               5.986584 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.012346189 = queryNorm
             1.0409445 = fieldWeight in 3593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.0625 = fieldNorm(doc=3593)
       0.56 = coord(14/25)
   

2. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.21

   0.21232039 = sum of:
     0.21232039 = product of:
       0.7582871 = sum of:
         0.028742611 = weight(abstract_txt:study in 3268) [ClassicSimilarity], result of:
           0.028742611 = score(doc=3268,freq=3.0), product of:
             0.051699363 = queryWeight, product of:
               1.223047 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.012346189 = queryNorm
             0.5559568 = fieldWeight in 3268, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.017461538 = weight(abstract_txt:results in 3268) [ClassicSimilarity], result of:
           0.017461538 = score(doc=3268,freq=1.0), product of:
             0.053484727 = queryWeight, product of:
               1.2439859 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.012346189 = queryNorm
             0.32647708 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.008713867 = weight(abstract_txt:this in 3268) [ClassicSimilarity], result of:
           0.008713867 = score(doc=3268,freq=1.0), product of:
             0.038519368 = queryWeight, product of:
               1.2929621 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.012346189 = queryNorm
             0.2262204 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.08317897 = weight(abstract_txt:relevance in 3268) [ClassicSimilarity], result of:
           0.08317897 = score(doc=3268,freq=5.0), product of:
             0.080453984 = queryWeight, product of:
               1.3213108 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012346189 = queryNorm
             1.0338701 = fieldWeight in 3268, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.030997533 = weight(abstract_txt:user in 3268) [ClassicSimilarity], result of:
           0.030997533 = score(doc=3268,freq=1.0), product of:
             0.08976147 = queryWeight, product of:
               1.9737456 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.012346189 = queryNorm
             0.34533226 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.045537636 = weight(abstract_txt:search in 3268) [ClassicSimilarity], result of:
           0.045537636 = score(doc=3268,freq=1.0), product of:
             0.13278508 = queryWeight, product of:
               2.9401295 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.012346189 = queryNorm
             0.34294242 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.5436549 = weight(abstract_txt:judgements in 3268) [ClassicSimilarity], result of:
           0.5436549 = score(doc=3268,freq=2.0), product of:
             0.5505218 = queryWeight, product of:
               5.986584 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.012346189 = queryNorm
             0.9875266 = fieldWeight in 3268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
       0.28 = coord(7/25)
   

3. Su, L.T.: ¬A comprehensive and systematic model of user evaluation of Web search engines : Il. An evaluation by undergraduates (2003) 0.21

   0.21044503 = sum of:
     0.21044503 = product of:
       0.52611256 = sum of:
         0.032662097 = weight(abstract_txt:evaluation in 2117) [ClassicSimilarity], result of:
           0.032662097 = score(doc=2117,freq=4.0), product of:
             0.06656717 = queryWeight, product of:
               1.2018803 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.012346189 = queryNorm
             0.49066377 = fieldWeight in 2117, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.016766524 = weight(abstract_txt:study in 2117) [ClassicSimilarity], result of:
           0.016766524 = score(doc=2117,freq=3.0), product of:
             0.051699363 = queryWeight, product of:
               1.223047 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.012346189 = queryNorm
             0.32430813 = fieldWeight in 2117, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.01764249 = weight(abstract_txt:results in 2117) [ClassicSimilarity], result of:
           0.01764249 = score(doc=2117,freq=3.0), product of:
             0.053484727 = queryWeight, product of:
               1.2439859 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.012346189 = queryNorm
             0.32986036 = fieldWeight in 2117, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.0050830888 = weight(abstract_txt:this in 2117) [ClassicSimilarity], result of:
           0.0050830888 = score(doc=2117,freq=1.0), product of:
             0.038519368 = queryWeight, product of:
               1.2929621 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.012346189 = queryNorm
             0.1319619 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.030687414 = weight(abstract_txt:relevance in 2117) [ClassicSimilarity], result of:
           0.030687414 = score(doc=2117,freq=2.0), product of:
             0.080453984 = queryWeight, product of:
               1.3213108 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012346189 = queryNorm
             0.38142815 = fieldWeight in 2117, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.047840197 = weight(abstract_txt:user in 2117) [ClassicSimilarity], result of:
           0.047840197 = score(doc=2117,freq=7.0), product of:
             0.08976147 = queryWeight, product of:
               1.9737456 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.012346189 = queryNorm
             0.53297025 = fieldWeight in 2117, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.034969445 = weight(abstract_txt:individual in 2117) [ClassicSimilarity], result of:
           0.034969445 = score(doc=2117,freq=1.0), product of:
             0.13111712 = queryWeight, product of:
               2.177636 = boost
               4.8768706 = idf(docFreq=915, maxDocs=44218)
               0.012346189 = queryNorm
             0.26670387 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.8768706 = idf(docFreq=915, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.036524266 = weight(abstract_txt:time in 2117) [ClassicSimilarity], result of:
           0.036524266 = score(doc=2117,freq=2.0), product of:
             0.11384266 = queryWeight, product of:
               2.2227914 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.012346189 = queryNorm
             0.32083112 = fieldWeight in 2117, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.079690866 = weight(abstract_txt:search in 2117) [ClassicSimilarity], result of:
           0.079690866 = score(doc=2117,freq=9.0), product of:
             0.13278508 = queryWeight, product of:
               2.9401295 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.012346189 = queryNorm
             0.6001492 = fieldWeight in 2117, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.2242462 = weight(abstract_txt:judgements in 2117) [ClassicSimilarity], result of:
           0.2242462 = score(doc=2117,freq=1.0), product of:
             0.5505218 = queryWeight, product of:
               5.986584 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.012346189 = queryNorm
             0.40733394 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
       0.4 = coord(10/25)
   

4. Dennis, S.; Bruza, P.; McArthur, R.: Web searching : a process-oriented experimental study of three interactive search paradigms (2002) 0.20

   0.19615348 = sum of:
     0.19615348 = product of:
       0.54487073 = sum of:
         0.0110630365 = weight(abstract_txt:study in 200) [ClassicSimilarity], result of:
           0.0110630365 = score(doc=200,freq=1.0), product of:
             0.051699363 = queryWeight, product of:
               1.223047 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.012346189 = queryNorm
             0.21398787 = fieldWeight in 200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.011641025 = weight(abstract_txt:results in 200) [ClassicSimilarity], result of:
           0.011641025 = score(doc=200,freq=1.0), product of:
             0.053484727 = queryWeight, product of:
               1.2439859 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.012346189 = queryNorm
             0.21765138 = fieldWeight in 200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.008215512 = weight(abstract_txt:this in 200) [ClassicSimilarity], result of:
           0.008215512 = score(doc=200,freq=2.0), product of:
             0.038519368 = queryWeight, product of:
               1.2929621 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.012346189 = queryNorm
             0.21328263 = fieldWeight in 200, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.049598355 = weight(abstract_txt:relevance in 200) [ClassicSimilarity], result of:
           0.049598355 = score(doc=200,freq=4.0), product of:
             0.080453984 = queryWeight, product of:
               1.3213108 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012346189 = queryNorm
             0.616481 = fieldWeight in 200, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.021077638 = weight(abstract_txt:over in 200) [ClassicSimilarity], result of:
           0.021077638 = score(doc=200,freq=1.0), product of:
             0.07945421 = queryWeight, product of:
               1.5162089 = boost
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.012346189 = queryNorm
             0.2652803 = fieldWeight in 200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.03579287 = weight(abstract_txt:user in 200) [ClassicSimilarity], result of:
           0.03579287 = score(doc=200,freq=3.0), product of:
             0.08976147 = queryWeight, product of:
               1.9737456 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.012346189 = queryNorm
             0.39875534 = fieldWeight in 200, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.04174202 = weight(abstract_txt:time in 200) [ClassicSimilarity], result of:
           0.04174202 = score(doc=200,freq=2.0), product of:
             0.11384266 = queryWeight, product of:
               2.2227914 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.012346189 = queryNorm
             0.36666414 = fieldWeight in 200, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.10945886 = weight(abstract_txt:search in 200) [ClassicSimilarity], result of:
           0.10945886 = score(doc=200,freq=13.0), product of:
             0.13278508 = queryWeight, product of:
               2.9401295 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.012346189 = queryNorm
             0.824331 = fieldWeight in 200, product of:
               3.6055512 = tf(freq=13.0), with freq of:
                 13.0 = termFreq=13.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
         0.25628138 = weight(abstract_txt:judgements in 200) [ClassicSimilarity], result of:
           0.25628138 = score(doc=200,freq=1.0), product of:
             0.5505218 = queryWeight, product of:
               5.986584 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.012346189 = queryNorm
             0.4655245 = fieldWeight in 200, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.0625 = fieldNorm(doc=200)
       0.36 = coord(9/25)
   

5. Bar-Ilan, J.; Levene, M.; Mat-Hassan, M.: Methods for evaluating dynamic changes in search engine rankings : a case study (2006) 0.19

   0.19374698 = sum of:
     0.19374698 = product of:
       0.48436746 = sum of:
         0.02993146 = weight(abstract_txt:ranking in 616) [ClassicSimilarity], result of:
           0.02993146 = score(doc=616,freq=2.0), product of:
             0.069124 = queryWeight, product of:
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.012346189 = queryNorm
             0.4330111 = fieldWeight in 616, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.598813 = idf(docFreq=444, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.030186472 = weight(abstract_txt:compare in 616) [ClassicSimilarity], result of:
           0.030186472 = score(doc=616,freq=2.0), product of:
             0.06951606 = queryWeight, product of:
               1.0028319 = boost
               5.6146684 = idf(docFreq=437, maxDocs=44218)
               0.012346189 = queryNorm
             0.43423736 = fieldWeight in 616, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.6146684 = idf(docFreq=437, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.02694935 = weight(abstract_txt:results in 616) [ClassicSimilarity], result of:
           0.02694935 = score(doc=616,freq=7.0), product of:
             0.053484727 = queryWeight, product of:
               1.2439859 = boost
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.012346189 = queryNorm
             0.50387 = fieldWeight in 616, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.482422 = idf(docFreq=3693, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.044545986 = weight(abstract_txt:stable in 616) [ClassicSimilarity], result of:
           0.044545986 = score(doc=616,freq=1.0), product of:
             0.113525294 = queryWeight, product of:
               1.2815392 = boost
               7.1750984 = idf(docFreq=91, maxDocs=44218)
               0.012346189 = queryNorm
             0.3923882 = fieldWeight in 616, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.1750984 = idf(docFreq=91, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.007188573 = weight(abstract_txt:this in 616) [ClassicSimilarity], result of:
           0.007188573 = score(doc=616,freq=2.0), product of:
             0.038519368 = queryWeight, product of:
               1.2929621 = boost
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.012346189 = queryNorm
             0.1866223 = fieldWeight in 616, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4130175 = idf(docFreq=10762, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.02550294 = weight(abstract_txt:point in 616) [ClassicSimilarity], result of:
           0.02550294 = score(doc=616,freq=1.0), product of:
             0.089600585 = queryWeight, product of:
               1.3943976 = boost
               5.2046475 = idf(docFreq=659, maxDocs=44218)
               0.012346189 = queryNorm
             0.28462917 = fieldWeight in 616, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2046475 = idf(docFreq=659, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.03688587 = weight(abstract_txt:over in 616) [ClassicSimilarity], result of:
           0.03688587 = score(doc=616,freq=4.0), product of:
             0.07945421 = queryWeight, product of:
               1.5162089 = boost
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.012346189 = queryNorm
             0.46424055 = fieldWeight in 616, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.16696167 = weight(abstract_txt:rankings in 616) [ClassicSimilarity], result of:
           0.16696167 = score(doc=616,freq=8.0), product of:
             0.15678285 = queryWeight, product of:
               1.8445077 = boost
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.012346189 = queryNorm
             1.064923 = fieldWeight in 616, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               6.8847027 = idf(docFreq=122, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.036524266 = weight(abstract_txt:time in 616) [ClassicSimilarity], result of:
           0.036524266 = score(doc=616,freq=2.0), product of:
             0.11384266 = queryWeight, product of:
               2.2227914 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.012346189 = queryNorm
             0.32083112 = fieldWeight in 616, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
         0.079690866 = weight(abstract_txt:search in 616) [ClassicSimilarity], result of:
           0.079690866 = score(doc=616,freq=9.0), product of:
             0.13278508 = queryWeight, product of:
               2.9401295 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.012346189 = queryNorm
             0.6001492 = fieldWeight in 616, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.0546875 = fieldNorm(doc=616)
       0.4 = coord(10/25)