Document (#40073)

Author

Zhitomirsky-Geffet, M.
Bar-Ilan, J.
Levene, M.

Title

Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings

Source

Aslib journal of information management. 68(2016) no.4, S.407-427

Year

2016

Abstract

Purpose - One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the "wisdom of crowds". The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements. Design/methodology/approach - In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated "wisdom of crowds" judgements were used as a means to compare the differences between human assessments of search results and search engine's rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties. Findings - It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings. Practical implications - The proposed "wisdom of crowds"-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine's ranking over time to changes in users preferences. Originality/value - This is a first study that applies the notion of "wisdom of crowds" to examine an under-explored in the literature phenomenon of "change in time" in user evaluation of relevance.

Content

Vgl.: http://www.emeraldinsight.com/doi/abs/10.1108/AJIM-10-2015-0165.

Similar documents (author)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 5.56

   5.564927 = sum of:
     5.564927 = sum of:
       1.0199649 = weight(author_txt:ilan in 5594) [ClassicSimilarity], result of:
         1.0199649 = score(doc=5594,freq=1.0), product of:
           0.4053526 = queryWeight, product of:
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.050342023 = queryNorm
           2.5162413 = fieldWeight in 5594, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.3125 = fieldNorm(doc=5594)
       1.4713223 = weight(author_txt:zhitomirsky in 5594) [ClassicSimilarity], result of:
         1.4713223 = score(doc=5594,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 5594, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=5594)
       1.4713223 = weight(author_txt:geffet in 5594) [ClassicSimilarity], result of:
         1.4713223 = score(doc=5594,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 5594, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=5594)
       1.6023173 = weight(author_txt:levene in 5594) [ClassicSimilarity], result of:
         1.6023173 = score(doc=5594,freq=1.0), product of:
           0.5477829 = queryWeight, product of:
             1.1624861 = boost
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.050342023 = queryNorm
           2.9250956 = fieldWeight in 5594, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.3125 = fieldNorm(doc=5594)
   

2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Categorical relevance judgment (2018) 5.56

   5.564927 = sum of:
     5.564927 = sum of:
       1.0199649 = weight(author_txt:ilan in 458) [ClassicSimilarity], result of:
         1.0199649 = score(doc=458,freq=1.0), product of:
           0.4053526 = queryWeight, product of:
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.050342023 = queryNorm
           2.5162413 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
       1.4713223 = weight(author_txt:zhitomirsky in 458) [ClassicSimilarity], result of:
         1.4713223 = score(doc=458,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
       1.4713223 = weight(author_txt:geffet in 458) [ClassicSimilarity], result of:
         1.4713223 = score(doc=458,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
       1.6023173 = weight(author_txt:levene in 458) [ClassicSimilarity], result of:
         1.6023173 = score(doc=458,freq=1.0), product of:
           0.5477829 = queryWeight, product of:
             1.1624861 = boost
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.050342023 = queryNorm
           2.9250956 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.: Towards maximal unification of semantically diverse ontologies for controversial domains (2014) 3.57

   3.5663486 = sum of:
     3.5663486 = product of:
       4.7551312 = sum of:
         1.223958 = weight(author_txt:ilan in 3635) [ClassicSimilarity], result of:
           1.223958 = score(doc=3635,freq=1.0), product of:
             0.4053526 = queryWeight, product of:
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.050342023 = queryNorm
             3.0194898 = fieldWeight in 3635, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.375 = fieldNorm(doc=3635)
         1.7655866 = weight(author_txt:zhitomirsky in 3635) [ClassicSimilarity], result of:
           1.7655866 = score(doc=3635,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             3.411728 = fieldWeight in 3635, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.375 = fieldNorm(doc=3635)
         1.7655866 = weight(author_txt:geffet in 3635) [ClassicSimilarity], result of:
           1.7655866 = score(doc=3635,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             3.411728 = fieldWeight in 3635, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.375 = fieldNorm(doc=3635)
       0.75 = coord(3/4)
   

4. Bar-Ilan, J.; Zhitomirsky-Geffet, M.; Miller, Y.; Shoham, S.: ¬The effects of background information and social interaction on image tagging (2010) 2.97

   2.9719572 = sum of:
     2.9719572 = product of:
       3.9626095 = sum of:
         1.0199649 = weight(author_txt:ilan in 454) [ClassicSimilarity], result of:
           1.0199649 = score(doc=454,freq=1.0), product of:
             0.4053526 = queryWeight, product of:
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.050342023 = queryNorm
             2.5162413 = fieldWeight in 454, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.3125 = fieldNorm(doc=454)
         1.4713223 = weight(author_txt:zhitomirsky in 454) [ClassicSimilarity], result of:
           1.4713223 = score(doc=454,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 454, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=454)
         1.4713223 = weight(author_txt:geffet in 454) [ClassicSimilarity], result of:
           1.4713223 = score(doc=454,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 454, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=454)
       0.75 = coord(3/4)
   

5. Zhitomirsky-Geffet, M.; Erez, E.S.; Bar-Ilan, J.: Toward multiviewpoint ontology construction by collaboration of non-experts and crowdsourcing : the case of the effect of diet on health (2017) 2.97

   2.9719572 = sum of:
     2.9719572 = product of:
       3.9626095 = sum of:
         1.0199649 = weight(author_txt:ilan in 5440) [ClassicSimilarity], result of:
           1.0199649 = score(doc=5440,freq=1.0), product of:
             0.4053526 = queryWeight, product of:
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.050342023 = queryNorm
             2.5162413 = fieldWeight in 5440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.3125 = fieldNorm(doc=5440)
         1.4713223 = weight(author_txt:zhitomirsky in 5440) [ClassicSimilarity], result of:
           1.4713223 = score(doc=5440,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 5440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=5440)
         1.4713223 = weight(author_txt:geffet in 5440) [ClassicSimilarity], result of:
           1.4713223 = score(doc=5440,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 5440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=5440)
       0.75 = coord(3/4)
   

Similar documents (content)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.59

   0.58889467 = sum of:
     0.58889467 = product of:
       1.0515976 = sum of:
         0.03399427 = weight(abstract_txt:ranking in 5594) [ClassicSimilarity], result of:
           0.03399427 = score(doc=5594,freq=2.0), product of:
             0.06868977 = queryWeight, product of:
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.01226799 = queryNorm
             0.49489567 = fieldWeight in 5594, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.018628117 = weight(abstract_txt:evaluation in 5594) [ClassicSimilarity], result of:
           0.018628117 = score(doc=5594,freq=1.0), product of:
             0.06633987 = queryWeight, product of:
               1.2036132 = boost
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.01226799 = queryNorm
             0.2807982 = fieldWeight in 5594, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.022903992 = weight(abstract_txt:study in 5594) [ClassicSimilarity], result of:
           0.022903992 = score(doc=5594,freq=4.0), product of:
             0.0527913 = queryWeight, product of:
               1.2397958 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.01226799 = queryNorm
             0.4338592 = fieldWeight in 5594, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.026420847 = weight(abstract_txt:results in 5594) [ClassicSimilarity], result of:
           0.026420847 = score(doc=5594,freq=5.0), product of:
             0.053903386 = queryWeight, product of:
               1.2527863 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.01226799 = queryNorm
             0.49015188 = fieldWeight in 5594, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.010374925 = weight(abstract_txt:this in 5594) [ClassicSimilarity], result of:
           0.010374925 = score(doc=5594,freq=3.0), product of:
             0.039230295 = queryWeight, product of:
               1.308957 = boost
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.01226799 = queryNorm
             0.26446208 = fieldWeight in 5594, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.049354028 = weight(abstract_txt:relevance in 5594) [ClassicSimilarity], result of:
           0.049354028 = score(doc=5594,freq=4.0), product of:
             0.0800185 = queryWeight, product of:
               1.3218884 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.01226799 = queryNorm
             0.6167827 = fieldWeight in 5594, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.04110216 = weight(abstract_txt:point in 5594) [ClassicSimilarity], result of:
           0.04110216 = score(doc=5594,freq=2.0), product of:
             0.089240454 = queryWeight, product of:
               1.3959842 = boost
               5.2108417 = idf(docFreq=633, maxDocs=42740)
               0.01226799 = queryNorm
             0.46057767 = fieldWeight in 5594, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2108417 = idf(docFreq=633, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.05972994 = weight(abstract_txt:scale in 5594) [ClassicSimilarity], result of:
           0.05972994 = score(doc=5594,freq=3.0), product of:
             0.100019105 = queryWeight, product of:
               1.4778866 = boost
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.01226799 = queryNorm
             0.5971853 = fieldWeight in 5594, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.021189187 = weight(abstract_txt:over in 5594) [ClassicSimilarity], result of:
           0.021189187 = score(doc=5594,freq=1.0), product of:
             0.079564214 = queryWeight, product of:
               1.5220463 = boost
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.01226799 = queryNorm
             0.26631555 = fieldWeight in 5594, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.06651906 = weight(abstract_txt:rankings in 5594) [ClassicSimilarity], result of:
           0.06651906 = score(doc=5594,freq=1.0), product of:
             0.1549861 = queryWeight, product of:
               1.8396956 = boost
               6.8671 = idf(docFreq=120, maxDocs=42740)
               0.01226799 = queryNorm
             0.42919374 = fieldWeight in 5594, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8671 = idf(docFreq=120, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.04577315 = weight(abstract_txt:user in 5594) [ClassicSimilarity], result of:
           0.04577315 = score(doc=5594,freq=5.0), product of:
             0.08900666 = queryWeight, product of:
               1.971632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.01226799 = queryNorm
             0.51426655 = fieldWeight in 5594, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.04199811 = weight(abstract_txt:time in 5594) [ClassicSimilarity], result of:
           0.04199811 = score(doc=5594,freq=2.0), product of:
             0.11406397 = queryWeight, product of:
               2.2319725 = boost
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.01226799 = queryNorm
             0.36819786 = fieldWeight in 5594, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.051967826 = weight(abstract_txt:search in 5594) [ClassicSimilarity], result of:
           0.051967826 = score(doc=5594,freq=3.0), product of:
             0.13146749 = queryWeight, product of:
               2.9347384 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.01226799 = queryNorm
             0.39529032 = fieldWeight in 5594, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
         0.56164193 = weight(abstract_txt:judgements in 5594) [ClassicSimilarity], result of:
           0.56164193 = score(doc=5594,freq=5.0), product of:
             0.5420243 = queryWeight, product of:
               5.9589486 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.01226799 = queryNorm
             1.0361933 = fieldWeight in 5594, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.0625 = fieldNorm(doc=5594)
       0.56 = coord(14/25)
   

2. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.21

   0.20937344 = sum of:
     0.20937344 = product of:
       0.7477623 = sum of:
         0.029753156 = weight(abstract_txt:study in 4269) [ClassicSimilarity], result of:
           0.029753156 = score(doc=4269,freq=3.0), product of:
             0.0527913 = queryWeight, product of:
               1.2397958 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.01226799 = queryNorm
             0.5635996 = fieldWeight in 4269, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.017723642 = weight(abstract_txt:results in 4269) [ClassicSimilarity], result of:
           0.017723642 = score(doc=4269,freq=1.0), product of:
             0.053903386 = queryWeight, product of:
               1.2527863 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.01226799 = queryNorm
             0.32880387 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.0089849485 = weight(abstract_txt:this in 4269) [ClassicSimilarity], result of:
           0.0089849485 = score(doc=4269,freq=1.0), product of:
             0.039230295 = queryWeight, product of:
               1.308957 = boost
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.01226799 = queryNorm
             0.22903088 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.08276922 = weight(abstract_txt:relevance in 4269) [ClassicSimilarity], result of:
           0.08276922 = score(doc=4269,freq=5.0), product of:
             0.0800185 = queryWeight, product of:
               1.3218884 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.01226799 = queryNorm
             1.0343761 = fieldWeight in 4269, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.030705562 = weight(abstract_txt:user in 4269) [ClassicSimilarity], result of:
           0.030705562 = score(doc=4269,freq=1.0), product of:
             0.08900666 = queryWeight, product of:
               1.971632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.01226799 = queryNorm
             0.34498048 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.04500546 = weight(abstract_txt:search in 4269) [ClassicSimilarity], result of:
           0.04500546 = score(doc=4269,freq=1.0), product of:
             0.13146749 = queryWeight, product of:
               2.9347384 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.01226799 = queryNorm
             0.34233147 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.53282034 = weight(abstract_txt:judgements in 4269) [ClassicSimilarity], result of:
           0.53282034 = score(doc=4269,freq=2.0), product of:
             0.5420243 = queryWeight, product of:
               5.9589486 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.01226799 = queryNorm
             0.98301923 = fieldWeight in 4269, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
       0.28 = coord(7/25)
   

3. Su, L.T.: ¬A comprehensive and systematic model of user evaluation of Web search engines : Il. An evaluation by undergraduates (2003) 0.21

   0.2085156 = sum of:
     0.2085156 = product of:
       0.521289 = sum of:
         0.032599203 = weight(abstract_txt:evaluation in 3118) [ClassicSimilarity], result of:
           0.032599203 = score(doc=3118,freq=4.0), product of:
             0.06633987 = queryWeight, product of:
               1.2036132 = boost
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.01226799 = queryNorm
             0.49139684 = fieldWeight in 3118, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.017356008 = weight(abstract_txt:study in 3118) [ClassicSimilarity], result of:
           0.017356008 = score(doc=3118,freq=3.0), product of:
             0.0527913 = queryWeight, product of:
               1.2397958 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.01226799 = queryNorm
             0.32876644 = fieldWeight in 3118, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.017907312 = weight(abstract_txt:results in 3118) [ClassicSimilarity], result of:
           0.017907312 = score(doc=3118,freq=3.0), product of:
             0.053903386 = queryWeight, product of:
               1.2527863 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.01226799 = queryNorm
             0.33221126 = fieldWeight in 3118, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.00524122 = weight(abstract_txt:this in 3118) [ClassicSimilarity], result of:
           0.00524122 = score(doc=3118,freq=1.0), product of:
             0.039230295 = queryWeight, product of:
               1.308957 = boost
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.01226799 = queryNorm
             0.13360134 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.030536246 = weight(abstract_txt:relevance in 3118) [ClassicSimilarity], result of:
           0.030536246 = score(doc=3118,freq=2.0), product of:
             0.0800185 = queryWeight, product of:
               1.3218884 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.01226799 = queryNorm
             0.38161483 = fieldWeight in 3118, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.047389578 = weight(abstract_txt:user in 3118) [ClassicSimilarity], result of:
           0.047389578 = score(doc=3118,freq=7.0), product of:
             0.08900666 = queryWeight, product of:
               1.971632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.01226799 = queryNorm
             0.5324273 = fieldWeight in 3118, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.034974355 = weight(abstract_txt:individual in 3118) [ClassicSimilarity], result of:
           0.034974355 = score(doc=3118,freq=1.0), product of:
             0.13084969 = queryWeight, product of:
               2.182279 = boost
               4.8875246 = idf(docFreq=875, maxDocs=42740)
               0.01226799 = queryNorm
             0.2672865 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.8875246 = idf(docFreq=875, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.036748346 = weight(abstract_txt:time in 3118) [ClassicSimilarity], result of:
           0.036748346 = score(doc=3118,freq=2.0), product of:
             0.11406397 = queryWeight, product of:
               2.2319725 = boost
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.01226799 = queryNorm
             0.32217312 = fieldWeight in 3118, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.07875956 = weight(abstract_txt:search in 3118) [ClassicSimilarity], result of:
           0.07875956 = score(doc=3118,freq=9.0), product of:
             0.13146749 = queryWeight, product of:
               2.9347384 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.01226799 = queryNorm
             0.5990801 = fieldWeight in 3118, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
         0.21977718 = weight(abstract_txt:judgements in 3118) [ClassicSimilarity], result of:
           0.21977718 = score(doc=3118,freq=1.0), product of:
             0.5420243 = queryWeight, product of:
               5.9589486 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.01226799 = queryNorm
             0.40547475 = fieldWeight in 3118, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.0546875 = fieldNorm(doc=3118)
       0.4 = coord(10/25)
   

4. Dennis, S.; Bruza, P.; McArthur, R.: Web searching : a process-oriented experimental study of three interactive search paradigms (2002) 0.19

   0.1940722 = sum of:
     0.1940722 = product of:
       0.53908944 = sum of:
         0.011451996 = weight(abstract_txt:study in 1201) [ClassicSimilarity], result of:
           0.011451996 = score(doc=1201,freq=1.0), product of:
             0.0527913 = queryWeight, product of:
               1.2397958 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.01226799 = queryNorm
             0.2169296 = fieldWeight in 1201, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.011815761 = weight(abstract_txt:results in 1201) [ClassicSimilarity], result of:
           0.011815761 = score(doc=1201,freq=1.0), product of:
             0.053903386 = queryWeight, product of:
               1.2527863 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.01226799 = queryNorm
             0.21920258 = fieldWeight in 1201, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.008471091 = weight(abstract_txt:this in 1201) [ClassicSimilarity], result of:
           0.008471091 = score(doc=1201,freq=2.0), product of:
             0.039230295 = queryWeight, product of:
               1.308957 = boost
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.01226799 = queryNorm
             0.21593238 = fieldWeight in 1201, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.049354028 = weight(abstract_txt:relevance in 1201) [ClassicSimilarity], result of:
           0.049354028 = score(doc=1201,freq=4.0), product of:
             0.0800185 = queryWeight, product of:
               1.3218884 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.01226799 = queryNorm
             0.6167827 = fieldWeight in 1201, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.021189187 = weight(abstract_txt:over in 1201) [ClassicSimilarity], result of:
           0.021189187 = score(doc=1201,freq=1.0), product of:
             0.079564214 = queryWeight, product of:
               1.5220463 = boost
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.01226799 = queryNorm
             0.26631555 = fieldWeight in 1201, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.035455726 = weight(abstract_txt:user in 1201) [ClassicSimilarity], result of:
           0.035455726 = score(doc=1201,freq=3.0), product of:
             0.08900666 = queryWeight, product of:
               1.971632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.01226799 = queryNorm
             0.39834914 = fieldWeight in 1201, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.04199811 = weight(abstract_txt:time in 1201) [ClassicSimilarity], result of:
           0.04199811 = score(doc=1201,freq=2.0), product of:
             0.11406397 = queryWeight, product of:
               2.2319725 = boost
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.01226799 = queryNorm
             0.36819786 = fieldWeight in 1201, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.10817965 = weight(abstract_txt:search in 1201) [ClassicSimilarity], result of:
           0.10817965 = score(doc=1201,freq=13.0), product of:
             0.13146749 = queryWeight, product of:
               2.9347384 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.01226799 = queryNorm
             0.8228624 = fieldWeight in 1201, product of:
               3.6055512 = tf(freq=13.0), with freq of:
                 13.0 = termFreq=13.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
         0.2511739 = weight(abstract_txt:judgements in 1201) [ClassicSimilarity], result of:
           0.2511739 = score(doc=1201,freq=1.0), product of:
             0.5420243 = queryWeight, product of:
               5.9589486 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.01226799 = queryNorm
             0.4633997 = fieldWeight in 1201, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.0625 = fieldNorm(doc=1201)
       0.36 = coord(9/25)
   

5. Bar-Ilan, J.; Levene, M.; Mat-Hassan, M.: Methods for evaluating dynamic changes in search engine rankings : a case study (2006) 0.19

   0.19277851 = sum of:
     0.19277851 = product of:
       0.4819463 = sum of:
         0.029744986 = weight(abstract_txt:ranking in 1742) [ClassicSimilarity], result of:
           0.029744986 = score(doc=1742,freq=2.0), product of:
             0.06868977 = queryWeight, product of:
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.01226799 = queryNorm
             0.4330337 = fieldWeight in 1742, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.030121576 = weight(abstract_txt:compare in 1742) [ClassicSimilarity], result of:
           0.030121576 = score(doc=1742,freq=2.0), product of:
             0.06926832 = queryWeight, product of:
               1.0042025 = boost
               5.622636 = idf(docFreq=419, maxDocs=42740)
               0.01226799 = queryNorm
             0.43485355 = fieldWeight in 1742, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.622636 = idf(docFreq=419, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.02735387 = weight(abstract_txt:results in 1742) [ClassicSimilarity], result of:
           0.02735387 = score(doc=1742,freq=7.0), product of:
             0.053903386 = queryWeight, product of:
               1.2527863 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.01226799 = queryNorm
             0.5074611 = fieldWeight in 1742, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.044667754 = weight(abstract_txt:stable in 1742) [ClassicSimilarity], result of:
           0.044667754 = score(doc=1742,freq=1.0), product of:
             0.11348949 = queryWeight, product of:
               1.2853807 = boost
               7.1969824 = idf(docFreq=86, maxDocs=42740)
               0.01226799 = queryNorm
             0.39358497 = fieldWeight in 1742, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.1969824 = idf(docFreq=86, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.0074122045 = weight(abstract_txt:this in 1742) [ClassicSimilarity], result of:
           0.0074122045 = score(doc=1742,freq=2.0), product of:
             0.039230295 = queryWeight, product of:
               1.308957 = boost
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.01226799 = queryNorm
             0.18894084 = fieldWeight in 1742, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.442996 = idf(docFreq=10095, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.025430664 = weight(abstract_txt:point in 1742) [ClassicSimilarity], result of:
           0.025430664 = score(doc=1742,freq=1.0), product of:
             0.089240454 = queryWeight, product of:
               1.3959842 = boost
               5.2108417 = idf(docFreq=633, maxDocs=42740)
               0.01226799 = queryNorm
             0.2849679 = fieldWeight in 1742, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2108417 = idf(docFreq=633, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.037081078 = weight(abstract_txt:over in 1742) [ClassicSimilarity], result of:
           0.037081078 = score(doc=1742,freq=4.0), product of:
             0.079564214 = queryWeight, product of:
               1.5220463 = boost
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.01226799 = queryNorm
             0.4660522 = fieldWeight in 1742, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.16462626 = weight(abstract_txt:rankings in 1742) [ClassicSimilarity], result of:
           0.16462626 = score(doc=1742,freq=8.0), product of:
             0.1549861 = queryWeight, product of:
               1.8396956 = boost
               6.8671 = idf(docFreq=120, maxDocs=42740)
               0.01226799 = queryNorm
             1.0622002 = fieldWeight in 1742, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               6.8671 = idf(docFreq=120, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.036748346 = weight(abstract_txt:time in 1742) [ClassicSimilarity], result of:
           0.036748346 = score(doc=1742,freq=2.0), product of:
             0.11406397 = queryWeight, product of:
               2.2319725 = boost
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.01226799 = queryNorm
             0.32217312 = fieldWeight in 1742, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
         0.07875956 = weight(abstract_txt:search in 1742) [ClassicSimilarity], result of:
           0.07875956 = score(doc=1742,freq=9.0), product of:
             0.13146749 = queryWeight, product of:
               2.9347384 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.01226799 = queryNorm
             0.5990801 = fieldWeight in 1742, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1742)
       0.4 = coord(10/25)