Document (#40595)

Author

Zhitomirsky-Geffet, M.
Bar-Ilan, J.
Levene, M.

Title

Analysis of change in users' assessment of search results over time

Source

Journal of the Association for Information Science and Technology. 68(2017) no.5, S.1137-1148

Year

2017

Abstract

We present the first systematic study of the influence of time on user judgements for rankings and relevance grades of web search engine results. The goal of this study is to evaluate the change in user assessment of search results and explore how users' judgements change. To this end, we conducted a large-scale user study with 86 participants who evaluated 2 different queries and 4 diverse result sets twice with an interval of 2 months. To analyze the results we investigate whether 2 types of patterns of user behavior from the theory of categorical thinking hold for the case of evaluation of search results: (a) coarseness and (b) locality. To quantify these patterns we devised 2 new measures of change in user judgements and distinguish between local (when users swap between close ranks and relevance values) and nonlocal changes. Two types of judgements were considered in this study: (a) relevance on a 4-point scale, and (b) ranking on a 10-point scale without ties. We found that users tend to change their judgements of the results over time in about 50% of cases for relevance and in 85% of cases for ranking. However, the majority of these changes were local.

Content

Vgl.: http://onlinelibrary.wiley.com/doi/10.1002/asi.23745/full.

Theme

Suchmaschinen
Benutzerstudien

Similar documents (author)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings (2016) 5.56

   5.564927 = sum of:
     5.564927 = sum of:
       1.0199649 = weight(author_txt:ilan in 5072) [ClassicSimilarity], result of:
         1.0199649 = score(doc=5072,freq=1.0), product of:
           0.4053526 = queryWeight, product of:
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.050342023 = queryNorm
           2.5162413 = fieldWeight in 5072, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.3125 = fieldNorm(doc=5072)
       1.4713223 = weight(author_txt:zhitomirsky in 5072) [ClassicSimilarity], result of:
         1.4713223 = score(doc=5072,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 5072, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=5072)
       1.4713223 = weight(author_txt:geffet in 5072) [ClassicSimilarity], result of:
         1.4713223 = score(doc=5072,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 5072, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=5072)
       1.6023173 = weight(author_txt:levene in 5072) [ClassicSimilarity], result of:
         1.6023173 = score(doc=5072,freq=1.0), product of:
           0.5477829 = queryWeight, product of:
             1.1624861 = boost
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.050342023 = queryNorm
           2.9250956 = fieldWeight in 5072, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.3125 = fieldNorm(doc=5072)
   

2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Categorical relevance judgment (2018) 5.56

   5.564927 = sum of:
     5.564927 = sum of:
       1.0199649 = weight(author_txt:ilan in 458) [ClassicSimilarity], result of:
         1.0199649 = score(doc=458,freq=1.0), product of:
           0.4053526 = queryWeight, product of:
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.050342023 = queryNorm
           2.5162413 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.051972 = idf(docFreq=36, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
       1.4713223 = weight(author_txt:zhitomirsky in 458) [ClassicSimilarity], result of:
         1.4713223 = score(doc=458,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
       1.4713223 = weight(author_txt:geffet in 458) [ClassicSimilarity], result of:
         1.4713223 = score(doc=458,freq=1.0), product of:
           0.5175051 = queryWeight, product of:
             1.1299022 = boost
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.050342023 = queryNorm
           2.8431067 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.097941 = idf(docFreq=12, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
       1.6023173 = weight(author_txt:levene in 458) [ClassicSimilarity], result of:
         1.6023173 = score(doc=458,freq=1.0), product of:
           0.5477829 = queryWeight, product of:
             1.1624861 = boost
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.050342023 = queryNorm
           2.9250956 = fieldWeight in 458, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.360306 = idf(docFreq=9, maxDocs=42740)
             0.3125 = fieldNorm(doc=458)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.: Towards maximal unification of semantically diverse ontologies for controversial domains (2014) 3.57

   3.5663486 = sum of:
     3.5663486 = product of:
       4.7551312 = sum of:
         1.223958 = weight(author_txt:ilan in 3635) [ClassicSimilarity], result of:
           1.223958 = score(doc=3635,freq=1.0), product of:
             0.4053526 = queryWeight, product of:
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.050342023 = queryNorm
             3.0194898 = fieldWeight in 3635, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.375 = fieldNorm(doc=3635)
         1.7655866 = weight(author_txt:zhitomirsky in 3635) [ClassicSimilarity], result of:
           1.7655866 = score(doc=3635,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             3.411728 = fieldWeight in 3635, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.375 = fieldNorm(doc=3635)
         1.7655866 = weight(author_txt:geffet in 3635) [ClassicSimilarity], result of:
           1.7655866 = score(doc=3635,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             3.411728 = fieldWeight in 3635, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.375 = fieldNorm(doc=3635)
       0.75 = coord(3/4)
   

4. Bar-Ilan, J.; Zhitomirsky-Geffet, M.; Miller, Y.; Shoham, S.: ¬The effects of background information and social interaction on image tagging (2010) 2.97

   2.9719572 = sum of:
     2.9719572 = product of:
       3.9626095 = sum of:
         1.0199649 = weight(author_txt:ilan in 454) [ClassicSimilarity], result of:
           1.0199649 = score(doc=454,freq=1.0), product of:
             0.4053526 = queryWeight, product of:
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.050342023 = queryNorm
             2.5162413 = fieldWeight in 454, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.3125 = fieldNorm(doc=454)
         1.4713223 = weight(author_txt:zhitomirsky in 454) [ClassicSimilarity], result of:
           1.4713223 = score(doc=454,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 454, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=454)
         1.4713223 = weight(author_txt:geffet in 454) [ClassicSimilarity], result of:
           1.4713223 = score(doc=454,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 454, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=454)
       0.75 = coord(3/4)
   

5. Zhitomirsky-Geffet, M.; Erez, E.S.; Bar-Ilan, J.: Toward multiviewpoint ontology construction by collaboration of non-experts and crowdsourcing : the case of the effect of diet on health (2017) 2.97

   2.9719572 = sum of:
     2.9719572 = product of:
       3.9626095 = sum of:
         1.0199649 = weight(author_txt:ilan in 5440) [ClassicSimilarity], result of:
           1.0199649 = score(doc=5440,freq=1.0), product of:
             0.4053526 = queryWeight, product of:
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.050342023 = queryNorm
             2.5162413 = fieldWeight in 5440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.051972 = idf(docFreq=36, maxDocs=42740)
               0.3125 = fieldNorm(doc=5440)
         1.4713223 = weight(author_txt:zhitomirsky in 5440) [ClassicSimilarity], result of:
           1.4713223 = score(doc=5440,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 5440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=5440)
         1.4713223 = weight(author_txt:geffet in 5440) [ClassicSimilarity], result of:
           1.4713223 = score(doc=5440,freq=1.0), product of:
             0.5175051 = queryWeight, product of:
               1.1299022 = boost
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.050342023 = queryNorm
             2.8431067 = fieldWeight in 5440, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.097941 = idf(docFreq=12, maxDocs=42740)
               0.3125 = fieldNorm(doc=5440)
       0.75 = coord(3/4)
   

Similar documents (content)

1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings (2016) 1.45

   1.4492661 = sum of:
     1.4492661 = product of:
       1.906929 = sum of:
         0.06618308 = weight(abstract_txt:twice in 5072) [ClassicSimilarity], result of:
           0.06618308 = score(doc=5072,freq=1.0), product of:
             0.14926903 = queryWeight, product of:
               8.107542 = idf(docFreq=34, maxDocs=42740)
               0.018411132 = queryNorm
             0.4433812 = fieldWeight in 5072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.107542 = idf(docFreq=34, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.067634545 = weight(abstract_txt:ties in 5072) [ClassicSimilarity], result of:
           0.067634545 = score(doc=5072,freq=1.0), product of:
             0.15144354 = queryWeight, product of:
               1.0072576 = boost
               8.166383 = idf(docFreq=32, maxDocs=42740)
               0.018411132 = queryNorm
             0.44659907 = fieldWeight in 5072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.166383 = idf(docFreq=32, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.0747698 = weight(abstract_txt:interval in 5072) [ClassicSimilarity], result of:
           0.0747698 = score(doc=5072,freq=1.0), product of:
             0.1619158 = queryWeight, product of:
               1.0415012 = boost
               8.444015 = idf(docFreq=24, maxDocs=42740)
               0.018411132 = queryNorm
             0.46178204 = fieldWeight in 5072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.444015 = idf(docFreq=24, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.03328277 = weight(abstract_txt:over in 5072) [ClassicSimilarity], result of:
           0.03328277 = score(doc=5072,freq=3.0), product of:
             0.08246208 = queryWeight, product of:
               1.0511321 = boost
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.018411132 = queryNorm
             0.40361306 = fieldWeight in 5072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.02253592 = weight(abstract_txt:types in 5072) [ClassicSimilarity], result of:
           0.02253592 = score(doc=5072,freq=1.0), product of:
             0.09170617 = queryWeight, product of:
               1.108484 = boost
               4.4935403 = idf(docFreq=1298, maxDocs=42740)
               0.018411132 = queryNorm
             0.24574049 = fieldWeight in 5072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4935403 = idf(docFreq=1298, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.03162594 = weight(abstract_txt:changes in 5072) [ClassicSimilarity], result of:
           0.03162594 = score(doc=5072,freq=1.0), product of:
             0.11495045 = queryWeight, product of:
               1.2410392 = boost
               5.0308886 = idf(docFreq=758, maxDocs=42740)
               0.018411132 = queryNorm
             0.27512673 = fieldWeight in 5072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0308886 = idf(docFreq=758, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.060868636 = weight(abstract_txt:point in 5072) [ClassicSimilarity], result of:
           0.060868636 = score(doc=5072,freq=3.0), product of:
             0.123321 = queryWeight, product of:
               1.2854307 = boost
               5.2108417 = idf(docFreq=633, maxDocs=42740)
               0.018411132 = queryNorm
             0.49357885 = fieldWeight in 5072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.2108417 = idf(docFreq=633, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.061656695 = weight(abstract_txt:ranking in 5072) [ClassicSimilarity], result of:
           0.061656695 = score(doc=5072,freq=2.0), product of:
             0.14238313 = queryWeight, product of:
               1.3812091 = boost
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.018411132 = queryNorm
             0.4330337 = fieldWeight in 5072, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.048371486 = weight(abstract_txt:assessment in 5072) [ClassicSimilarity], result of:
           0.048371486 = score(doc=5072,freq=1.0), product of:
             0.15259536 = queryWeight, product of:
               1.4298841 = boost
               5.7964225 = idf(docFreq=352, maxDocs=42740)
               0.018411132 = queryNorm
             0.31699187 = fieldWeight in 5072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7964225 = idf(docFreq=352, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.06022049 = weight(abstract_txt:time in 5072) [ClassicSimilarity], result of:
           0.06022049 = score(doc=5072,freq=5.0), product of:
             0.11821838 = queryWeight, product of:
               1.5414103 = boost
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.018411132 = queryNorm
             0.5094004 = fieldWeight in 5072, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.041541845 = weight(abstract_txt:study in 5072) [ClassicSimilarity], result of:
           0.041541845 = score(doc=5072,freq=4.0), product of:
             0.10942811 = queryWeight, product of:
               1.7124172 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.018411132 = queryNorm
             0.3796268 = fieldWeight in 5072, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.039269883 = weight(abstract_txt:users in 5072) [ClassicSimilarity], result of:
           0.039269883 = score(doc=5072,freq=3.0), product of:
             0.116008826 = queryWeight, product of:
               1.7631557 = boost
               3.5737147 = idf(docFreq=3258, maxDocs=42740)
               0.018411132 = queryNorm
             0.3385077 = fieldWeight in 5072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.5737147 = idf(docFreq=3258, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.063990414 = weight(abstract_txt:search in 5072) [ClassicSimilarity], result of:
           0.063990414 = score(doc=5072,freq=7.0), product of:
             0.12111622 = queryWeight, product of:
               1.8015499 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.018411132 = queryNorm
             0.5283389 = fieldWeight in 5072, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.10833447 = weight(abstract_txt:scale in 5072) [ClassicSimilarity], result of:
           0.10833447 = score(doc=5072,freq=3.0), product of:
             0.20732394 = queryWeight, product of:
               2.0412703 = boost
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.018411132 = queryNorm
             0.5225372 = fieldWeight in 5072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.07578699 = weight(abstract_txt:user in 5072) [ClassicSimilarity], result of:
           0.07578699 = score(doc=5072,freq=6.0), product of:
             0.15374736 = queryWeight, product of:
               2.2693632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.018411132 = queryNorm
             0.49293196 = fieldWeight in 5072, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.103363335 = weight(abstract_txt:relevance in 5072) [ClassicSimilarity], result of:
           0.103363335 = score(doc=5072,freq=3.0), product of:
             0.22115442 = queryWeight, product of:
               2.4344058 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.018411132 = queryNorm
             0.46738082 = fieldWeight in 5072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.055678584 = weight(abstract_txt:results in 5072) [ClassicSimilarity], result of:
           0.055678584 = score(doc=5072,freq=3.0), product of:
             0.16759993 = queryWeight, product of:
               2.5955398 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.018411132 = queryNorm
             0.33221126 = fieldWeight in 5072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.13254124 = weight(abstract_txt:change in 5072) [ClassicSimilarity], result of:
           0.13254124 = score(doc=5072,freq=2.0), product of:
             0.3218739 = queryWeight, product of:
               3.2835455 = boost
               5.3242965 = idf(docFreq=565, maxDocs=42740)
               0.018411132 = queryNorm
             0.41178003 = fieldWeight in 5072, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.3242965 = idf(docFreq=565, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
         0.75927275 = weight(abstract_txt:judgements in 5072) [ClassicSimilarity], result of:
           0.75927275 = score(doc=5072,freq=9.0), product of:
             0.6241842 = queryWeight, product of:
               4.57253 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.018411132 = queryNorm
             1.2164242 = fieldWeight in 5072, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.0546875 = fieldNorm(doc=5072)
       0.76 = coord(19/25)
   

2. Spink, A.; Greisdorf, H.: Users' partial relevance judgements during online searching (1997) 0.37

   0.37203726 = sum of:
     0.37203726 = product of:
       1.0334368 = sum of:
         0.04517991 = weight(abstract_txt:changes in 1624) [ClassicSimilarity], result of:
           0.04517991 = score(doc=1624,freq=1.0), product of:
             0.11495045 = queryWeight, product of:
               1.2410392 = boost
               5.0308886 = idf(docFreq=758, maxDocs=42740)
               0.018411132 = queryNorm
             0.39303815 = fieldWeight in 1624, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0308886 = idf(docFreq=758, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.029672746 = weight(abstract_txt:study in 1624) [ClassicSimilarity], result of:
           0.029672746 = score(doc=1624,freq=1.0), product of:
             0.10942811 = queryWeight, product of:
               1.7124172 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.018411132 = queryNorm
             0.271162 = fieldWeight in 1624, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.07933714 = weight(abstract_txt:users in 1624) [ClassicSimilarity], result of:
           0.07933714 = score(doc=1624,freq=6.0), product of:
             0.116008826 = queryWeight, product of:
               1.7631557 = boost
               3.5737147 = idf(docFreq=3258, maxDocs=42740)
               0.018411132 = queryNorm
             0.68388885 = fieldWeight in 1624, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.5737147 = idf(docFreq=3258, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.048863307 = weight(abstract_txt:search in 1624) [ClassicSimilarity], result of:
           0.048863307 = score(doc=1624,freq=2.0), product of:
             0.12111622 = queryWeight, product of:
               1.8015499 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.018411132 = queryNorm
             0.4034415 = fieldWeight in 1624, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.08935276 = weight(abstract_txt:scale in 1624) [ClassicSimilarity], result of:
           0.08935276 = score(doc=1624,freq=1.0), product of:
             0.20732394 = queryWeight, product of:
               2.0412703 = boost
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.018411132 = queryNorm
             0.43098137 = fieldWeight in 1624, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.044199865 = weight(abstract_txt:user in 1624) [ClassicSimilarity], result of:
           0.044199865 = score(doc=1624,freq=1.0), product of:
             0.15374736 = queryWeight, product of:
               2.2693632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.018411132 = queryNorm
             0.28748372 = fieldWeight in 1624, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.12056544 = weight(abstract_txt:relevance in 1624) [ClassicSimilarity], result of:
           0.12056544 = score(doc=1624,freq=2.0), product of:
             0.22115442 = queryWeight, product of:
               2.4344058 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.018411132 = queryNorm
             0.54516405 = fieldWeight in 1624, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.06494482 = weight(abstract_txt:results in 1624) [ClassicSimilarity], result of:
           0.06494482 = score(doc=1624,freq=2.0), product of:
             0.16759993 = queryWeight, product of:
               2.5955398 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.018411132 = queryNorm
             0.38749906 = fieldWeight in 1624, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
         0.5113209 = weight(abstract_txt:judgements in 1624) [ClassicSimilarity], result of:
           0.5113209 = score(doc=1624,freq=2.0), product of:
             0.6241842 = queryWeight, product of:
               4.57253 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.018411132 = queryNorm
             0.8191827 = fieldWeight in 1624, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.078125 = fieldNorm(doc=1624)
       0.36 = coord(9/25)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Categorical relevance judgment (2018) 0.27

   0.27333403 = sum of:
     0.27333403 = product of:
       0.68333507 = sum of:
         0.08669653 = weight(abstract_txt:categorical in 458) [ClassicSimilarity], result of:
           0.08669653 = score(doc=458,freq=1.0), product of:
             0.16348511 = queryWeight, product of:
               1.0465362 = boost
               8.484837 = idf(docFreq=23, maxDocs=42740)
               0.018411132 = queryNorm
             0.5303023 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.484837 = idf(docFreq=23, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.021960935 = weight(abstract_txt:over in 458) [ClassicSimilarity], result of:
           0.021960935 = score(doc=458,freq=1.0), product of:
             0.08246208 = queryWeight, product of:
               1.0511321 = boost
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.018411132 = queryNorm
             0.26631555 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.261049 = idf(docFreq=1638, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.049826134 = weight(abstract_txt:ranking in 458) [ClassicSimilarity], result of:
           0.049826134 = score(doc=458,freq=1.0), product of:
             0.14238313 = queryWeight, product of:
               1.3812091 = boost
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.018411132 = queryNorm
             0.34994408 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.03077877 = weight(abstract_txt:time in 458) [ClassicSimilarity], result of:
           0.03077877 = score(doc=458,freq=1.0), product of:
             0.11821838 = queryWeight, product of:
               1.5414103 = boost
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.018411132 = queryNorm
             0.2603552 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1656833 = idf(docFreq=1802, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.023738196 = weight(abstract_txt:study in 458) [ClassicSimilarity], result of:
           0.023738196 = score(doc=458,freq=1.0), product of:
             0.10942811 = queryWeight, product of:
               1.7124172 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.018411132 = queryNorm
             0.2169296 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.073288515 = weight(abstract_txt:users in 458) [ClassicSimilarity], result of:
           0.073288515 = score(doc=458,freq=8.0), product of:
             0.116008826 = queryWeight, product of:
               1.7631557 = boost
               3.5737147 = idf(docFreq=3258, maxDocs=42740)
               0.018411132 = queryNorm
             0.63174945 = fieldWeight in 458, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               3.5737147 = idf(docFreq=3258, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.055282522 = weight(abstract_txt:search in 458) [ClassicSimilarity], result of:
           0.055282522 = score(doc=458,freq=4.0), product of:
             0.12111622 = queryWeight, product of:
               1.8015499 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.018411132 = queryNorm
             0.45644194 = fieldWeight in 458, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.15250456 = weight(abstract_txt:relevance in 458) [ClassicSimilarity], result of:
           0.15250456 = score(doc=458,freq=5.0), product of:
             0.22115442 = queryWeight, product of:
               2.4344058 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.018411132 = queryNorm
             0.6895841 = fieldWeight in 458, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.08214942 = weight(abstract_txt:results in 458) [ClassicSimilarity], result of:
           0.08214942 = score(doc=458,freq=5.0), product of:
             0.16759993 = queryWeight, product of:
               2.5955398 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.018411132 = queryNorm
             0.49015188 = fieldWeight in 458, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
         0.1071095 = weight(abstract_txt:change in 458) [ClassicSimilarity], result of:
           0.1071095 = score(doc=458,freq=1.0), product of:
             0.3218739 = queryWeight, product of:
               3.2835455 = boost
               5.3242965 = idf(docFreq=565, maxDocs=42740)
               0.018411132 = queryNorm
             0.33276853 = fieldWeight in 458, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3242965 = idf(docFreq=565, maxDocs=42740)
               0.0625 = fieldNorm(doc=458)
       0.4 = coord(10/25)
   

4. Spink, S.: Interaction with information retrieval systems : reflections on feedback (1993) 0.26

   0.26044437 = sum of:
     0.26044437 = product of:
       0.9301585 = sum of:
         0.04507184 = weight(abstract_txt:types in 7958) [ClassicSimilarity], result of:
           0.04507184 = score(doc=7958,freq=1.0), product of:
             0.09170617 = queryWeight, product of:
               1.108484 = boost
               4.4935403 = idf(docFreq=1298, maxDocs=42740)
               0.018411132 = queryNorm
             0.49148098 = fieldWeight in 7958, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4935403 = idf(docFreq=1298, maxDocs=42740)
               0.109375 = fieldNorm(doc=7958)
         0.09183737 = weight(abstract_txt:cases in 7958) [ClassicSimilarity], result of:
           0.09183737 = score(doc=7958,freq=1.0), product of:
             0.14739226 = queryWeight, product of:
               1.405295 = boost
               5.696744 = idf(docFreq=389, maxDocs=42740)
               0.018411132 = queryNorm
             0.6230814 = fieldWeight in 7958, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.696744 = idf(docFreq=389, maxDocs=42740)
               0.109375 = fieldNorm(doc=7958)
         0.041541845 = weight(abstract_txt:study in 7958) [ClassicSimilarity], result of:
           0.041541845 = score(doc=7958,freq=1.0), product of:
             0.10942811 = queryWeight, product of:
               1.7124172 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.018411132 = queryNorm
             0.3796268 = fieldWeight in 7958, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.109375 = fieldNorm(doc=7958)
         0.06187981 = weight(abstract_txt:user in 7958) [ClassicSimilarity], result of:
           0.06187981 = score(doc=7958,freq=1.0), product of:
             0.15374736 = queryWeight, product of:
               2.2693632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.018411132 = queryNorm
             0.4024772 = fieldWeight in 7958, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.109375 = fieldNorm(doc=7958)
         0.1193537 = weight(abstract_txt:relevance in 7958) [ClassicSimilarity], result of:
           0.1193537 = score(doc=7958,freq=1.0), product of:
             0.22115442 = queryWeight, product of:
               2.4344058 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.018411132 = queryNorm
             0.5396849 = fieldWeight in 7958, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.109375 = fieldNorm(doc=7958)
         0.064292096 = weight(abstract_txt:results in 7958) [ClassicSimilarity], result of:
           0.064292096 = score(doc=7958,freq=1.0), product of:
             0.16759993 = queryWeight, product of:
               2.5955398 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.018411132 = queryNorm
             0.38360453 = fieldWeight in 7958, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.109375 = fieldNorm(doc=7958)
         0.50618184 = weight(abstract_txt:judgements in 7958) [ClassicSimilarity], result of:
           0.50618184 = score(doc=7958,freq=1.0), product of:
             0.6241842 = queryWeight, product of:
               4.57253 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.018411132 = queryNorm
             0.8109495 = fieldWeight in 7958, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.109375 = fieldNorm(doc=7958)
       0.28 = coord(7/25)
   

5. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.25

   0.25286993 = sum of:
     0.25286993 = product of:
       1.0536247 = sum of:
         0.061673637 = weight(abstract_txt:study in 4269) [ClassicSimilarity], result of:
           0.061673637 = score(doc=4269,freq=3.0), product of:
             0.10942811 = queryWeight, product of:
               1.7124172 = boost
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.018411132 = queryNorm
             0.5635996 = fieldWeight in 4269, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4708736 = idf(docFreq=3611, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.041461892 = weight(abstract_txt:search in 4269) [ClassicSimilarity], result of:
           0.041461892 = score(doc=4269,freq=1.0), product of:
             0.12111622 = queryWeight, product of:
               1.8015499 = boost
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.018411132 = queryNorm
             0.34233147 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6515355 = idf(docFreq=3014, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.053039838 = weight(abstract_txt:user in 4269) [ClassicSimilarity], result of:
           0.053039838 = score(doc=4269,freq=1.0), product of:
             0.15374736 = queryWeight, product of:
               2.2693632 = boost
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.018411132 = queryNorm
             0.34498048 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6797917 = idf(docFreq=2930, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.22875686 = weight(abstract_txt:relevance in 4269) [ClassicSimilarity], result of:
           0.22875686 = score(doc=4269,freq=5.0), product of:
             0.22115442 = queryWeight, product of:
               2.4344058 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.018411132 = queryNorm
             1.0343761 = fieldWeight in 4269, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.055107504 = weight(abstract_txt:results in 4269) [ClassicSimilarity], result of:
           0.055107504 = score(doc=4269,freq=1.0), product of:
             0.16759993 = queryWeight, product of:
               2.5955398 = boost
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.018411132 = queryNorm
             0.32880387 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.5072412 = idf(docFreq=3482, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.61358505 = weight(abstract_txt:judgements in 4269) [ClassicSimilarity], result of:
           0.61358505 = score(doc=4269,freq=2.0), product of:
             0.6241842 = queryWeight, product of:
               4.57253 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.018411132 = queryNorm
             0.98301923 = fieldWeight in 4269, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
       0.24 = coord(6/25)