Document (#40073)

Author
Zhitomirsky-Geffet, M.
Bar-Ilan, J.
Levene, M.
Title
Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings
Source
Aslib journal of information management. 68(2016) no.4, S.407-427
Year
2016
Abstract
Purpose - One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the "wisdom of crowds". The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements. Design/methodology/approach - In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated "wisdom of crowds" judgements were used as a means to compare the differences between human assessments of search results and search engine's rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties. Findings - It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings. Practical implications - The proposed "wisdom of crowds"-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine's ranking over time to changes in users preferences. Originality/value - This is a first study that applies the notion of "wisdom of crowds" to examine an under-explored in the literature phenomenon of "change in time" in user evaluation of relevance.
Content
Vgl.: http://www.emeraldinsight.com/doi/abs/10.1108/AJIM-10-2015-0165.

Similar documents (author)

  1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 5.57
    5.5723953 = sum of:
      5.5723953 = sum of:
        1.0217929 = weight(author_txt:ilan in 5058) [ClassicSimilarity], result of:
          1.0217929 = score(doc=5058,freq=1.0), product of:
            0.40547708 = queryWeight, product of:
              8.063927 = idf(docFreq=36, maxDocs=43254)
              0.050282832 = queryNorm
            2.519977 = fieldWeight in 5058, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              8.063927 = idf(docFreq=36, maxDocs=43254)
              0.3125 = fieldNorm(doc=5058)
        1.4732056 = weight(author_txt:geffet in 5058) [ClassicSimilarity], result of:
          1.4732056 = score(doc=5058,freq=1.0), product of:
            0.5174876 = queryWeight, product of:
              1.1297096 = boost
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.050282832 = queryNorm
            2.8468423 = fieldWeight in 5058, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.3125 = fieldNorm(doc=5058)
        1.4732056 = weight(author_txt:zhitomirsky in 5058) [ClassicSimilarity], result of:
          1.4732056 = score(doc=5058,freq=1.0), product of:
            0.5174876 = queryWeight, product of:
              1.1297096 = boost
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.050282832 = queryNorm
            2.8468423 = fieldWeight in 5058, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.3125 = fieldNorm(doc=5058)
        1.6041914 = weight(author_txt:levene in 5058) [ClassicSimilarity], result of:
          1.6041914 = score(doc=5058,freq=1.0), product of:
            0.54772407 = queryWeight, product of:
              1.1622452 = boost
              9.37226 = idf(docFreq=9, maxDocs=43254)
              0.050282832 = queryNorm
            2.9288313 = fieldWeight in 5058, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.37226 = idf(docFreq=9, maxDocs=43254)
              0.3125 = fieldNorm(doc=5058)
    
  2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Categorical relevance judgment (2018) 5.57
    5.5723953 = sum of:
      5.5723953 = sum of:
        1.0217929 = weight(author_txt:ilan in 458) [ClassicSimilarity], result of:
          1.0217929 = score(doc=458,freq=1.0), product of:
            0.40547708 = queryWeight, product of:
              8.063927 = idf(docFreq=36, maxDocs=43254)
              0.050282832 = queryNorm
            2.519977 = fieldWeight in 458, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              8.063927 = idf(docFreq=36, maxDocs=43254)
              0.3125 = fieldNorm(doc=458)
        1.4732056 = weight(author_txt:geffet in 458) [ClassicSimilarity], result of:
          1.4732056 = score(doc=458,freq=1.0), product of:
            0.5174876 = queryWeight, product of:
              1.1297096 = boost
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.050282832 = queryNorm
            2.8468423 = fieldWeight in 458, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.3125 = fieldNorm(doc=458)
        1.4732056 = weight(author_txt:zhitomirsky in 458) [ClassicSimilarity], result of:
          1.4732056 = score(doc=458,freq=1.0), product of:
            0.5174876 = queryWeight, product of:
              1.1297096 = boost
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.050282832 = queryNorm
            2.8468423 = fieldWeight in 458, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.109896 = idf(docFreq=12, maxDocs=43254)
              0.3125 = fieldNorm(doc=458)
        1.6041914 = weight(author_txt:levene in 458) [ClassicSimilarity], result of:
          1.6041914 = score(doc=458,freq=1.0), product of:
            0.54772407 = queryWeight, product of:
              1.1622452 = boost
              9.37226 = idf(docFreq=9, maxDocs=43254)
              0.050282832 = queryNorm
            2.9288313 = fieldWeight in 458, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.37226 = idf(docFreq=9, maxDocs=43254)
              0.3125 = fieldNorm(doc=458)
    
  3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.: Towards maximal unification of semantically diverse ontologies for controversial domains (2014) 3.57
    3.571384 = sum of:
      3.571384 = product of:
        4.761845 = sum of:
          1.2261516 = weight(author_txt:ilan in 3099) [ClassicSimilarity], result of:
            1.2261516 = score(doc=3099,freq=1.0), product of:
              0.40547708 = queryWeight, product of:
                8.063927 = idf(docFreq=36, maxDocs=43254)
                0.050282832 = queryNorm
              3.0239725 = fieldWeight in 3099, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.063927 = idf(docFreq=36, maxDocs=43254)
                0.375 = fieldNorm(doc=3099)
          1.7678467 = weight(author_txt:geffet in 3099) [ClassicSimilarity], result of:
            1.7678467 = score(doc=3099,freq=1.0), product of:
              0.5174876 = queryWeight, product of:
                1.1297096 = boost
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.050282832 = queryNorm
              3.416211 = fieldWeight in 3099, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.375 = fieldNorm(doc=3099)
          1.7678467 = weight(author_txt:zhitomirsky in 3099) [ClassicSimilarity], result of:
            1.7678467 = score(doc=3099,freq=1.0), product of:
              0.5174876 = queryWeight, product of:
                1.1297096 = boost
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.050282832 = queryNorm
              3.416211 = fieldWeight in 3099, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.375 = fieldNorm(doc=3099)
        0.75 = coord(3/4)
    
  4. Bar-Ilan, J.; Zhitomirsky-Geffet, M.; Miller, Y.; Shoham, S.: ¬The effects of background information and social interaction on image tagging (2010) 2.98
    2.976153 = sum of:
      2.976153 = product of:
        3.968204 = sum of:
          1.0217929 = weight(author_txt:ilan in 454) [ClassicSimilarity], result of:
            1.0217929 = score(doc=454,freq=1.0), product of:
              0.40547708 = queryWeight, product of:
                8.063927 = idf(docFreq=36, maxDocs=43254)
                0.050282832 = queryNorm
              2.519977 = fieldWeight in 454, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.063927 = idf(docFreq=36, maxDocs=43254)
                0.3125 = fieldNorm(doc=454)
          1.4732056 = weight(author_txt:geffet in 454) [ClassicSimilarity], result of:
            1.4732056 = score(doc=454,freq=1.0), product of:
              0.5174876 = queryWeight, product of:
                1.1297096 = boost
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.050282832 = queryNorm
              2.8468423 = fieldWeight in 454, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.3125 = fieldNorm(doc=454)
          1.4732056 = weight(author_txt:zhitomirsky in 454) [ClassicSimilarity], result of:
            1.4732056 = score(doc=454,freq=1.0), product of:
              0.5174876 = queryWeight, product of:
                1.1297096 = boost
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.050282832 = queryNorm
              2.8468423 = fieldWeight in 454, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.3125 = fieldNorm(doc=454)
        0.75 = coord(3/4)
    
  5. Zhitomirsky-Geffet, M.; Erez, E.S.; Bar-Ilan, J.: Toward multiviewpoint ontology construction by collaboration of non-experts and crowdsourcing : the case of the effect of diet on health (2017) 2.98
    2.976153 = sum of:
      2.976153 = product of:
        3.968204 = sum of:
          1.0217929 = weight(author_txt:ilan in 4904) [ClassicSimilarity], result of:
            1.0217929 = score(doc=4904,freq=1.0), product of:
              0.40547708 = queryWeight, product of:
                8.063927 = idf(docFreq=36, maxDocs=43254)
                0.050282832 = queryNorm
              2.519977 = fieldWeight in 4904, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.063927 = idf(docFreq=36, maxDocs=43254)
                0.3125 = fieldNorm(doc=4904)
          1.4732056 = weight(author_txt:geffet in 4904) [ClassicSimilarity], result of:
            1.4732056 = score(doc=4904,freq=1.0), product of:
              0.5174876 = queryWeight, product of:
                1.1297096 = boost
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.050282832 = queryNorm
              2.8468423 = fieldWeight in 4904, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.3125 = fieldNorm(doc=4904)
          1.4732056 = weight(author_txt:zhitomirsky in 4904) [ClassicSimilarity], result of:
            1.4732056 = score(doc=4904,freq=1.0), product of:
              0.5174876 = queryWeight, product of:
                1.1297096 = boost
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.050282832 = queryNorm
              2.8468423 = fieldWeight in 4904, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.109896 = idf(docFreq=12, maxDocs=43254)
                0.3125 = fieldNorm(doc=4904)
        0.75 = coord(3/4)
    

Similar documents (content)

  1. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.59
    0.5939196 = sum of:
      0.5939196 = product of:
        1.0605707 = sum of:
          0.034276806 = weight(abstract_txt:ranking in 5058) [ClassicSimilarity], result of:
            0.034276806 = score(doc=5058,freq=2.0), product of:
              0.06922735 = queryWeight, product of:
                5.6018004 = idf(docFreq=433, maxDocs=43254)
                0.012358054 = queryNorm
              0.49513388 = fieldWeight in 5058, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.6018004 = idf(docFreq=433, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.018723859 = weight(abstract_txt:evaluation in 5058) [ClassicSimilarity], result of:
            0.018723859 = score(doc=5058,freq=1.0), product of:
              0.06671879 = queryWeight, product of:
                1.2023499 = boost
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.012358054 = queryNorm
              0.2806385 = fieldWeight in 5058, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.022692626 = weight(abstract_txt:study in 5058) [ClassicSimilarity], result of:
            0.022692626 = score(doc=5058,freq=4.0), product of:
              0.052585658 = queryWeight, product of:
                1.2325649 = boost
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.012358054 = queryNorm
              0.4315364 = fieldWeight in 5058, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.026453568 = weight(abstract_txt:results in 5058) [ClassicSimilarity], result of:
            0.026453568 = score(doc=5058,freq=5.0), product of:
              0.054070905 = queryWeight, product of:
                1.2498502 = boost
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.012358054 = queryNorm
              0.4892385 = fieldWeight in 5058, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.010298738 = weight(abstract_txt:this in 5058) [ClassicSimilarity], result of:
            0.010298738 = score(doc=5058,freq=3.0), product of:
              0.039127026 = queryWeight, product of:
                1.3021475 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.012358054 = queryNorm
              0.26321292 = fieldWeight in 5058, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.04965868 = weight(abstract_txt:relevance in 5058) [ClassicSimilarity], result of:
            0.04965868 = score(doc=5058,freq=4.0), product of:
              0.08053067 = queryWeight, product of:
                1.3209532 = boost
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.012358054 = queryNorm
              0.616643 = fieldWeight in 5058, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.041407194 = weight(abstract_txt:point in 5058) [ClassicSimilarity], result of:
            0.041407194 = score(doc=5058,freq=2.0), product of:
              0.08988594 = queryWeight, product of:
                1.3955734 = boost
                5.2118154 = idf(docFreq=640, maxDocs=43254)
                0.012358054 = queryNorm
              0.46066374 = fieldWeight in 5058, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.2118154 = idf(docFreq=640, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.05970175 = weight(abstract_txt:scale in 5058) [ClassicSimilarity], result of:
            0.05970175 = score(doc=5058,freq=3.0), product of:
              0.10021564 = queryWeight, product of:
                1.4735829 = boost
                5.5031443 = idf(docFreq=478, maxDocs=43254)
                0.012358054 = queryNorm
              0.5957328 = fieldWeight in 5058, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                5.5031443 = idf(docFreq=478, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.021268686 = weight(abstract_txt:over in 5058) [ClassicSimilarity], result of:
            0.021268686 = score(doc=5058,freq=1.0), product of:
              0.07994499 = queryWeight, product of:
                1.5197487 = boost
                4.2566643 = idf(docFreq=1665, maxDocs=43254)
                0.012358054 = queryNorm
              0.26604152 = fieldWeight in 5058, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.2566643 = idf(docFreq=1665, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.06732556 = weight(abstract_txt:rankings in 5058) [ClassicSimilarity], result of:
            0.06732556 = score(doc=5058,freq=1.0), product of:
              0.1565926 = queryWeight, product of:
                1.8420117 = boost
                6.8790545 = idf(docFreq=120, maxDocs=43254)
                0.012358054 = queryNorm
              0.4299409 = fieldWeight in 5058, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.8790545 = idf(docFreq=120, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.046064824 = weight(abstract_txt:user in 5058) [ClassicSimilarity], result of:
            0.046064824 = score(doc=5058,freq=5.0), product of:
              0.0895882 = queryWeight, product of:
                1.9703673 = boost
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.012358054 = queryNorm
              0.51418406 = fieldWeight in 5058, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.042081542 = weight(abstract_txt:time in 5058) [ClassicSimilarity], result of:
            0.042081542 = score(doc=5058,freq=2.0), product of:
              0.11447544 = queryWeight, product of:
                2.2272959 = boost
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.012358054 = queryNorm
              0.36760324 = fieldWeight in 5058, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.052384213 = weight(abstract_txt:search in 5058) [ClassicSimilarity], result of:
            0.052384213 = score(doc=5058,freq=3.0), product of:
              0.13247019 = queryWeight, product of:
                2.9344485 = boost
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.012358054 = queryNorm
              0.3954415 = fieldWeight in 5058, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
          0.5682326 = weight(abstract_txt:judgements in 5058) [ClassicSimilarity], result of:
            0.5682326 = score(doc=5058,freq=5.0), product of:
              0.547502 = queryWeight, product of:
                5.965684 = boost
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.012358054 = queryNorm
              1.037864 = fieldWeight in 5058, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.0625 = fieldNorm(doc=5058)
        0.56 = coord(14/25)
    
  2. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.21
    0.2113337 = sum of:
      0.2113337 = product of:
        0.75476325 = sum of:
          0.029478585 = weight(abstract_txt:study in 5269) [ClassicSimilarity], result of:
            0.029478585 = score(doc=5269,freq=3.0), product of:
              0.052585658 = queryWeight, product of:
                1.2325649 = boost
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.012358054 = queryNorm
              0.5605822 = fieldWeight in 5269, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.09375 = fieldNorm(doc=5269)
          0.017745594 = weight(abstract_txt:results in 5269) [ClassicSimilarity], result of:
            0.017745594 = score(doc=5269,freq=1.0), product of:
              0.054070905 = queryWeight, product of:
                1.2498502 = boost
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.012358054 = queryNorm
              0.32819116 = fieldWeight in 5269, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.09375 = fieldNorm(doc=5269)
          0.00891897 = weight(abstract_txt:this in 5269) [ClassicSimilarity], result of:
            0.00891897 = score(doc=5269,freq=1.0), product of:
              0.039127026 = queryWeight, product of:
                1.3021475 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.012358054 = queryNorm
              0.22794908 = fieldWeight in 5269, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.09375 = fieldNorm(doc=5269)
          0.08328013 = weight(abstract_txt:relevance in 5269) [ClassicSimilarity], result of:
            0.08328013 = score(doc=5269,freq=5.0), product of:
              0.08053067 = queryWeight, product of:
                1.3209532 = boost
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.012358054 = queryNorm
              1.0341418 = fieldWeight in 5269, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.09375 = fieldNorm(doc=5269)
          0.030901223 = weight(abstract_txt:user in 5269) [ClassicSimilarity], result of:
            0.030901223 = score(doc=5269,freq=1.0), product of:
              0.0895882 = queryWeight, product of:
                1.9703673 = boost
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.012358054 = queryNorm
              0.34492514 = fieldWeight in 5269, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.09375 = fieldNorm(doc=5269)
          0.04536606 = weight(abstract_txt:search in 5269) [ClassicSimilarity], result of:
            0.04536606 = score(doc=5269,freq=1.0), product of:
              0.13247019 = queryWeight, product of:
                2.9344485 = boost
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.012358054 = queryNorm
              0.3424624 = fieldWeight in 5269, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.09375 = fieldNorm(doc=5269)
          0.5390727 = weight(abstract_txt:judgements in 5269) [ClassicSimilarity], result of:
            0.5390727 = score(doc=5269,freq=2.0), product of:
              0.547502 = queryWeight, product of:
                5.965684 = boost
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.012358054 = queryNorm
              0.9846041 = fieldWeight in 5269, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.09375 = fieldNorm(doc=5269)
        0.28 = coord(7/25)
    
  3. Su, L.T.: ¬A comprehensive and systematic model of user evaluation of Web search engines : Il. An evaluation by undergraduates (2003) 0.21
    0.21004498 = sum of:
      0.21004498 = product of:
        0.52511245 = sum of:
          0.032766756 = weight(abstract_txt:evaluation in 4118) [ClassicSimilarity], result of:
            0.032766756 = score(doc=4118,freq=4.0), product of:
              0.06671879 = queryWeight, product of:
                1.2023499 = boost
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.012358054 = queryNorm
              0.49111736 = fieldWeight in 4118, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.01719584 = weight(abstract_txt:study in 4118) [ClassicSimilarity], result of:
            0.01719584 = score(doc=4118,freq=3.0), product of:
              0.052585658 = queryWeight, product of:
                1.2325649 = boost
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.012358054 = queryNorm
              0.32700628 = fieldWeight in 4118, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.01792949 = weight(abstract_txt:results in 4118) [ClassicSimilarity], result of:
            0.01792949 = score(doc=4118,freq=3.0), product of:
              0.054070905 = queryWeight, product of:
                1.2498502 = boost
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.012358054 = queryNorm
              0.3315922 = fieldWeight in 4118, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.005202732 = weight(abstract_txt:this in 4118) [ClassicSimilarity], result of:
            0.005202732 = score(doc=4118,freq=1.0), product of:
              0.039127026 = queryWeight, product of:
                1.3021475 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.012358054 = queryNorm
              0.13297029 = fieldWeight in 4118, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.030724738 = weight(abstract_txt:relevance in 4118) [ClassicSimilarity], result of:
            0.030724738 = score(doc=4118,freq=2.0), product of:
              0.08053067 = queryWeight, product of:
                1.3209532 = boost
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.012358054 = queryNorm
              0.38152838 = fieldWeight in 4118, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.047691554 = weight(abstract_txt:user in 4118) [ClassicSimilarity], result of:
            0.047691554 = score(doc=4118,freq=7.0), product of:
              0.0895882 = queryWeight, product of:
                1.9703673 = boost
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.012358054 = queryNorm
              0.5323419 = fieldWeight in 4118, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.035033233 = weight(abstract_txt:individual in 4118) [ClassicSimilarity], result of:
            0.035033233 = score(doc=4118,freq=1.0), product of:
              0.13129522 = queryWeight, product of:
                2.1774871 = boost
                4.8791394 = idf(docFreq=893, maxDocs=43254)
                0.012358054 = queryNorm
              0.26682794 = fieldWeight in 4118, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.8791394 = idf(docFreq=893, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.036821347 = weight(abstract_txt:time in 4118) [ClassicSimilarity], result of:
            0.036821347 = score(doc=4118,freq=2.0), product of:
              0.11447544 = queryWeight, product of:
                2.2272959 = boost
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.012358054 = queryNorm
              0.32165283 = fieldWeight in 4118, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.07939061 = weight(abstract_txt:search in 4118) [ClassicSimilarity], result of:
            0.07939061 = score(doc=4118,freq=9.0), product of:
              0.13247019 = queryWeight, product of:
                2.9344485 = boost
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.012358054 = queryNorm
              0.5993092 = fieldWeight in 4118, product of:
                3.0 = tf(freq=9.0), with freq of:
                  9.0 = termFreq=9.0
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
          0.22235616 = weight(abstract_txt:judgements in 4118) [ClassicSimilarity], result of:
            0.22235616 = score(doc=4118,freq=1.0), product of:
              0.547502 = queryWeight, product of:
                5.965684 = boost
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.012358054 = queryNorm
              0.4061285 = fieldWeight in 4118, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.0546875 = fieldNorm(doc=4118)
        0.4 = coord(10/25)
    
  4. Dennis, S.; Bruza, P.; McArthur, R.: Web searching : a process-oriented experimental study of three interactive search paradigms (2002) 0.20
    0.19563982 = sum of:
      0.19563982 = product of:
        0.5434439 = sum of:
          0.011346313 = weight(abstract_txt:study in 2201) [ClassicSimilarity], result of:
            0.011346313 = score(doc=2201,freq=1.0), product of:
              0.052585658 = queryWeight, product of:
                1.2325649 = boost
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.012358054 = queryNorm
              0.2157682 = fieldWeight in 2201, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4522913 = idf(docFreq=3723, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.011830395 = weight(abstract_txt:results in 2201) [ClassicSimilarity], result of:
            0.011830395 = score(doc=2201,freq=1.0), product of:
              0.054070905 = queryWeight, product of:
                1.2498502 = boost
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.012358054 = queryNorm
              0.2187941 = fieldWeight in 2201, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.0084088845 = weight(abstract_txt:this in 2201) [ClassicSimilarity], result of:
            0.0084088845 = score(doc=2201,freq=2.0), product of:
              0.039127026 = queryWeight, product of:
                1.3021475 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.012358054 = queryNorm
              0.21491244 = fieldWeight in 2201, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.04965868 = weight(abstract_txt:relevance in 2201) [ClassicSimilarity], result of:
            0.04965868 = score(doc=2201,freq=4.0), product of:
              0.08053067 = queryWeight, product of:
                1.3209532 = boost
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.012358054 = queryNorm
              0.616643 = fieldWeight in 2201, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.933144 = idf(docFreq=846, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.021268686 = weight(abstract_txt:over in 2201) [ClassicSimilarity], result of:
            0.021268686 = score(doc=2201,freq=1.0), product of:
              0.07994499 = queryWeight, product of:
                1.5197487 = boost
                4.2566643 = idf(docFreq=1665, maxDocs=43254)
                0.012358054 = queryNorm
              0.26604152 = fieldWeight in 2201, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.2566643 = idf(docFreq=1665, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.035681657 = weight(abstract_txt:user in 2201) [ClassicSimilarity], result of:
            0.035681657 = score(doc=2201,freq=3.0), product of:
              0.0895882 = queryWeight, product of:
                1.9703673 = boost
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.012358054 = queryNorm
              0.3982852 = fieldWeight in 2201, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                3.6792014 = idf(docFreq=2967, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.042081542 = weight(abstract_txt:time in 2201) [ClassicSimilarity], result of:
            0.042081542 = score(doc=2201,freq=2.0), product of:
              0.11447544 = queryWeight, product of:
                2.2272959 = boost
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.012358054 = queryNorm
              0.36760324 = fieldWeight in 2201, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.10904644 = weight(abstract_txt:search in 2201) [ClassicSimilarity], result of:
            0.10904644 = score(doc=2201,freq=13.0), product of:
              0.13247019 = queryWeight, product of:
                2.9344485 = boost
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.012358054 = queryNorm
              0.82317716 = fieldWeight in 2201, product of:
                3.6055512 = tf(freq=13.0), with freq of:
                  13.0 = termFreq=13.0
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
          0.25412133 = weight(abstract_txt:judgements in 2201) [ClassicSimilarity], result of:
            0.25412133 = score(doc=2201,freq=1.0), product of:
              0.547502 = queryWeight, product of:
                5.965684 = boost
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.012358054 = queryNorm
              0.46414685 = fieldWeight in 2201, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.4263496 = idf(docFreq=69, maxDocs=43254)
                0.0625 = fieldNorm(doc=2201)
        0.36 = coord(9/25)
    
  5. Bar-Ilan, J.; Levene, M.; Mat-Hassan, M.: Methods for evaluating dynamic changes in search engine rankings : a case study (2006) 0.19
    0.1942902 = sum of:
      0.1942902 = product of:
        0.4857255 = sum of:
          0.029992206 = weight(abstract_txt:ranking in 1742) [ClassicSimilarity], result of:
            0.029992206 = score(doc=1742,freq=2.0), product of:
              0.06922735 = queryWeight, product of:
                5.6018004 = idf(docFreq=433, maxDocs=43254)
                0.012358054 = queryNorm
              0.43324214 = fieldWeight in 1742, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.6018004 = idf(docFreq=433, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.030330056 = weight(abstract_txt:compare in 1742) [ClassicSimilarity], result of:
            0.030330056 = score(doc=1742,freq=2.0), product of:
              0.06974626 = queryWeight, product of:
                1.0037409 = boost
                5.622756 = idf(docFreq=424, maxDocs=43254)
                0.012358054 = queryNorm
              0.43486282 = fieldWeight in 1742, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.622756 = idf(docFreq=424, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.027387748 = weight(abstract_txt:results in 1742) [ClassicSimilarity], result of:
            0.027387748 = score(doc=1742,freq=7.0), product of:
              0.054070905 = queryWeight, product of:
                1.2498502 = boost
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.012358054 = queryNorm
              0.50651544 = fieldWeight in 1742, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                3.5007057 = idf(docFreq=3547, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.044983894 = weight(abstract_txt:stable in 1742) [ClassicSimilarity], result of:
            0.044983894 = score(doc=1742,freq=1.0), product of:
              0.11428436 = queryWeight, product of:
                1.2848562 = boost
                7.1975083 = idf(docFreq=87, maxDocs=43254)
                0.012358054 = queryNorm
              0.39361373 = fieldWeight in 1742, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.1975083 = idf(docFreq=87, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.0073577743 = weight(abstract_txt:this in 1742) [ClassicSimilarity], result of:
            0.0073577743 = score(doc=1742,freq=2.0), product of:
              0.039127026 = queryWeight, product of:
                1.3021475 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.012358054 = queryNorm
              0.18804839 = fieldWeight in 1742, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.025619395 = weight(abstract_txt:point in 1742) [ClassicSimilarity], result of:
            0.025619395 = score(doc=1742,freq=1.0), product of:
              0.08988594 = queryWeight, product of:
                1.3955734 = boost
                5.2118154 = idf(docFreq=640, maxDocs=43254)
                0.012358054 = queryNorm
              0.28502116 = fieldWeight in 1742, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.2118154 = idf(docFreq=640, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.037220202 = weight(abstract_txt:over in 1742) [ClassicSimilarity], result of:
            0.037220202 = score(doc=1742,freq=4.0), product of:
              0.07994499 = queryWeight, product of:
                1.5197487 = boost
                4.2566643 = idf(docFreq=1665, maxDocs=43254)
                0.012358054 = queryNorm
              0.46557266 = fieldWeight in 1742, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.2566643 = idf(docFreq=1665, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.16662225 = weight(abstract_txt:rankings in 1742) [ClassicSimilarity], result of:
            0.16662225 = score(doc=1742,freq=8.0), product of:
              0.1565926 = queryWeight, product of:
                1.8420117 = boost
                6.8790545 = idf(docFreq=120, maxDocs=43254)
                0.012358054 = queryNorm
              1.0640494 = fieldWeight in 1742, product of:
                2.828427 = tf(freq=8.0), with freq of:
                  8.0 = termFreq=8.0
                6.8790545 = idf(docFreq=120, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.036821347 = weight(abstract_txt:time in 1742) [ClassicSimilarity], result of:
            0.036821347 = score(doc=1742,freq=2.0), product of:
              0.11447544 = queryWeight, product of:
                2.2272959 = boost
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.012358054 = queryNorm
              0.32165283 = fieldWeight in 1742, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.158956 = idf(docFreq=1836, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
          0.07939061 = weight(abstract_txt:search in 1742) [ClassicSimilarity], result of:
            0.07939061 = score(doc=1742,freq=9.0), product of:
              0.13247019 = queryWeight, product of:
                2.9344485 = boost
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.012358054 = queryNorm
              0.5993092 = fieldWeight in 1742, product of:
                3.0 = tf(freq=9.0), with freq of:
                  9.0 = termFreq=9.0
                3.6529322 = idf(docFreq=3046, maxDocs=43254)
                0.0546875 = fieldNorm(doc=1742)
        0.4 = coord(10/25)