Document (#33878)

Author

Xu, Y.
Wang, D.

Title

Order effect in relevance judgment : mediation and causality

Source

Journal of the American Society for Information Science and Technology. 59(2008) no.8, S.1264-1275

Year

2008

Abstract

The order effect of relevance judgment refers to the different relevance perceptions of a document when it appears in different positions in a list. Although the order effect of relevance judgment has significant theoretical and practical implications, the extant literature is inconclusive regarding its existence and forming mechanisms. This study proposes a set of order effect forming mechanisms, including the learning effect, the subneed scheduling effect, and the cursoriness effect based on the conceptualization of dynamic relevance and the psychology of cognitive elaboration. Our empirical study indicates that in an interactive information retrieval setting, when a document list is reasonably long, order effects demonstrate a curvilinear pattern that conforms to the combined effect of the three mechanisms. Moreover, the curvilinear pattern of order effect could differ for documents of different relevance levels.

Theme

Retrievalalgorithmen

Similar documents (author)

1. Wang, H.; Wang, C.: Ontologies for universal information systems (1995) 4.64

   4.63939 = sum of:
     4.63939 = weight(author_txt:wang in 3194) [ClassicSimilarity], result of:
       4.63939 = fieldWeight in 3194, product of:
         1.4142135 = tf(freq=2.0), with freq of:
           2.0 = termFreq=2.0
         6.5610886 = idf(docFreq=169, maxDocs=44218)
         0.5 = fieldNorm(doc=3194)
   

2. Wang, F.; Wang, X.: Tracing theory diffusion : a text mining and citation-based analysis of TAM (2020) 4.64

   4.63939 = sum of:
     4.63939 = weight(author_txt:wang in 5980) [ClassicSimilarity], result of:
       4.63939 = fieldWeight in 5980, product of:
         1.4142135 = tf(freq=2.0), with freq of:
           2.0 = termFreq=2.0
         6.5610886 = idf(docFreq=169, maxDocs=44218)
         0.5 = fieldNorm(doc=5980)
   

3. Wang, C.: ¬The online catalogue, subject access and user reactions : a review (1985) 4.10

   4.1006804 = sum of:
     4.1006804 = weight(author_txt:wang in 986) [ClassicSimilarity], result of:
       4.1006804 = fieldWeight in 986, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         6.5610886 = idf(docFreq=169, maxDocs=44218)
         0.625 = fieldNorm(doc=986)
   

4. Wang, C.: Bibliometrics : a textbook (1990) 4.10

   4.1006804 = sum of:
     4.1006804 = weight(author_txt:wang in 5040) [ClassicSimilarity], result of:
       4.1006804 = fieldWeight in 5040, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         6.5610886 = idf(docFreq=169, maxDocs=44218)
         0.625 = fieldNorm(doc=5040)
   

5. Wang, P.: Users' information needs at different stages of a research project : a cognitive view (1997) 4.10

   4.1006804 = sum of:
     4.1006804 = weight(author_txt:wang in 320) [ClassicSimilarity], result of:
       4.1006804 = fieldWeight in 320, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         6.5610886 = idf(docFreq=169, maxDocs=44218)
         0.625 = fieldNorm(doc=320)
   

Similar documents (content)

1. Xu, Y.: Relevance judgment in epistemic and hedonic information searches (2007) 0.20

   0.2023641 = sum of:
     0.2023641 = product of:
       0.84318376 = sum of:
         0.022819316 = weight(abstract_txt:study in 64) [ClassicSimilarity], result of:
           0.022819316 = score(doc=64,freq=3.0), product of:
             0.04925415 = queryWeight, product of:
               1.0537397 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.013652127 = queryNorm
             0.46329734 = fieldWeight in 64, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.07929619 = weight(abstract_txt:extant in 64) [ClassicSimilarity], result of:
           0.07929619 = score(doc=64,freq=1.0), product of:
             0.12935196 = queryWeight, product of:
               1.2074897 = boost
               7.84674 = idf(docFreq=46, maxDocs=44218)
               0.013652127 = queryNorm
             0.61302656 = fieldWeight in 64, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.84674 = idf(docFreq=46, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.023433374 = weight(abstract_txt:when in 64) [ClassicSimilarity], result of:
           0.023433374 = score(doc=64,freq=1.0), product of:
             0.07230551 = queryWeight, product of:
               1.2767258 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.013652127 = queryNorm
             0.32408836 = fieldWeight in 64, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.025964402 = weight(abstract_txt:document in 64) [ClassicSimilarity], result of:
           0.025964402 = score(doc=64,freq=1.0), product of:
             0.07742247 = queryWeight, product of:
               1.3211297 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.013652127 = queryNorm
             0.33536002 = fieldWeight in 64, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.33727667 = weight(abstract_txt:judgment in 64) [ClassicSimilarity], result of:
           0.33727667 = score(doc=64,freq=3.0), product of:
             0.33956978 = queryWeight, product of:
               3.3886137 = boost
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.013652127 = queryNorm
             0.99324703 = fieldWeight in 64, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.3543938 = weight(abstract_txt:relevance in 64) [ClassicSimilarity], result of:
           0.3543938 = score(doc=64,freq=9.0), product of:
             0.30659506 = queryWeight, product of:
               4.5536027 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013652127 = queryNorm
             1.1559019 = fieldWeight in 64, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
       0.24 = coord(6/25)
   

2. Chu, H.: Factors affecting relevance judgment : a report from TREC Legal track (2011) 0.15

   0.15191996 = sum of:
     0.15191996 = product of:
       0.7595998 = sum of:
         0.021079581 = weight(abstract_txt:study in 4540) [ClassicSimilarity], result of:
           0.021079581 = score(doc=4540,freq=4.0), product of:
             0.04925415 = queryWeight, product of:
               1.0537397 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.013652127 = queryNorm
             0.42797574 = fieldWeight in 4540, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
         0.0187467 = weight(abstract_txt:when in 4540) [ClassicSimilarity], result of:
           0.0187467 = score(doc=4540,freq=1.0), product of:
             0.07230551 = queryWeight, product of:
               1.2767258 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.013652127 = queryNorm
             0.2592707 = fieldWeight in 4540, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
         0.057998072 = weight(abstract_txt:list in 4540) [ClassicSimilarity], result of:
           0.057998072 = score(doc=4540,freq=2.0), product of:
             0.1218486 = queryWeight, product of:
               1.6573805 = boost
               5.3851523 = idf(docFreq=550, maxDocs=44218)
               0.013652127 = queryNorm
             0.47598472 = fieldWeight in 4540, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.3851523 = idf(docFreq=550, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
         0.34833783 = weight(abstract_txt:judgment in 4540) [ClassicSimilarity], result of:
           0.34833783 = score(doc=4540,freq=5.0), product of:
             0.33956978 = queryWeight, product of:
               3.3886137 = boost
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.013652127 = queryNorm
             1.0258211 = fieldWeight in 4540, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
         0.31343767 = weight(abstract_txt:relevance in 4540) [ClassicSimilarity], result of:
           0.31343767 = score(doc=4540,freq=11.0), product of:
             0.30659506 = queryWeight, product of:
               4.5536027 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013652127 = queryNorm
             1.0223181 = fieldWeight in 4540, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
       0.2 = coord(5/25)
   

3. Huang, M.-h.; Wang, H.-y.: ¬The influence of document presentation order and number of documents judged an users' judgments of relevance (2004) 0.15

   0.14835654 = sum of:
     0.14835654 = product of:
       0.92722845 = sum of:
         0.046395715 = weight(abstract_txt:when in 2885) [ClassicSimilarity], result of:
           0.046395715 = score(doc=2885,freq=2.0), product of:
             0.07230551 = queryWeight, product of:
               1.2767258 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.013652127 = queryNorm
             0.64166224 = fieldWeight in 2885, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.109375 = fieldNorm(doc=2885)
         0.03635016 = weight(abstract_txt:document in 2885) [ClassicSimilarity], result of:
           0.03635016 = score(doc=2885,freq=1.0), product of:
             0.07742247 = queryWeight, product of:
               1.3211297 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.013652127 = queryNorm
             0.46950403 = fieldWeight in 2885, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.109375 = fieldNorm(doc=2885)
         0.27110595 = weight(abstract_txt:order in 2885) [ClassicSimilarity], result of:
           0.27110595 = score(doc=2885,freq=5.0), product of:
             0.24927147 = queryWeight, product of:
               4.1059046 = boost
               4.446962 = idf(docFreq=1407, maxDocs=44218)
               0.013652127 = queryNorm
             1.0875932 = fieldWeight in 2885, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.446962 = idf(docFreq=1407, maxDocs=44218)
               0.109375 = fieldNorm(doc=2885)
         0.57337666 = weight(abstract_txt:effect in 2885) [ClassicSimilarity], result of:
           0.57337666 = score(doc=2885,freq=3.0), product of:
             0.5574252 = queryWeight, product of:
               7.51989 = boost
               5.4296865 = idf(docFreq=526, maxDocs=44218)
               0.013652127 = queryNorm
             1.0286163 = fieldWeight in 2885, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4296865 = idf(docFreq=526, maxDocs=44218)
               0.109375 = fieldNorm(doc=2885)
       0.16 = coord(4/25)
   

4. Balatsoukas, P.; Ruthven, I.: ¬An eye-tracking approach to the analysis of relevance judgments on the Web : the case of Google search engine (2012) 0.13

   0.1349609 = sum of:
     0.1349609 = product of:
       0.6748045 = sum of:
         0.010539791 = weight(abstract_txt:study in 379) [ClassicSimilarity], result of:
           0.010539791 = score(doc=379,freq=1.0), product of:
             0.04925415 = queryWeight, product of:
               1.0537397 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.013652127 = queryNorm
             0.21398787 = fieldWeight in 379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0625 = fieldNorm(doc=379)
         0.15578142 = weight(abstract_txt:judgment in 379) [ClassicSimilarity], result of:
           0.15578142 = score(doc=379,freq=1.0), product of:
             0.33956978 = queryWeight, product of:
               3.3886137 = boost
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.013652127 = queryNorm
             0.45876116 = fieldWeight in 379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.0625 = fieldNorm(doc=379)
         0.069281295 = weight(abstract_txt:order in 379) [ClassicSimilarity], result of:
           0.069281295 = score(doc=379,freq=1.0), product of:
             0.24927147 = queryWeight, product of:
               4.1059046 = boost
               4.446962 = idf(docFreq=1407, maxDocs=44218)
               0.013652127 = queryNorm
             0.27793512 = fieldWeight in 379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.446962 = idf(docFreq=1407, maxDocs=44218)
               0.0625 = fieldNorm(doc=379)
         0.25003675 = weight(abstract_txt:relevance in 379) [ClassicSimilarity], result of:
           0.25003675 = score(doc=379,freq=7.0), product of:
             0.30659506 = queryWeight, product of:
               4.5536027 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013652127 = queryNorm
             0.8155277 = fieldWeight in 379, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=379)
         0.18916526 = weight(abstract_txt:effect in 379) [ClassicSimilarity], result of:
           0.18916526 = score(doc=379,freq=1.0), product of:
             0.5574252 = queryWeight, product of:
               7.51989 = boost
               5.4296865 = idf(docFreq=526, maxDocs=44218)
               0.013652127 = queryNorm
             0.3393554 = fieldWeight in 379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4296865 = idf(docFreq=526, maxDocs=44218)
               0.0625 = fieldNorm(doc=379)
       0.2 = coord(5/25)
   

5. Xu, Y.; Yin, H.: Novelty and topicality in interactive information retrieval (2008) 0.13

   0.13311094 = sum of:
     0.13311094 = product of:
       0.6655547 = sum of:
         0.021079581 = weight(abstract_txt:study in 1355) [ClassicSimilarity], result of:
           0.021079581 = score(doc=1355,freq=4.0), product of:
             0.04925415 = queryWeight, product of:
               1.0537397 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.013652127 = queryNorm
             0.42797574 = fieldWeight in 1355, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.0625 = fieldNorm(doc=1355)
         0.02077152 = weight(abstract_txt:document in 1355) [ClassicSimilarity], result of:
           0.02077152 = score(doc=1355,freq=1.0), product of:
             0.07742247 = queryWeight, product of:
               1.3211297 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.013652127 = queryNorm
             0.26828802 = fieldWeight in 1355, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=1355)
         0.01939975 = weight(abstract_txt:different in 1355) [ClassicSimilarity], result of:
           0.01939975 = score(doc=1355,freq=1.0), product of:
             0.084680356 = queryWeight, product of:
               1.6921892 = boost
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.013652127 = queryNorm
             0.22909386 = fieldWeight in 1355, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6655018 = idf(docFreq=3075, maxDocs=44218)
               0.0625 = fieldNorm(doc=1355)
         0.4406164 = weight(abstract_txt:judgment in 1355) [ClassicSimilarity], result of:
           0.4406164 = score(doc=1355,freq=8.0), product of:
             0.33956978 = queryWeight, product of:
               3.3886137 = boost
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.013652127 = queryNorm
             1.2975725 = fieldWeight in 1355, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               7.3401785 = idf(docFreq=77, maxDocs=44218)
               0.0625 = fieldNorm(doc=1355)
         0.16368748 = weight(abstract_txt:relevance in 1355) [ClassicSimilarity], result of:
           0.16368748 = score(doc=1355,freq=3.0), product of:
             0.30659506 = queryWeight, product of:
               4.5536027 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013652127 = queryNorm
             0.5338882 = fieldWeight in 1355, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=1355)
       0.2 = coord(5/25)