Document (#3309)

Author

Burgin, R.

Title

Variations in relevance judgements and the evaluation of retrieval performance

Source

Information processing and management. 28(1992) no.5, S.619-627

Year

1992

Abstract

Relevance judgements used to evaluate the performance of information retrieval systems are known to vary among judges and to vary under certain conditions extraneous to the relevance relationship between queries and documents. Investigates the degree to which variations in relevance judgements affect the evaluation of retrieval performance. Four sets of relevance judgements were used to test the retrieval effectiveness of 6 document representations. In no case was there a noticeable or material difference in retrieval performance due to variations in relevance judgement. Detailed examination of reasons why variations in relevance judgements may not affect recall and precision

Similar documents (author)

1. Burgin, R.: ¬The retrieval effectiveness of 5 clustering algorithms as a function of indexing exhaustivity (1995) 6.01

   6.010904 = sum of:
     6.010904 = weight(author_txt:burgin in 3365) [ClassicSimilarity], result of:
       6.010904 = fieldWeight in 3365, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.625 = fieldNorm(doc=3365)
   

2. Burgin, M.: Terminological and teleological systems (1996) 6.01

   6.010904 = sum of:
     6.010904 = weight(author_txt:burgin in 6323) [ClassicSimilarity], result of:
       6.010904 = fieldWeight in 6323, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.625 = fieldNorm(doc=6323)
   

3. Burgin, R.: ¬The Monte Carlo method and the evaluation of retrieval system performance (1999) 6.01

   6.010904 = sum of:
     6.010904 = weight(author_txt:burgin in 2946) [ClassicSimilarity], result of:
       6.010904 = fieldWeight in 2946, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.625 = fieldNorm(doc=2946)
   

4. Burgin, R.: ¬The effect of indexing exhaustivity on retrieval performance (1991) 6.01

   6.010904 = sum of:
     6.010904 = weight(author_txt:burgin in 5262) [ClassicSimilarity], result of:
       6.010904 = fieldWeight in 5262, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.625 = fieldNorm(doc=5262)
   

5. Shaw, W.M.; Burgin, R.; Howell, P.: Performance standards and evaluations in IR test collections : cluster-based retrieval models (1997) 3.61

   3.606542 = sum of:
     3.606542 = weight(author_txt:burgin in 7256) [ClassicSimilarity], result of:
       3.606542 = fieldWeight in 7256, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.375 = fieldNorm(doc=7256)
   

Similar documents (content)

1. Wallis, P.; Thom, J.A.: Relevance judgements for assessing recall (1996) 0.30

   0.3035133 = sum of:
     0.3035133 = product of:
       1.083976 = sum of:
         0.041802477 = weight(abstract_txt:precision in 4949) [ClassicSimilarity], result of:
           0.041802477 = score(doc=4949,freq=1.0), product of:
             0.06917306 = queryWeight, product of:
               1.0378729 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.012062738 = queryNorm
             0.6043173 = fieldWeight in 4949, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.109375 = fieldNorm(doc=4949)
         0.06659163 = weight(abstract_txt:recall in 4949) [ClassicSimilarity], result of:
           0.06659163 = score(doc=4949,freq=2.0), product of:
             0.07488682 = queryWeight, product of:
               1.0798872 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.012062738 = queryNorm
             0.88923025 = fieldWeight in 4949, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.109375 = fieldNorm(doc=4949)
         0.018790454 = weight(abstract_txt:used in 4949) [ClassicSimilarity], result of:
           0.018790454 = score(doc=4949,freq=1.0), product of:
             0.05114115 = queryWeight, product of:
               1.2620481 = boost
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.012062738 = queryNorm
             0.36742336 = fieldWeight in 4949, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.109375 = fieldNorm(doc=4949)
         0.05200533 = weight(abstract_txt:retrieval in 4949) [ClassicSimilarity], result of:
           0.05200533 = score(doc=4949,freq=1.0), product of:
             0.1368225 = queryWeight, product of:
               3.2639196 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.012062738 = queryNorm
             0.38009337 = fieldWeight in 4949, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.109375 = fieldNorm(doc=4949)
         0.098419785 = weight(abstract_txt:performance in 4949) [ClassicSimilarity], result of:
           0.098419785 = score(doc=4949,freq=1.0), product of:
             0.19433184 = queryWeight, product of:
               3.479186 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.012062738 = queryNorm
             0.5064522 = fieldWeight in 4949, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.109375 = fieldNorm(doc=4949)
         0.29430884 = weight(abstract_txt:relevance in 4949) [ClassicSimilarity], result of:
           0.29430884 = score(doc=4949,freq=2.0), product of:
             0.3857985 = queryWeight, product of:
               6.4849253 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012062738 = queryNorm
             0.7628563 = fieldWeight in 4949, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.109375 = fieldNorm(doc=4949)
         0.51205754 = weight(abstract_txt:judgements in 4949) [ClassicSimilarity], result of:
           0.51205754 = score(doc=4949,freq=1.0), product of:
             0.6285476 = queryWeight, product of:
               6.995677 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.012062738 = queryNorm
             0.8146679 = fieldWeight in 4949, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.109375 = fieldNorm(doc=4949)
       0.28 = coord(7/25)
   

2. Harter, S.P.: Variations in relevance assessments and the measurement of retrieval effectiveness (1996) 0.28

   0.28218576 = sum of:
     0.28218576 = product of:
       1.0078063 = sum of:
         0.048599124 = weight(abstract_txt:conditions in 3004) [ClassicSimilarity], result of:
           0.048599124 = score(doc=3004,freq=2.0), product of:
             0.08815291 = queryWeight, product of:
               1.1716394 = boost
               6.237302 = idf(docFreq=234, maxDocs=44218)
               0.012062738 = queryNorm
             0.5513048 = fieldWeight in 3004, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.237302 = idf(docFreq=234, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.044290595 = weight(abstract_txt:evaluation in 3004) [ClassicSimilarity], result of:
           0.044290595 = score(doc=3004,freq=3.0), product of:
             0.091202125 = queryWeight, product of:
               1.6853619 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.012062738 = queryNorm
             0.48563117 = fieldWeight in 3004, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.10190437 = weight(abstract_txt:affect in 3004) [ClassicSimilarity], result of:
           0.10190437 = score(doc=3004,freq=3.0), product of:
             0.1589495 = queryWeight, product of:
               2.22495 = boost
               5.9223356 = idf(docFreq=321, maxDocs=44218)
               0.012062738 = queryNorm
             0.6411116 = fieldWeight in 3004, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.9223356 = idf(docFreq=321, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.06644997 = weight(abstract_txt:retrieval in 3004) [ClassicSimilarity], result of:
           0.06644997 = score(doc=3004,freq=5.0), product of:
             0.1368225 = queryWeight, product of:
               3.2639196 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.012062738 = queryNorm
             0.4856655 = fieldWeight in 3004, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.09741032 = weight(abstract_txt:performance in 3004) [ClassicSimilarity], result of:
           0.09741032 = score(doc=3004,freq=3.0), product of:
             0.19433184 = queryWeight, product of:
               3.479186 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.012062738 = queryNorm
             0.50125766 = fieldWeight in 3004, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.38324153 = weight(abstract_txt:variations in 3004) [ClassicSimilarity], result of:
           0.38324153 = score(doc=3004,freq=5.0), product of:
             0.4084818 = queryWeight, product of:
               5.044197 = boost
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.012062738 = queryNorm
             0.93820953 = fieldWeight in 3004, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.26591036 = weight(abstract_txt:relevance in 3004) [ClassicSimilarity], result of:
           0.26591036 = score(doc=3004,freq=5.0), product of:
             0.3857985 = queryWeight, product of:
               6.4849253 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012062738 = queryNorm
             0.6892467 = fieldWeight in 3004, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
       0.28 = coord(7/25)
   

3. Ruthven, I.: Relevance behaviour in TREC (2014) 0.27

   0.27374122 = sum of:
     0.27374122 = product of:
       0.9776472 = sum of:
         0.021366404 = weight(abstract_txt:material in 1785) [ClassicSimilarity], result of:
           0.021366404 = score(doc=1785,freq=1.0), product of:
             0.0642168 = queryWeight, product of:
               5.3235674 = idf(docFreq=585, maxDocs=44218)
               0.012062738 = queryNorm
             0.33272296 = fieldWeight in 1785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3235674 = idf(docFreq=585, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.024106242 = weight(abstract_txt:certain in 1785) [ClassicSimilarity], result of:
           0.024106242 = score(doc=1785,freq=1.0), product of:
             0.06959543 = queryWeight, product of:
               1.0410367 = boost
               5.542029 = idf(docFreq=470, maxDocs=44218)
               0.012062738 = queryNorm
             0.3463768 = fieldWeight in 1785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.542029 = idf(docFreq=470, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.015184978 = weight(abstract_txt:used in 1785) [ClassicSimilarity], result of:
           0.015184978 = score(doc=1785,freq=2.0), product of:
             0.05114115 = queryWeight, product of:
               1.2620481 = boost
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.012062738 = queryNorm
             0.2969229 = fieldWeight in 1785, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.025571188 = weight(abstract_txt:evaluation in 1785) [ClassicSimilarity], result of:
           0.025571188 = score(doc=1785,freq=1.0), product of:
             0.091202125 = queryWeight, product of:
               1.6853619 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.012062738 = queryNorm
             0.2803793 = fieldWeight in 1785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.08320457 = weight(abstract_txt:affect in 1785) [ClassicSimilarity], result of:
           0.08320457 = score(doc=1785,freq=2.0), product of:
             0.1589495 = queryWeight, product of:
               2.22495 = boost
               5.9223356 = idf(docFreq=321, maxDocs=44218)
               0.012062738 = queryNorm
             0.52346545 = fieldWeight in 1785, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.9223356 = idf(docFreq=321, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.39440882 = weight(abstract_txt:relevance in 1785) [ClassicSimilarity], result of:
           0.39440882 = score(doc=1785,freq=11.0), product of:
             0.3857985 = queryWeight, product of:
               6.4849253 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012062738 = queryNorm
             1.0223181 = fieldWeight in 1785, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.41380498 = weight(abstract_txt:judgements in 1785) [ClassicSimilarity], result of:
           0.41380498 = score(doc=1785,freq=2.0), product of:
             0.6285476 = queryWeight, product of:
               6.995677 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.012062738 = queryNorm
             0.65835106 = fieldWeight in 1785, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
       0.28 = coord(7/25)
   

4. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.25

   0.24623619 = sum of:
     0.24623619 = product of:
       1.2311809 = sum of:
         0.0466675 = weight(abstract_txt:investigates in 3268) [ClassicSimilarity], result of:
           0.0466675 = score(doc=3268,freq=1.0), product of:
             0.08249797 = queryWeight, product of:
               1.1334368 = boost
               6.033927 = idf(docFreq=287, maxDocs=44218)
               0.012062738 = queryNorm
             0.5656806 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.033927 = idf(docFreq=287, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.016106103 = weight(abstract_txt:used in 3268) [ClassicSimilarity], result of:
           0.016106103 = score(doc=3268,freq=1.0), product of:
             0.05114115 = queryWeight, product of:
               1.2620481 = boost
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.012062738 = queryNorm
             0.3149343 = fieldWeight in 3268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.14883435 = weight(abstract_txt:judgement in 3268) [ClassicSimilarity], result of:
           0.14883435 = score(doc=3268,freq=2.0), product of:
             0.14187057 = queryWeight, product of:
               1.4863526 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.012062738 = queryNorm
             1.0490854 = fieldWeight in 3268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.39886555 = weight(abstract_txt:relevance in 3268) [ClassicSimilarity], result of:
           0.39886555 = score(doc=3268,freq=5.0), product of:
             0.3857985 = queryWeight, product of:
               6.4849253 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012062738 = queryNorm
             1.0338701 = fieldWeight in 3268, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
         0.62070745 = weight(abstract_txt:judgements in 3268) [ClassicSimilarity], result of:
           0.62070745 = score(doc=3268,freq=2.0), product of:
             0.6285476 = queryWeight, product of:
               6.995677 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.012062738 = queryNorm
             0.9875266 = fieldWeight in 3268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.09375 = fieldNorm(doc=3268)
       0.2 = coord(5/25)
   

5. Lesk, M.E.; Salton, G.: Relevance assements and retrieval system evaluation (1969) 0.22

   0.22214484 = sum of:
     0.22214484 = product of:
       0.7933744 = sum of:
         0.07166138 = weight(abstract_txt:precision in 4151) [ClassicSimilarity], result of:
           0.07166138 = score(doc=4151,freq=4.0), product of:
             0.06917306 = queryWeight, product of:
               1.0378729 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.012062738 = queryNorm
             1.0359725 = fieldWeight in 4151, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.080721244 = weight(abstract_txt:recall in 4151) [ClassicSimilarity], result of:
           0.080721244 = score(doc=4151,freq=4.0), product of:
             0.07488682 = queryWeight, product of:
               1.0798872 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.012062738 = queryNorm
             1.0779098 = fieldWeight in 4151, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.016106103 = weight(abstract_txt:used in 4151) [ClassicSimilarity], result of:
           0.016106103 = score(doc=4151,freq=1.0), product of:
             0.05114115 = queryWeight, product of:
               1.2620481 = boost
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.012062738 = queryNorm
             0.3149343 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.03835678 = weight(abstract_txt:evaluation in 4151) [ClassicSimilarity], result of:
           0.03835678 = score(doc=4151,freq=1.0), product of:
             0.091202125 = queryWeight, product of:
               1.6853619 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.012062738 = queryNorm
             0.42056894 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.044575993 = weight(abstract_txt:retrieval in 4151) [ClassicSimilarity], result of:
           0.044575993 = score(doc=4151,freq=1.0), product of:
             0.1368225 = queryWeight, product of:
               3.2639196 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.012062738 = queryNorm
             0.3257943 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.36357483 = weight(abstract_txt:variations in 4151) [ClassicSimilarity], result of:
           0.36357483 = score(doc=4151,freq=2.0), product of:
             0.4084818 = queryWeight, product of:
               5.044197 = boost
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.012062738 = queryNorm
             0.8900637 = fieldWeight in 4151, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.7132807 = idf(docFreq=145, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.17837809 = weight(abstract_txt:relevance in 4151) [ClassicSimilarity], result of:
           0.17837809 = score(doc=4151,freq=1.0), product of:
             0.3857985 = queryWeight, product of:
               6.4849253 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.012062738 = queryNorm
             0.46236074 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
       0.28 = coord(7/25)