Document (#3309)

Author

Burgin, R.

Title

Variations in relevance judgements and the evaluation of retrieval performance

Source

Information processing and management. 28(1992) no.5, S.619-627

Year

1992

Abstract

Relevance judgements used to evaluate the performance of information retrieval systems are known to vary among judges and to vary under certain conditions extraneous to the relevance relationship between queries and documents. Investigates the degree to which variations in relevance judgements affect the evaluation of retrieval performance. Four sets of relevance judgements were used to test the retrieval effectiveness of 6 document representations. In no case was there a noticeable or material difference in retrieval performance due to variations in relevance judgement. Detailed examination of reasons why variations in relevance judgements may not affect recall and precision

Similar documents (author)

1. Burgin, R.: ¬The retrieval effectiveness of 5 clustering algorithms as a function of indexing exhaustivity (1995) 5.99

   5.989656 = sum of:
     5.989656 = weight(author_txt:burgin in 3434) [ClassicSimilarity], result of:
       5.989656 = fieldWeight in 3434, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.583449 = idf(docFreq=7, maxDocs=42740)
         0.625 = fieldNorm(doc=3434)
   

2. Burgin, M.: Terminological and teleological systems (1996) 5.99

   5.989656 = sum of:
     5.989656 = weight(author_txt:burgin in 6392) [ClassicSimilarity], result of:
       5.989656 = fieldWeight in 6392, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.583449 = idf(docFreq=7, maxDocs=42740)
         0.625 = fieldNorm(doc=6392)
   

3. Burgin, R.: ¬The Monte Carlo method and the evaluation of retrieval system performance (1999) 5.99

   5.989656 = sum of:
     5.989656 = weight(author_txt:burgin in 3947) [ClassicSimilarity], result of:
       5.989656 = fieldWeight in 3947, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.583449 = idf(docFreq=7, maxDocs=42740)
         0.625 = fieldNorm(doc=3947)
   

4. Burgin, R.: ¬The effect of indexing exhaustivity on retrieval performance (1991) 5.99

   5.989656 = sum of:
     5.989656 = weight(author_txt:burgin in 178) [ClassicSimilarity], result of:
       5.989656 = fieldWeight in 178, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.583449 = idf(docFreq=7, maxDocs=42740)
         0.625 = fieldNorm(doc=178)
   

5. Shaw, W.M.; Burgin, R.; Howell, P.: Performance standards and evaluations in IR test collections : cluster-based retrieval models (1997) 3.59

   3.5937934 = sum of:
     3.5937934 = weight(author_txt:burgin in 326) [ClassicSimilarity], result of:
       3.5937934 = fieldWeight in 326, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.583449 = idf(docFreq=7, maxDocs=42740)
         0.375 = fieldNorm(doc=326)
   

Similar documents (content)

1. Wallis, P.; Thom, J.A.: Relevance judgements for assessing recall (1996) 0.30

   0.3012233 = sum of:
     0.3012233 = product of:
       1.0757976 = sum of:
         0.041549798 = weight(abstract_txt:precision in 5018) [ClassicSimilarity], result of:
           0.041549798 = score(doc=5018,freq=1.0), product of:
             0.06886244 = queryWeight, product of:
               1.0371406 = boost
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.012035837 = queryNorm
             0.6033739 = fieldWeight in 5018, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.109375 = fieldNorm(doc=5018)
         0.065961845 = weight(abstract_txt:recall in 5018) [ClassicSimilarity], result of:
           0.065961845 = score(doc=5018,freq=2.0), product of:
             0.074379794 = queryWeight, product of:
               1.0778887 = boost
               5.733301 = idf(docFreq=375, maxDocs=42740)
               0.012035837 = queryNorm
             0.8868248 = fieldWeight in 5018, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.733301 = idf(docFreq=375, maxDocs=42740)
               0.109375 = fieldNorm(doc=5018)
         0.019026564 = weight(abstract_txt:used in 5018) [ClassicSimilarity], result of:
           0.019026564 = score(doc=5018,freq=1.0), product of:
             0.05154502 = queryWeight, product of:
               1.26898 = boost
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.012035837 = queryNorm
             0.3691252 = fieldWeight in 5018, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.109375 = fieldNorm(doc=5018)
         0.051471706 = weight(abstract_txt:retrieval in 5018) [ClassicSimilarity], result of:
           0.051471706 = score(doc=5018,freq=1.0), product of:
             0.13582264 = queryWeight, product of:
               3.2569983 = boost
               3.4648013 = idf(docFreq=3633, maxDocs=42740)
               0.012035837 = queryNorm
             0.37896264 = fieldWeight in 5018, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4648013 = idf(docFreq=3633, maxDocs=42740)
               0.109375 = fieldNorm(doc=5018)
         0.09906844 = weight(abstract_txt:performance in 5018) [ClassicSimilarity], result of:
           0.09906844 = score(doc=5018,freq=1.0), product of:
             0.19509524 = queryWeight, product of:
               3.4914012 = boost
               4.6426997 = idf(docFreq=1118, maxDocs=42740)
               0.012035837 = queryNorm
             0.5077953 = fieldWeight in 5018, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6426997 = idf(docFreq=1118, maxDocs=42740)
               0.109375 = fieldNorm(doc=5018)
         0.29433584 = weight(abstract_txt:relevance in 5018) [ClassicSimilarity], result of:
           0.29433584 = score(doc=5018,freq=2.0), product of:
             0.38564518 = queryWeight, product of:
               6.493658 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.012035837 = queryNorm
             0.76322967 = fieldWeight in 5018, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.109375 = fieldNorm(doc=5018)
         0.5043834 = weight(abstract_txt:judgements in 5018) [ClassicSimilarity], result of:
           0.5043834 = score(doc=5018,freq=1.0), product of:
             0.6219664 = queryWeight, product of:
               6.969713 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.012035837 = queryNorm
             0.8109495 = fieldWeight in 5018, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.109375 = fieldNorm(doc=5018)
       0.28 = coord(7/25)
   

2. Harter, S.P.: Variations in relevance assessments and the measurement of retrieval effectiveness (1996) 0.29

   0.2857677 = sum of:
     0.2857677 = product of:
       1.0205989 = sum of:
         0.050048687 = weight(abstract_txt:conditions in 3073) [ClassicSimilarity], result of:
           0.050048687 = score(doc=3073,freq=2.0), product of:
             0.08985597 = queryWeight, product of:
               1.1847314 = boost
               6.3015985 = idf(docFreq=212, maxDocs=42740)
               0.012035837 = queryNorm
             0.5569879 = fieldWeight in 3073, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.3015985 = idf(docFreq=212, maxDocs=42740)
               0.0625 = fieldNorm(doc=3073)
         0.044428233 = weight(abstract_txt:evaluation in 3073) [ClassicSimilarity], result of:
           0.044428233 = score(doc=3073,freq=3.0), product of:
             0.091349065 = queryWeight, product of:
               1.689326 = boost
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.012035837 = queryNorm
             0.48635674 = fieldWeight in 3073, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.0625 = fieldNorm(doc=3073)
         0.10472963 = weight(abstract_txt:affect in 3073) [ClassicSimilarity], result of:
           0.10472963 = score(doc=3073,freq=3.0), product of:
             0.16179973 = queryWeight, product of:
               2.2482796 = boost
               5.979311 = idf(docFreq=293, maxDocs=42740)
               0.012035837 = queryNorm
             0.6472794 = fieldWeight in 3073, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.979311 = idf(docFreq=293, maxDocs=42740)
               0.0625 = fieldNorm(doc=3073)
         0.06576814 = weight(abstract_txt:retrieval in 3073) [ClassicSimilarity], result of:
           0.06576814 = score(doc=3073,freq=5.0), product of:
             0.13582264 = queryWeight, product of:
               3.2569983 = boost
               3.4648013 = idf(docFreq=3633, maxDocs=42740)
               0.012035837 = queryNorm
             0.4842207 = fieldWeight in 3073, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4648013 = idf(docFreq=3633, maxDocs=42740)
               0.0625 = fieldNorm(doc=3073)
         0.09805232 = weight(abstract_txt:performance in 3073) [ClassicSimilarity], result of:
           0.09805232 = score(doc=3073,freq=3.0), product of:
             0.19509524 = queryWeight, product of:
               3.4914012 = boost
               4.6426997 = idf(docFreq=1118, maxDocs=42740)
               0.012035837 = queryNorm
             0.50258696 = fieldWeight in 3073, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.6426997 = idf(docFreq=1118, maxDocs=42740)
               0.0625 = fieldNorm(doc=3073)
         0.39163703 = weight(abstract_txt:variations in 3073) [ClassicSimilarity], result of:
           0.39163703 = score(doc=3073,freq=5.0), product of:
             0.4142358 = queryWeight, product of:
               5.087451 = boost
               6.765051 = idf(docFreq=133, maxDocs=42740)
               0.012035837 = queryNorm
             0.94544464 = fieldWeight in 3073, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.765051 = idf(docFreq=133, maxDocs=42740)
               0.0625 = fieldNorm(doc=3073)
         0.26593477 = weight(abstract_txt:relevance in 3073) [ClassicSimilarity], result of:
           0.26593477 = score(doc=3073,freq=5.0), product of:
             0.38564518 = queryWeight, product of:
               6.493658 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.012035837 = queryNorm
             0.6895841 = fieldWeight in 3073, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0625 = fieldNorm(doc=3073)
       0.28 = coord(7/25)
   

3. Ruthven, I.: Relevance behaviour in TREC (2014) 0.27

   0.27273795 = sum of:
     0.27273795 = product of:
       0.9740641 = sum of:
         0.02128227 = weight(abstract_txt:material in 3786) [ClassicSimilarity], result of:
           0.02128227 = score(doc=3786,freq=1.0), product of:
             0.06401874 = queryWeight, product of:
               5.3190103 = idf(docFreq=568, maxDocs=42740)
               0.012035837 = queryNorm
             0.33243814 = fieldWeight in 3786, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3190103 = idf(docFreq=568, maxDocs=42740)
               0.0625 = fieldNorm(doc=3786)
         0.024195716 = weight(abstract_txt:certain in 3786) [ClassicSimilarity], result of:
           0.024195716 = score(doc=3786,freq=1.0), product of:
             0.069735534 = queryWeight, product of:
               1.0436947 = boost
               5.551423 = idf(docFreq=450, maxDocs=42740)
               0.012035837 = queryNorm
             0.34696394 = fieldWeight in 3786, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.551423 = idf(docFreq=450, maxDocs=42740)
               0.0625 = fieldNorm(doc=3786)
         0.0153757855 = weight(abstract_txt:used in 3786) [ClassicSimilarity], result of:
           0.0153757855 = score(doc=3786,freq=2.0), product of:
             0.05154502 = queryWeight, product of:
               1.26898 = boost
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.012035837 = queryNorm
             0.29829818 = fieldWeight in 3786, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.0625 = fieldNorm(doc=3786)
         0.025650652 = weight(abstract_txt:evaluation in 3786) [ClassicSimilarity], result of:
           0.025650652 = score(doc=3786,freq=1.0), product of:
             0.091349065 = queryWeight, product of:
               1.689326 = boost
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.012035837 = queryNorm
             0.2807982 = fieldWeight in 3786, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.0625 = fieldNorm(doc=3786)
         0.08551138 = weight(abstract_txt:affect in 3786) [ClassicSimilarity], result of:
           0.08551138 = score(doc=3786,freq=2.0), product of:
             0.16179973 = queryWeight, product of:
               2.2482796 = boost
               5.979311 = idf(docFreq=293, maxDocs=42740)
               0.012035837 = queryNorm
             0.5285014 = fieldWeight in 3786, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.979311 = idf(docFreq=293, maxDocs=42740)
               0.0625 = fieldNorm(doc=3786)
         0.394445 = weight(abstract_txt:relevance in 3786) [ClassicSimilarity], result of:
           0.394445 = score(doc=3786,freq=11.0), product of:
             0.38564518 = queryWeight, product of:
               6.493658 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.012035837 = queryNorm
             1.0228184 = fieldWeight in 3786, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0625 = fieldNorm(doc=3786)
         0.4076033 = weight(abstract_txt:judgements in 3786) [ClassicSimilarity], result of:
           0.4076033 = score(doc=3786,freq=2.0), product of:
             0.6219664 = queryWeight, product of:
               6.969713 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.012035837 = queryNorm
             0.65534616 = fieldWeight in 3786, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.0625 = fieldNorm(doc=3786)
       0.28 = coord(7/25)
   

4. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.24

   0.2443265 = sum of:
     0.2443265 = product of:
       1.2216325 = sum of:
         0.04829481 = weight(abstract_txt:investigates in 4269) [ClassicSimilarity], result of:
           0.04829481 = score(doc=4269,freq=1.0), product of:
             0.08436609 = queryWeight, product of:
               1.1479696 = boost
               6.1060624 = idf(docFreq=258, maxDocs=42740)
               0.012035837 = queryNorm
             0.57244337 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.1060624 = idf(docFreq=258, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.016308485 = weight(abstract_txt:used in 4269) [ClassicSimilarity], result of:
           0.016308485 = score(doc=4269,freq=1.0), product of:
             0.05154502 = queryWeight, product of:
               1.26898 = boost
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.012035837 = queryNorm
             0.31639302 = fieldWeight in 4269, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.14672211 = weight(abstract_txt:judgement in 4269) [ClassicSimilarity], result of:
           0.14672211 = score(doc=4269,freq=2.0), product of:
             0.14046066 = queryWeight, product of:
               1.4812343 = boost
               7.8787007 = idf(docFreq=43, maxDocs=42740)
               0.012035837 = queryNorm
             1.044578 = fieldWeight in 4269, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.8787007 = idf(docFreq=43, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.39890218 = weight(abstract_txt:relevance in 4269) [ClassicSimilarity], result of:
           0.39890218 = score(doc=4269,freq=5.0), product of:
             0.38564518 = queryWeight, product of:
               6.493658 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.012035837 = queryNorm
             1.0343761 = fieldWeight in 4269, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
         0.61140496 = weight(abstract_txt:judgements in 4269) [ClassicSimilarity], result of:
           0.61140496 = score(doc=4269,freq=2.0), product of:
             0.6219664 = queryWeight, product of:
               6.969713 = boost
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.012035837 = queryNorm
             0.98301923 = fieldWeight in 4269, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4143953 = idf(docFreq=69, maxDocs=42740)
               0.09375 = fieldNorm(doc=4269)
       0.2 = coord(5/25)
   

5. Lesk, M.E.; Salton, G.: Relevance assements and retrieval system evaluation (1969) 0.22

   0.22400646 = sum of:
     0.22400646 = product of:
       0.8000231 = sum of:
         0.07122823 = weight(abstract_txt:precision in 4220) [ClassicSimilarity], result of:
           0.07122823 = score(doc=4220,freq=4.0), product of:
             0.06886244 = queryWeight, product of:
               1.0371406 = boost
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.012035837 = queryNorm
             1.0343553 = fieldWeight in 4220, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.09375 = fieldNorm(doc=4220)
         0.07995783 = weight(abstract_txt:recall in 4220) [ClassicSimilarity], result of:
           0.07995783 = score(doc=4220,freq=4.0), product of:
             0.074379794 = queryWeight, product of:
               1.0778887 = boost
               5.733301 = idf(docFreq=375, maxDocs=42740)
               0.012035837 = queryNorm
             1.074994 = fieldWeight in 4220, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.733301 = idf(docFreq=375, maxDocs=42740)
               0.09375 = fieldNorm(doc=4220)
         0.016308485 = weight(abstract_txt:used in 4220) [ClassicSimilarity], result of:
           0.016308485 = score(doc=4220,freq=1.0), product of:
             0.05154502 = queryWeight, product of:
               1.26898 = boost
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.012035837 = queryNorm
             0.31639302 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3748589 = idf(docFreq=3975, maxDocs=42740)
               0.09375 = fieldNorm(doc=4220)
         0.03847598 = weight(abstract_txt:evaluation in 4220) [ClassicSimilarity], result of:
           0.03847598 = score(doc=4220,freq=1.0), product of:
             0.091349065 = queryWeight, product of:
               1.689326 = boost
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.012035837 = queryNorm
             0.4211973 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.492771 = idf(docFreq=1299, maxDocs=42740)
               0.09375 = fieldNorm(doc=4220)
         0.0441186 = weight(abstract_txt:retrieval in 4220) [ClassicSimilarity], result of:
           0.0441186 = score(doc=4220,freq=1.0), product of:
             0.13582264 = queryWeight, product of:
               3.2569983 = boost
               3.4648013 = idf(docFreq=3633, maxDocs=42740)
               0.012035837 = queryNorm
             0.3248251 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4648013 = idf(docFreq=3633, maxDocs=42740)
               0.09375 = fieldNorm(doc=4220)
         0.37153947 = weight(abstract_txt:variations in 4220) [ClassicSimilarity], result of:
           0.37153947 = score(doc=4220,freq=2.0), product of:
             0.4142358 = queryWeight, product of:
               5.087451 = boost
               6.765051 = idf(docFreq=133, maxDocs=42740)
               0.012035837 = queryNorm
             0.8969275 = fieldWeight in 4220, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.765051 = idf(docFreq=133, maxDocs=42740)
               0.09375 = fieldNorm(doc=4220)
         0.17839447 = weight(abstract_txt:relevance in 4220) [ClassicSimilarity], result of:
           0.17839447 = score(doc=4220,freq=1.0), product of:
             0.38564518 = queryWeight, product of:
               6.493658 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.012035837 = queryNorm
             0.46258706 = fieldWeight in 4220, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.09375 = fieldNorm(doc=4220)
       0.28 = coord(7/25)