Document (#4277)

Author

Janes, J.W.
McKinney, R.

Title

Relevance judgements of actual users and secondary judges : a comparative study

Source

Library quarterly. 62(1992) no.2, S.150-168

Year

1992

Abstract

Examines judgements of relevance of document representations to query statements made by people other than the the originators of the queries. A small group of graduate students in the School of Information and Library Studies and undergraduates of Michigan Univ. judges sets of documents that had been retrieved for and judged by real users for a previous study. The assessment of relevance, by the secondary judges, were analysed by themselves and in comparison with the users' assessments. The judges performed reasonably well but some important differences were identified. Secondary judges use the various fields of document records in different ways than users and have a higher threshold of relevance

Theme

Retrievalstudien

Similar documents (author)

1. Janes, J.W.: ¬An alternative to precision (1991) 5.50

   5.504072 = sum of:
     5.504072 = weight(author_txt:janes in 3054) [ClassicSimilarity], result of:
       5.504072 = fieldWeight in 3054, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.806516 = idf(docFreq=17, maxDocs=44218)
         0.625 = fieldNorm(doc=3054)
   

2. Janes, J.W.: Relevance judgements and the incremental presentation of document representations (1991) 5.50

   5.504072 = sum of:
     5.504072 = weight(author_txt:janes in 3831) [ClassicSimilarity], result of:
       5.504072 = fieldWeight in 3831, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.806516 = idf(docFreq=17, maxDocs=44218)
         0.625 = fieldNorm(doc=3831)
   

3. Janes, J.W.: Towards a search theory of information (1989) 5.50

   5.504072 = sum of:
     5.504072 = weight(author_txt:janes in 4491) [ClassicSimilarity], result of:
       5.504072 = fieldWeight in 4491, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.806516 = idf(docFreq=17, maxDocs=44218)
         0.625 = fieldNorm(doc=4491)
   

4. Janes, J.W.: ¬The binary nature of continous relevance judgements : a study of users' perceptions (1991) 5.50

   5.504072 = sum of:
     5.504072 = weight(author_txt:janes in 4845) [ClassicSimilarity], result of:
       5.504072 = fieldWeight in 4845, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.806516 = idf(docFreq=17, maxDocs=44218)
         0.625 = fieldNorm(doc=4845)
   

5. Janes, J.W.: Other people's judgments : a comparison of users' and others' judgments of document relevance, topicality, and utility (1994) 5.50

   5.504072 = sum of:
     5.504072 = weight(author_txt:janes in 6929) [ClassicSimilarity], result of:
       5.504072 = fieldWeight in 6929, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.806516 = idf(docFreq=17, maxDocs=44218)
         0.625 = fieldNorm(doc=6929)
   

Similar documents (content)

1. Burgin, R.: Variations in relevance judgements and the evaluation of retrieval performance (1992) 0.30

   0.30488542 = sum of:
     0.30488542 = product of:
       1.2703559 = sum of:
         0.046303336 = weight(abstract_txt:representations in 3308) [ClassicSimilarity], result of:
           0.046303336 = score(doc=3308,freq=1.0), product of:
             0.08222757 = queryWeight, product of:
               1.0433949 = boost
               6.006528 = idf(docFreq=295, maxDocs=44218)
               0.013120345 = queryNorm
             0.563112 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.006528 = idf(docFreq=295, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.021124797 = weight(abstract_txt:were in 3308) [ClassicSimilarity], result of:
           0.021124797 = score(doc=3308,freq=1.0), product of:
             0.061397076 = queryWeight, product of:
               1.2750547 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.013120345 = queryNorm
             0.34406847 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.033801463 = weight(abstract_txt:document in 3308) [ClassicSimilarity], result of:
           0.033801463 = score(doc=3308,freq=1.0), product of:
             0.08399297 = queryWeight, product of:
               1.4913393 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.013120345 = queryNorm
             0.40243202 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.35317448 = weight(abstract_txt:judgements in 3308) [ClassicSimilarity], result of:
           0.35317448 = score(doc=3308,freq=4.0), product of:
             0.2528864 = queryWeight, product of:
               2.5877225 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.013120345 = queryNorm
             1.3965735 = fieldWeight in 3308, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.25113454 = weight(abstract_txt:relevance in 3308) [ClassicSimilarity], result of:
           0.25113454 = score(doc=3308,freq=6.0), product of:
             0.22174296 = queryWeight, product of:
               3.4268484 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013120345 = queryNorm
             1.132548 = fieldWeight in 3308, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
         0.56481725 = weight(abstract_txt:judges in 3308) [ClassicSimilarity], result of:
           0.56481725 = score(doc=3308,freq=1.0), product of:
             0.7450829 = queryWeight, product of:
               7.0230746 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.013120345 = queryNorm
             0.7580596 = fieldWeight in 3308, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.09375 = fieldNorm(doc=3308)
       0.24 = coord(6/25)
   

2. Park, H.: Relevance of science information : origins and dimensions of relevance and their implications to information retrieval (1997) 0.23

   0.22982918 = sum of:
     0.22982918 = product of:
       0.8208185 = sum of:
         0.053384036 = weight(abstract_txt:graduate in 154) [ClassicSimilarity], result of:
           0.053384036 = score(doc=154,freq=1.0), product of:
             0.102094986 = queryWeight, product of:
               1.1626318 = boost
               6.6929407 = idf(docFreq=148, maxDocs=44218)
               0.013120345 = queryNorm
             0.522886 = fieldWeight in 154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6929407 = idf(docFreq=148, maxDocs=44218)
               0.078125 = fieldNorm(doc=154)
         0.0142928185 = weight(abstract_txt:study in 154) [ClassicSimilarity], result of:
           0.0142928185 = score(doc=154,freq=1.0), product of:
             0.053434126 = queryWeight, product of:
               1.1894997 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.013120345 = queryNorm
             0.26748484 = fieldWeight in 154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.078125 = fieldNorm(doc=154)
         0.030491019 = weight(abstract_txt:were in 154) [ClassicSimilarity], result of:
           0.030491019 = score(doc=154,freq=3.0), product of:
             0.061397076 = queryWeight, product of:
               1.2750547 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.013120345 = queryNorm
             0.49662006 = fieldWeight in 154, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.078125 = fieldNorm(doc=154)
         0.032399934 = weight(abstract_txt:users in 154) [ClassicSimilarity], result of:
           0.032399934 = score(doc=154,freq=1.0), product of:
             0.116175056 = queryWeight, product of:
               2.4804268 = boost
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.013120345 = queryNorm
             0.2788889 = fieldWeight in 154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.078125 = fieldNorm(doc=154)
         0.2548817 = weight(abstract_txt:judgements in 154) [ClassicSimilarity], result of:
           0.2548817 = score(doc=154,freq=3.0), product of:
             0.2528864 = queryWeight, product of:
               2.5877225 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.013120345 = queryNorm
             1.0078901 = fieldWeight in 154, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.078125 = fieldNorm(doc=154)
         0.27017772 = weight(abstract_txt:relevance in 154) [ClassicSimilarity], result of:
           0.27017772 = score(doc=154,freq=10.0), product of:
             0.22174296 = queryWeight, product of:
               3.4268484 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013120345 = queryNorm
             1.2184275 = fieldWeight in 154, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=154)
         0.16519122 = weight(abstract_txt:secondary in 154) [ClassicSimilarity], result of:
           0.16519122 = score(doc=154,freq=1.0), product of:
             0.31267646 = queryWeight, product of:
               3.5240998 = boost
               6.7624135 = idf(docFreq=138, maxDocs=44218)
               0.013120345 = queryNorm
             0.5283136 = fieldWeight in 154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.7624135 = idf(docFreq=138, maxDocs=44218)
               0.078125 = fieldNorm(doc=154)
       0.28 = coord(7/25)
   

3. Janes, J.W.: Other people's judgments : a comparison of users' and others' judgments of document relevance, topicality, and utility (1994) 0.22

   0.22364238 = sum of:
     0.22364238 = product of:
       0.7987228 = sum of:
         0.014083199 = weight(abstract_txt:were in 6929) [ClassicSimilarity], result of:
           0.014083199 = score(doc=6929,freq=1.0), product of:
             0.061397076 = queryWeight, product of:
               1.2750547 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.013120345 = queryNorm
             0.22937898 = fieldWeight in 6929, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.0625 = fieldNorm(doc=6929)
         0.073847614 = weight(abstract_txt:reasonably in 6929) [ClassicSimilarity], result of:
           0.073847614 = score(doc=6929,freq=1.0), product of:
             0.14708254 = queryWeight, product of:
               1.3954701 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.013120345 = queryNorm
             0.5020828 = fieldWeight in 6929, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.0625 = fieldNorm(doc=6929)
         0.031868324 = weight(abstract_txt:document in 6929) [ClassicSimilarity], result of:
           0.031868324 = score(doc=6929,freq=2.0), product of:
             0.08399297 = queryWeight, product of:
               1.4913393 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.013120345 = queryNorm
             0.37941656 = fieldWeight in 6929, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=6929)
         0.051839896 = weight(abstract_txt:users in 6929) [ClassicSimilarity], result of:
           0.051839896 = score(doc=6929,freq=4.0), product of:
             0.116175056 = queryWeight, product of:
               2.4804268 = boost
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.013120345 = queryNorm
             0.44622225 = fieldWeight in 6929, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.0625 = fieldNorm(doc=6929)
         0.118385956 = weight(abstract_txt:relevance in 6929) [ClassicSimilarity], result of:
           0.118385956 = score(doc=6929,freq=3.0), product of:
             0.22174296 = queryWeight, product of:
               3.4268484 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013120345 = queryNorm
             0.5338882 = fieldWeight in 6929, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=6929)
         0.13215297 = weight(abstract_txt:secondary in 6929) [ClassicSimilarity], result of:
           0.13215297 = score(doc=6929,freq=1.0), product of:
             0.31267646 = queryWeight, product of:
               3.5240998 = boost
               6.7624135 = idf(docFreq=138, maxDocs=44218)
               0.013120345 = queryNorm
             0.42265084 = fieldWeight in 6929, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.7624135 = idf(docFreq=138, maxDocs=44218)
               0.0625 = fieldNorm(doc=6929)
         0.37654483 = weight(abstract_txt:judges in 6929) [ClassicSimilarity], result of:
           0.37654483 = score(doc=6929,freq=1.0), product of:
             0.7450829 = queryWeight, product of:
               7.0230746 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.013120345 = queryNorm
             0.50537306 = fieldWeight in 6929, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.0625 = fieldNorm(doc=6929)
       0.28 = coord(7/25)
   

4. Janes, J.W.: ¬The binary nature of continous relevance judgements : a study of users' perceptions (1991) 0.21

   0.21064313 = sum of:
     0.21064313 = product of:
       0.6582598 = sum of:
         0.024255717 = weight(abstract_txt:study in 4845) [ClassicSimilarity], result of:
           0.024255717 = score(doc=4845,freq=2.0), product of:
             0.053434126 = queryWeight, product of:
               1.1894997 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.013120345 = queryNorm
             0.45393682 = fieldWeight in 4845, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
         0.083629586 = weight(abstract_txt:univ in 4845) [ClassicSimilarity], result of:
           0.083629586 = score(doc=4845,freq=1.0), product of:
             0.12195023 = queryWeight, product of:
               1.2706656 = boost
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.013120345 = queryNorm
             0.6857682 = fieldWeight in 4845, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
         0.021124797 = weight(abstract_txt:were in 4845) [ClassicSimilarity], result of:
           0.021124797 = score(doc=4845,freq=1.0), product of:
             0.061397076 = queryWeight, product of:
               1.2750547 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.013120345 = queryNorm
             0.34406847 = fieldWeight in 4845, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
         0.102402136 = weight(abstract_txt:michigan in 4845) [ClassicSimilarity], result of:
           0.102402136 = score(doc=4845,freq=1.0), product of:
             0.13957746 = queryWeight, product of:
               1.3594011 = boost
               7.825686 = idf(docFreq=47, maxDocs=44218)
               0.013120345 = queryNorm
             0.7336581 = fieldWeight in 4845, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.825686 = idf(docFreq=47, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
         0.033801463 = weight(abstract_txt:document in 4845) [ClassicSimilarity], result of:
           0.033801463 = score(doc=4845,freq=1.0), product of:
             0.08399297 = queryWeight, product of:
               1.4913393 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.013120345 = queryNorm
             0.40243202 = fieldWeight in 4845, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
         0.03887992 = weight(abstract_txt:users in 4845) [ClassicSimilarity], result of:
           0.03887992 = score(doc=4845,freq=1.0), product of:
             0.116175056 = queryWeight, product of:
               2.4804268 = boost
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.013120345 = queryNorm
             0.33466667 = fieldWeight in 4845, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
         0.17658724 = weight(abstract_txt:judgements in 4845) [ClassicSimilarity], result of:
           0.17658724 = score(doc=4845,freq=1.0), product of:
             0.2528864 = queryWeight, product of:
               2.5877225 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.013120345 = queryNorm
             0.6982868 = fieldWeight in 4845, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
         0.17757893 = weight(abstract_txt:relevance in 4845) [ClassicSimilarity], result of:
           0.17757893 = score(doc=4845,freq=3.0), product of:
             0.22174296 = queryWeight, product of:
               3.4268484 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013120345 = queryNorm
             0.80083233 = fieldWeight in 4845, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=4845)
       0.32 = coord(8/25)
   

5. Spink, A.; Saracevic, T.: Sources and use of search terms in online searching (1992) 0.20

   0.19891162 = sum of:
     0.19891162 = product of:
       0.9945581 = sum of:
         0.017151382 = weight(abstract_txt:study in 4523) [ClassicSimilarity], result of:
           0.017151382 = score(doc=4523,freq=1.0), product of:
             0.053434126 = queryWeight, product of:
               1.1894997 = boost
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.013120345 = queryNorm
             0.3209818 = fieldWeight in 4523, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.423806 = idf(docFreq=3916, maxDocs=44218)
               0.09375 = fieldNorm(doc=4523)
         0.021124797 = weight(abstract_txt:were in 4523) [ClassicSimilarity], result of:
           0.021124797 = score(doc=4523,freq=1.0), product of:
             0.061397076 = queryWeight, product of:
               1.2750547 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.013120345 = queryNorm
             0.34406847 = fieldWeight in 4523, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.09375 = fieldNorm(doc=4523)
         0.054984514 = weight(abstract_txt:users in 4523) [ClassicSimilarity], result of:
           0.054984514 = score(doc=4523,freq=2.0), product of:
             0.116175056 = queryWeight, product of:
               2.4804268 = boost
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.013120345 = queryNorm
             0.47329018 = fieldWeight in 4523, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.09375 = fieldNorm(doc=4523)
         0.10252524 = weight(abstract_txt:relevance in 4523) [ClassicSimilarity], result of:
           0.10252524 = score(doc=4523,freq=1.0), product of:
             0.22174296 = queryWeight, product of:
               3.4268484 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.013120345 = queryNorm
             0.46236074 = fieldWeight in 4523, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=4523)
         0.79877216 = weight(abstract_txt:judges in 4523) [ClassicSimilarity], result of:
           0.79877216 = score(doc=4523,freq=2.0), product of:
             0.7450829 = queryWeight, product of:
               7.0230746 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.013120345 = queryNorm
             1.0720581 = fieldWeight in 4523, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.09375 = fieldNorm(doc=4523)
       0.2 = coord(5/25)