Document (#10815)

Author

Yao, Y.Y.

Title

Measuring retrieval effectiveness based on user preference of documents

Source

Journal of the American Society for Information Science. 46(1995) no.2, S.133-145

Year

1995

Abstract

The notion of user preference is adopted for the representation, interpretation, and measurement of the relevance or usefulness of documents. Use judgements on documents may be formally describes by a weak order (i.e. user ranking) and measured using an ordinal scale. Within this framework, a new measure of system performance is suggested based on the distance between user ranking and system ranking. It only uses the relative order of documents and therefore confirms to the valid use of an ordinal scale measuring relevance. It is also applicable to multilevel relevance judgements and ranked system output. The appropriateness of the proposed measure is demonstrated through an axiomatic approach. The inherent relationships between the new measure and many existing measures provide further supporting evidence

Similar documents (content)

1. Bartell, B.T.; Cottrell, G.W.; Belew, R.K.: Optimizing similarity using multi-query relevance feedback (1998) 0.23

   0.2299914 = sum of:
     0.2299914 = product of:
       0.638865 = sum of:
         0.045581542 = weight(abstract_txt:ranked in 2150) [ClassicSimilarity], result of:
           0.045581542 = score(doc=2150,freq=1.0), product of:
             0.113706104 = queryWeight, product of:
               6.4139447 = idf(docFreq=193, maxDocs=43556)
               0.017727952 = queryNorm
             0.40087155 = fieldWeight in 2150, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.4139447 = idf(docFreq=193, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.0144632505 = weight(abstract_txt:between in 2150) [ClassicSimilarity], result of:
           0.0144632505 = score(doc=2150,freq=1.0), product of:
             0.06664666 = queryWeight, product of:
               1.0827104 = boost
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.017727952 = queryNorm
             0.2170139 = fieldWeight in 2150, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.019768259 = weight(abstract_txt:system in 2150) [ClassicSimilarity], result of:
           0.019768259 = score(doc=2150,freq=1.0), product of:
             0.09396017 = queryWeight, product of:
               1.5744927 = boost
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.017727952 = queryNorm
             0.21038976 = fieldWeight in 2150, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.12091213 = weight(abstract_txt:preference in 2150) [ClassicSimilarity], result of:
           0.12091213 = score(doc=2150,freq=1.0), product of:
             0.27452445 = queryWeight, product of:
               2.1974227 = boost
               7.047074 = idf(docFreq=102, maxDocs=43556)
               0.017727952 = queryNorm
             0.44044212 = fieldWeight in 2150, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.047074 = idf(docFreq=102, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.034458306 = weight(abstract_txt:user in 2150) [ClassicSimilarity], result of:
           0.034458306 = score(doc=2150,freq=1.0), product of:
             0.14978686 = queryWeight, product of:
               2.2954876 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.017727952 = queryNorm
             0.23004892 = fieldWeight in 2150, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.062257133 = weight(abstract_txt:relevance in 2150) [ClassicSimilarity], result of:
           0.062257133 = score(doc=2150,freq=1.0), product of:
             0.2018789 = queryWeight, product of:
               2.3078852 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.017727952 = queryNorm
             0.3083885 = fieldWeight in 2150, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.16646777 = weight(abstract_txt:measure in 2150) [ClassicSimilarity], result of:
           0.16646777 = score(doc=2150,freq=4.0), product of:
             0.24499938 = queryWeight, product of:
               2.542443 = boost
               5.435696 = idf(docFreq=515, maxDocs=43556)
               0.017727952 = queryNorm
             0.679462 = fieldWeight in 2150, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.435696 = idf(docFreq=515, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.083741575 = weight(abstract_txt:documents in 2150) [ClassicSimilarity], result of:
           0.083741575 = score(doc=2150,freq=3.0), product of:
             0.18772903 = queryWeight, product of:
               2.5698266 = boost
               4.1206813 = idf(docFreq=1921, maxDocs=43556)
               0.017727952 = queryNorm
             0.44607684 = fieldWeight in 2150, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1206813 = idf(docFreq=1921, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
         0.09121501 = weight(abstract_txt:ranking in 2150) [ClassicSimilarity], result of:
           0.09121501 = score(doc=2150,freq=1.0), product of:
             0.26042083 = queryWeight, product of:
               2.6212387 = boost
               5.6041603 = idf(docFreq=435, maxDocs=43556)
               0.017727952 = queryNorm
             0.35026002 = fieldWeight in 2150, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6041603 = idf(docFreq=435, maxDocs=43556)
               0.0625 = fieldNorm(doc=2150)
       0.36 = coord(9/25)
   

2. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.18

   0.184288 = sum of:
     0.184288 = product of:
       0.7678667 = sum of:
         0.020454125 = weight(abstract_txt:between in 591) [ClassicSimilarity], result of:
           0.020454125 = score(doc=591,freq=2.0), product of:
             0.06664666 = queryWeight, product of:
               1.0827104 = boost
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.017727952 = queryNorm
             0.306904 = fieldWeight in 591, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.0625 = fieldNorm(doc=591)
         0.09954631 = weight(abstract_txt:scale in 591) [ClassicSimilarity], result of:
           0.09954631 = score(doc=591,freq=3.0), product of:
             0.16720304 = queryWeight, product of:
               1.7149252 = boost
               5.4997177 = idf(docFreq=483, maxDocs=43556)
               0.017727952 = queryNorm
             0.5953619 = fieldWeight in 591, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4997177 = idf(docFreq=483, maxDocs=43556)
               0.0625 = fieldNorm(doc=591)
         0.07705111 = weight(abstract_txt:user in 591) [ClassicSimilarity], result of:
           0.07705111 = score(doc=591,freq=5.0), product of:
             0.14978686 = queryWeight, product of:
               2.2954876 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.017727952 = queryNorm
             0.514405 = fieldWeight in 591, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.0625 = fieldNorm(doc=591)
         0.12451427 = weight(abstract_txt:relevance in 591) [ClassicSimilarity], result of:
           0.12451427 = score(doc=591,freq=4.0), product of:
             0.2018789 = queryWeight, product of:
               2.3078852 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.017727952 = queryNorm
             0.616777 = fieldWeight in 591, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.0625 = fieldNorm(doc=591)
         0.3173034 = weight(abstract_txt:judgements in 591) [ClassicSimilarity], result of:
           0.3173034 = score(doc=591,freq=5.0), product of:
             0.30544117 = queryWeight, product of:
               2.3178585 = boost
               7.4333076 = idf(docFreq=69, maxDocs=43556)
               0.017727952 = queryNorm
             1.0388364 = fieldWeight in 591, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.4333076 = idf(docFreq=69, maxDocs=43556)
               0.0625 = fieldNorm(doc=591)
         0.12899749 = weight(abstract_txt:ranking in 591) [ClassicSimilarity], result of:
           0.12899749 = score(doc=591,freq=2.0), product of:
             0.26042083 = queryWeight, product of:
               2.6212387 = boost
               5.6041603 = idf(docFreq=435, maxDocs=43556)
               0.017727952 = queryNorm
             0.49534246 = fieldWeight in 591, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.6041603 = idf(docFreq=435, maxDocs=43556)
               0.0625 = fieldNorm(doc=591)
       0.24 = coord(6/25)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings (2016) 0.18

   0.18079992 = sum of:
     0.18079992 = product of:
       0.753333 = sum of:
         0.012655345 = weight(abstract_txt:between in 69) [ClassicSimilarity], result of:
           0.012655345 = score(doc=69,freq=1.0), product of:
             0.06664666 = queryWeight, product of:
               1.0827104 = boost
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.017727952 = queryNorm
             0.18988717 = fieldWeight in 69, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.0546875 = fieldNorm(doc=69)
         0.08710303 = weight(abstract_txt:scale in 69) [ClassicSimilarity], result of:
           0.08710303 = score(doc=69,freq=3.0), product of:
             0.16720304 = queryWeight, product of:
               1.7149252 = boost
               5.4997177 = idf(docFreq=483, maxDocs=43556)
               0.017727952 = queryNorm
             0.5209417 = fieldWeight in 69, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4997177 = idf(docFreq=483, maxDocs=43556)
               0.0546875 = fieldNorm(doc=69)
         0.07385461 = weight(abstract_txt:user in 69) [ClassicSimilarity], result of:
           0.07385461 = score(doc=69,freq=6.0), product of:
             0.14978686 = queryWeight, product of:
               2.2954876 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.017727952 = queryNorm
             0.49306467 = fieldWeight in 69, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.0546875 = fieldNorm(doc=69)
         0.09435345 = weight(abstract_txt:relevance in 69) [ClassicSimilarity], result of:
           0.09435345 = score(doc=69,freq=3.0), product of:
             0.2018789 = queryWeight, product of:
               2.3078852 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.017727952 = queryNorm
             0.46737647 = fieldWeight in 69, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.0546875 = fieldNorm(doc=69)
         0.37249374 = weight(abstract_txt:judgements in 69) [ClassicSimilarity], result of:
           0.37249374 = score(doc=69,freq=9.0), product of:
             0.30544117 = queryWeight, product of:
               2.3178585 = boost
               7.4333076 = idf(docFreq=69, maxDocs=43556)
               0.017727952 = queryNorm
             1.219527 = fieldWeight in 69, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.4333076 = idf(docFreq=69, maxDocs=43556)
               0.0546875 = fieldNorm(doc=69)
         0.11287281 = weight(abstract_txt:ranking in 69) [ClassicSimilarity], result of:
           0.11287281 = score(doc=69,freq=2.0), product of:
             0.26042083 = queryWeight, product of:
               2.6212387 = boost
               5.6041603 = idf(docFreq=435, maxDocs=43556)
               0.017727952 = queryNorm
             0.43342465 = fieldWeight in 69, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.6041603 = idf(docFreq=435, maxDocs=43556)
               0.0546875 = fieldNorm(doc=69)
       0.24 = coord(6/25)
   

4. Wong, S.K.M.; Yao, Y.Y.; Salton, G.; Buckley, C.: Evaluation of an adaptive linear model (1991) 0.17

   0.16886051 = sum of:
     0.16886051 = product of:
       0.84430254 = sum of:
         0.24182425 = weight(abstract_txt:preference in 4833) [ClassicSimilarity], result of:
           0.24182425 = score(doc=4833,freq=1.0), product of:
             0.27452445 = queryWeight, product of:
               2.1974227 = boost
               7.047074 = idf(docFreq=102, maxDocs=43556)
               0.017727952 = queryNorm
             0.88088423 = fieldWeight in 4833, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.047074 = idf(docFreq=102, maxDocs=43556)
               0.125 = fieldNorm(doc=4833)
         0.0974628 = weight(abstract_txt:user in 4833) [ClassicSimilarity], result of:
           0.0974628 = score(doc=4833,freq=2.0), product of:
             0.14978686 = queryWeight, product of:
               2.2954876 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.017727952 = queryNorm
             0.6506766 = fieldWeight in 4833, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.125 = fieldNorm(doc=4833)
         0.12451427 = weight(abstract_txt:relevance in 4833) [ClassicSimilarity], result of:
           0.12451427 = score(doc=4833,freq=1.0), product of:
             0.2018789 = queryWeight, product of:
               2.3078852 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.017727952 = queryNorm
             0.616777 = fieldWeight in 4833, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.125 = fieldNorm(doc=4833)
         0.28380477 = weight(abstract_txt:judgements in 4833) [ClassicSimilarity], result of:
           0.28380477 = score(doc=4833,freq=1.0), product of:
             0.30544117 = queryWeight, product of:
               2.3178585 = boost
               7.4333076 = idf(docFreq=69, maxDocs=43556)
               0.017727952 = queryNorm
             0.92916346 = fieldWeight in 4833, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4333076 = idf(docFreq=69, maxDocs=43556)
               0.125 = fieldNorm(doc=4833)
         0.09669644 = weight(abstract_txt:documents in 4833) [ClassicSimilarity], result of:
           0.09669644 = score(doc=4833,freq=1.0), product of:
             0.18772903 = queryWeight, product of:
               2.5698266 = boost
               4.1206813 = idf(docFreq=1921, maxDocs=43556)
               0.017727952 = queryNorm
             0.51508516 = fieldWeight in 4833, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1206813 = idf(docFreq=1921, maxDocs=43556)
               0.125 = fieldNorm(doc=4833)
       0.2 = coord(5/25)
   

5. Cosijn, E.: Relevance judgments and measurements (2009) 0.16

   0.16370301 = sum of:
     0.16370301 = product of:
       0.5846536 = sum of:
         0.05697693 = weight(abstract_txt:ranked in 853) [ClassicSimilarity], result of:
           0.05697693 = score(doc=853,freq=1.0), product of:
             0.113706104 = queryWeight, product of:
               6.4139447 = idf(docFreq=193, maxDocs=43556)
               0.017727952 = queryNorm
             0.50108945 = fieldWeight in 853, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.4139447 = idf(docFreq=193, maxDocs=43556)
               0.078125 = fieldNorm(doc=853)
         0.0720799 = weight(abstract_txt:measurement in 853) [ClassicSimilarity], result of:
           0.0720799 = score(doc=853,freq=1.0), product of:
             0.13300273 = queryWeight, product of:
               1.0815295 = boost
               6.9368706 = idf(docFreq=114, maxDocs=43556)
               0.017727952 = queryNorm
             0.541943 = fieldWeight in 853, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9368706 = idf(docFreq=114, maxDocs=43556)
               0.078125 = fieldNorm(doc=853)
         0.031313855 = weight(abstract_txt:between in 853) [ClassicSimilarity], result of:
           0.031313855 = score(doc=853,freq=3.0), product of:
             0.06664666 = queryWeight, product of:
               1.0827104 = boost
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.017727952 = queryNorm
             0.46984884 = fieldWeight in 853, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4722223 = idf(docFreq=3675, maxDocs=43556)
               0.078125 = fieldNorm(doc=853)
         0.024710322 = weight(abstract_txt:system in 853) [ClassicSimilarity], result of:
           0.024710322 = score(doc=853,freq=1.0), product of:
             0.09396017 = queryWeight, product of:
               1.5744927 = boost
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.017727952 = queryNorm
             0.2629872 = fieldWeight in 853, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3662362 = idf(docFreq=4086, maxDocs=43556)
               0.078125 = fieldNorm(doc=853)
         0.060914256 = weight(abstract_txt:user in 853) [ClassicSimilarity], result of:
           0.060914256 = score(doc=853,freq=2.0), product of:
             0.14978686 = queryWeight, product of:
               2.2954876 = boost
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.017727952 = queryNorm
             0.4066729 = fieldWeight in 853, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6807828 = idf(docFreq=2983, maxDocs=43556)
               0.078125 = fieldNorm(doc=853)
         0.19062278 = weight(abstract_txt:relevance in 853) [ClassicSimilarity], result of:
           0.19062278 = score(doc=853,freq=6.0), product of:
             0.2018789 = queryWeight, product of:
               2.3078852 = boost
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.017727952 = queryNorm
             0.94424313 = fieldWeight in 853, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.934216 = idf(docFreq=851, maxDocs=43556)
               0.078125 = fieldNorm(doc=853)
         0.14803559 = weight(abstract_txt:documents in 853) [ClassicSimilarity], result of:
           0.14803559 = score(doc=853,freq=6.0), product of:
             0.18772903 = queryWeight, product of:
               2.5698266 = boost
               4.1206813 = idf(docFreq=1921, maxDocs=43556)
               0.017727952 = queryNorm
             0.7885599 = fieldWeight in 853, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.1206813 = idf(docFreq=1921, maxDocs=43556)
               0.078125 = fieldNorm(doc=853)
       0.28 = coord(7/25)