Document (#13224)

Author

Good, I.J.

Title

¬The decision-theory approach to the evaluation of information-retrieval systems

Source

Information storage review. 3(1967), S.31-34

Year

1967

Abstract

It is argued that the evaluation of information-retrieval systems should ultimately be based on the principle of rationality, the maximization of expected utility. In full generality this would involve an estimation of both the cost and value of a system, but the emphasis in this paper is on the problem of value, in terms of which the effiency of the system could be defined. One implication of the discussion is that it is not legitimate to superimpose the 2x2 contingency tables that refer to select/discarded and relevant/irrelevant, correspondending to each request,but it might be all right to superimpose them after applying a monotonic function to the entries. In particular, it is questionable whether a useful statistic is the ratio of the total number of relevant selected documents to the total number of relevant ones, over a sample of requests.

Theme

Retrievalstudien

Similar documents (content)

1. Lancaster, F.W.: Evaluating the performance of a large computerized information system (1985) 0.11

   0.11205794 = sum of:
     0.11205794 = product of:
       0.35018107 = sum of:
         0.04705011 = weight(abstract_txt:request in 3649) [ClassicSimilarity], result of:
           0.04705011 = score(doc=3649,freq=1.0), product of:
             0.14474076 = queryWeight, product of:
               1.0024501 = boost
               6.9347134 = idf(docFreq=116, maxDocs=44218)
               0.020820905 = queryNorm
             0.3250647 = fieldWeight in 3649, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9347134 = idf(docFreq=116, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
         0.023683822 = weight(abstract_txt:retrieval in 3649) [ClassicSimilarity], result of:
           0.023683822 = score(doc=3649,freq=4.0), product of:
             0.07269563 = queryWeight, product of:
               1.0047002 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.020820905 = queryNorm
             0.3257943 = fieldWeight in 3649, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
         0.0125903515 = weight(abstract_txt:that in 3649) [ClassicSimilarity], result of:
           0.0125903515 = score(doc=3649,freq=5.0), product of:
             0.050694335 = queryWeight, product of:
               1.0275612 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020820905 = queryNorm
             0.24835816 = fieldWeight in 3649, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
         0.050749958 = weight(abstract_txt:ultimately in 3649) [ClassicSimilarity], result of:
           0.050749958 = score(doc=3649,freq=1.0), product of:
             0.15223256 = queryWeight, product of:
               1.0280663 = boost
               7.11192 = idf(docFreq=97, maxDocs=44218)
               0.020820905 = queryNorm
             0.33337125 = fieldWeight in 3649, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.11192 = idf(docFreq=97, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
         0.057640232 = weight(abstract_txt:irrelevant in 3649) [ClassicSimilarity], result of:
           0.057640232 = score(doc=3649,freq=1.0), product of:
             0.1657172 = queryWeight, product of:
               1.0726329 = boost
               7.4202213 = idf(docFreq=71, maxDocs=44218)
               0.020820905 = queryNorm
             0.34782287 = fieldWeight in 3649, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4202213 = idf(docFreq=71, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
         0.034494616 = weight(abstract_txt:number in 3649) [ClassicSimilarity], result of:
           0.034494616 = score(doc=3649,freq=3.0), product of:
             0.10280658 = queryWeight, product of:
               1.1947933 = boost
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.020820905 = queryNorm
             0.33552927 = fieldWeight in 3649, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
         0.05094868 = weight(abstract_txt:evaluation in 3649) [ClassicSimilarity], result of:
           0.05094868 = score(doc=3649,freq=4.0), product of:
             0.121142276 = queryWeight, product of:
               1.2969708 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.020820905 = queryNorm
             0.42056894 = fieldWeight in 3649, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
         0.07302332 = weight(abstract_txt:relevant in 3649) [ClassicSimilarity], result of:
           0.07302332 = score(doc=3649,freq=3.0), product of:
             0.1940252 = queryWeight, product of:
               2.0102823 = boost
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.020820905 = queryNorm
             0.37635997 = fieldWeight in 3649, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.046875 = fieldNorm(doc=3649)
       0.32 = coord(8/25)
   

2. Blagden, J.F.: How much noise in a role-free and link-free co-ordinate indexing system? (1966) 0.08

   0.07874056 = sum of:
     0.07874056 = product of:
       0.32808566 = sum of:
         0.019736517 = weight(abstract_txt:retrieval in 2718) [ClassicSimilarity], result of:
           0.019736517 = score(doc=2718,freq=1.0), product of:
             0.07269563 = queryWeight, product of:
               1.0047002 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.020820905 = queryNorm
             0.27149525 = fieldWeight in 2718, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.078125 = fieldNorm(doc=2718)
         0.009384294 = weight(abstract_txt:that in 2718) [ClassicSimilarity], result of:
           0.009384294 = score(doc=2718,freq=1.0), product of:
             0.050694335 = queryWeight, product of:
               1.0275612 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020820905 = queryNorm
             0.18511525 = fieldWeight in 2718, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=2718)
         0.13585933 = weight(abstract_txt:irrelevant in 2718) [ClassicSimilarity], result of:
           0.13585933 = score(doc=2718,freq=2.0), product of:
             0.1657172 = queryWeight, product of:
               1.0726329 = boost
               7.4202213 = idf(docFreq=71, maxDocs=44218)
               0.020820905 = queryNorm
             0.81982636 = fieldWeight in 2718, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4202213 = idf(docFreq=71, maxDocs=44218)
               0.078125 = fieldNorm(doc=2718)
         0.033192463 = weight(abstract_txt:number in 2718) [ClassicSimilarity], result of:
           0.033192463 = score(doc=2718,freq=1.0), product of:
             0.10280658 = queryWeight, product of:
               1.1947933 = boost
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.020820905 = queryNorm
             0.3228632 = fieldWeight in 2718, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.078125 = fieldNorm(doc=2718)
         0.042457234 = weight(abstract_txt:evaluation in 2718) [ClassicSimilarity], result of:
           0.042457234 = score(doc=2718,freq=1.0), product of:
             0.121142276 = queryWeight, product of:
               1.2969708 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.020820905 = queryNorm
             0.35047412 = fieldWeight in 2718, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=2718)
         0.08745584 = weight(abstract_txt:total in 2718) [ClassicSimilarity], result of:
           0.08745584 = score(doc=2718,freq=1.0), product of:
             0.19611935 = queryWeight, product of:
               1.6502228 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.020820905 = queryNorm
             0.4459317 = fieldWeight in 2718, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.078125 = fieldNorm(doc=2718)
       0.24 = coord(6/25)
   

3. Plaunt, C.; Norgard, B.A.: ¬An association-based method for automatic indexing with a controlled vocabulary (1998) 0.08

   0.0780888 = sum of:
     0.0780888 = product of:
       0.32536998 = sum of:
         0.015789215 = weight(abstract_txt:retrieval in 1794) [ClassicSimilarity], result of:
           0.015789215 = score(doc=1794,freq=1.0), product of:
             0.07269563 = queryWeight, product of:
               1.0047002 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.020820905 = queryNorm
             0.21719621 = fieldWeight in 1794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=1794)
         0.0106171165 = weight(abstract_txt:that in 1794) [ClassicSimilarity], result of:
           0.0106171165 = score(doc=1794,freq=2.0), product of:
             0.050694335 = queryWeight, product of:
               1.0275612 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020820905 = queryNorm
             0.20943399 = fieldWeight in 1794, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=1794)
         0.08265907 = weight(abstract_txt:ratio in 1794) [ClassicSimilarity], result of:
           0.08265907 = score(doc=1794,freq=1.0), product of:
             0.1739609 = queryWeight, product of:
               1.0989885 = boost
               7.602543 = idf(docFreq=59, maxDocs=44218)
               0.020820905 = queryNorm
             0.47515893 = fieldWeight in 1794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.602543 = idf(docFreq=59, maxDocs=44218)
               0.0625 = fieldNorm(doc=1794)
         0.1261254 = weight(abstract_txt:statistic in 1794) [ClassicSimilarity], result of:
           0.1261254 = score(doc=1794,freq=1.0), product of:
             0.23056477 = queryWeight, product of:
               1.2652135 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.020820905 = queryNorm
             0.547028 = fieldWeight in 1794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=1794)
         0.033965785 = weight(abstract_txt:evaluation in 1794) [ClassicSimilarity], result of:
           0.033965785 = score(doc=1794,freq=1.0), product of:
             0.121142276 = queryWeight, product of:
               1.2969708 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.020820905 = queryNorm
             0.2803793 = fieldWeight in 1794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=1794)
         0.05621338 = weight(abstract_txt:relevant in 1794) [ClassicSimilarity], result of:
           0.05621338 = score(doc=1794,freq=1.0), product of:
             0.1940252 = queryWeight, product of:
               2.0102823 = boost
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.020820905 = queryNorm
             0.28972206 = fieldWeight in 1794, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.0625 = fieldNorm(doc=1794)
       0.24 = coord(6/25)
   

4. Kantor, P.; Kim, M.H.; Ibraev, U.; Atasoy, K.: Estimating the number of relevant documents in enormous collections (1999) 0.07

   0.07321561 = sum of:
     0.07321561 = product of:
       0.36607805 = sum of:
         0.022329321 = weight(abstract_txt:retrieval in 6690) [ClassicSimilarity], result of:
           0.022329321 = score(doc=6690,freq=2.0), product of:
             0.07269563 = queryWeight, product of:
               1.0047002 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.020820905 = queryNorm
             0.3071618 = fieldWeight in 6690, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=6690)
         0.016787136 = weight(abstract_txt:that in 6690) [ClassicSimilarity], result of:
           0.016787136 = score(doc=6690,freq=5.0), product of:
             0.050694335 = queryWeight, product of:
               1.0275612 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020820905 = queryNorm
             0.3311442 = fieldWeight in 6690, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=6690)
         0.059376482 = weight(abstract_txt:number in 6690) [ClassicSimilarity], result of:
           0.059376482 = score(doc=6690,freq=5.0), product of:
             0.10280658 = queryWeight, product of:
               1.1947933 = boost
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.020820905 = queryNorm
             0.57755524 = fieldWeight in 6690, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.0625 = fieldNorm(doc=6690)
         0.09894498 = weight(abstract_txt:total in 6690) [ClassicSimilarity], result of:
           0.09894498 = score(doc=6690,freq=2.0), product of:
             0.19611935 = queryWeight, product of:
               1.6502228 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.020820905 = queryNorm
             0.5045141 = fieldWeight in 6690, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.0625 = fieldNorm(doc=6690)
         0.16864014 = weight(abstract_txt:relevant in 6690) [ClassicSimilarity], result of:
           0.16864014 = score(doc=6690,freq=9.0), product of:
             0.1940252 = queryWeight, product of:
               2.0102823 = boost
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.020820905 = queryNorm
             0.86916614 = fieldWeight in 6690, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.635553 = idf(docFreq=1165, maxDocs=44218)
               0.0625 = fieldNorm(doc=6690)
       0.2 = coord(5/25)
   

5. Bletsas, A.; Sahalos, J.N.: Hirsch index rankings require scaling and higher moment (2009) 0.07

   0.06895842 = sum of:
     0.06895842 = product of:
       0.34479207 = sum of:
         0.016254075 = weight(abstract_txt:that in 3141) [ClassicSimilarity], result of:
           0.016254075 = score(doc=3141,freq=3.0), product of:
             0.050694335 = queryWeight, product of:
               1.0275612 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020820905 = queryNorm
             0.320629 = fieldWeight in 3141, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=3141)
         0.11550267 = weight(abstract_txt:estimation in 3141) [ClassicSimilarity], result of:
           0.11550267 = score(doc=3141,freq=1.0), product of:
             0.18737541 = queryWeight, product of:
               1.1405745 = boost
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.020820905 = queryNorm
             0.61642385 = fieldWeight in 3141, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.078125 = fieldNorm(doc=3141)
         0.057491034 = weight(abstract_txt:number in 3141) [ClassicSimilarity], result of:
           0.057491034 = score(doc=3141,freq=3.0), product of:
             0.10280658 = queryWeight, product of:
               1.1947933 = boost
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.020820905 = queryNorm
             0.5592155 = fieldWeight in 3141, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.132649 = idf(docFreq=1927, maxDocs=44218)
               0.078125 = fieldNorm(doc=3141)
         0.06808845 = weight(abstract_txt:value in 3141) [ClassicSimilarity], result of:
           0.06808845 = score(doc=3141,freq=3.0), product of:
             0.11508087 = queryWeight, product of:
               1.2641072 = boost
               4.3723974 = idf(docFreq=1516, maxDocs=44218)
               0.020820905 = queryNorm
             0.5916574 = fieldWeight in 3141, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.3723974 = idf(docFreq=1516, maxDocs=44218)
               0.078125 = fieldNorm(doc=3141)
         0.08745584 = weight(abstract_txt:total in 3141) [ClassicSimilarity], result of:
           0.08745584 = score(doc=3141,freq=1.0), product of:
             0.19611935 = queryWeight, product of:
               1.6502228 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.020820905 = queryNorm
             0.4459317 = fieldWeight in 3141, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.078125 = fieldNorm(doc=3141)
       0.2 = coord(5/25)