Document (#24977)

Author

Losee, R.M.

Title

Term dependence : a basis for Luhn and Zipf models

Source

Journal of the American Society for Information Science and technology. 52(2001) no.12, S.1019-1025

Year

2001

Abstract

There are regularities in the statistical information provided by natural language terms about neighboring terms. We find that when phrase rank increases, moving from common to less common phrases, the value of the expected mutual information measure (EMIM) between the terms regularly decreases. Luhn's model suggests that midrange terms are the best index terms and relevance discriminators. We suggest reasons for this principle based on the empirical relationships shown here between the rank of terms within phrases and the average mutual information between terms, which we refer to as the Inverse Representation- EMIM principle. We also suggest an Inverse EMIM term weight for indexing or retrieval applications that is consistent with Luhn's distribution. An information theoretic interpretation of Zipf's Law is provided. Using the regularity noted here, we suggest that Zipf's Law is a consequence of the statistical dependencies that exist between terms, described here using information theoretic concepts.

Theme

Informetrie

Object

Luhn-Modell
Zipf-Gesetz

Similar documents (author)

1. Losee, R.M.: ¬A Gray code based ordering for documents on shelves : classification for browsing and retrieval (1992) 5.18

   5.184806 = sum of:
     5.184806 = weight(author_txt:losee in 2335) [ClassicSimilarity], result of:
       5.184806 = fieldWeight in 2335, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.29569 = idf(docFreq=29, maxDocs=44218)
         0.625 = fieldNorm(doc=2335)
   

2. Losee, R.M.: ¬The relative shelf location of circulated books : a study of classification, users, and browsing (1993) 5.18

   5.184806 = sum of:
     5.184806 = weight(author_txt:losee in 4485) [ClassicSimilarity], result of:
       5.184806 = fieldWeight in 4485, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.29569 = idf(docFreq=29, maxDocs=44218)
         0.625 = fieldNorm(doc=4485)
   

3. Losee, R.M.: Seven fundamental questions for the science of library classification (1993) 5.18

   5.184806 = sum of:
     5.184806 = weight(author_txt:losee in 4508) [ClassicSimilarity], result of:
       5.184806 = fieldWeight in 4508, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.29569 = idf(docFreq=29, maxDocs=44218)
         0.625 = fieldNorm(doc=4508)
   

4. Losee, R.M.: Term dependence : truncating the Bahadur Lazarsfeld expansion (1994) 5.18

   5.184806 = sum of:
     5.184806 = weight(author_txt:losee in 7390) [ClassicSimilarity], result of:
       5.184806 = fieldWeight in 7390, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.29569 = idf(docFreq=29, maxDocs=44218)
         0.625 = fieldNorm(doc=7390)
   

5. Losee, R.M.: Upper bounds for retrieval performance and their user measuring performance and generating optimal queries : can it get any better than this? (1994) 5.18

   5.184806 = sum of:
     5.184806 = weight(author_txt:losee in 7418) [ClassicSimilarity], result of:
       5.184806 = fieldWeight in 7418, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.29569 = idf(docFreq=29, maxDocs=44218)
         0.625 = fieldNorm(doc=7418)
   

Similar documents (content)

1. Ferrer-i-Cancho, R.; Vitevitch, M.S.: ¬The origins of Zipf's meaning-frequency law (2018) 0.22

   0.22401409 = sum of:
     0.22401409 = product of:
       0.93339205 = sum of:
         0.113814466 = weight(abstract_txt:zipf in 4546) [ClassicSimilarity], result of:
           0.113814466 = score(doc=4546,freq=1.0), product of:
             0.16928324 = queryWeight, product of:
               1.0967414 = boost
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.017935613 = queryNorm
             0.6723316 = fieldWeight in 4546, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.078125 = fieldNorm(doc=4546)
         0.023756009 = weight(abstract_txt:that in 4546) [ClassicSimilarity], result of:
           0.023756009 = score(doc=4546,freq=4.0), product of:
             0.06416546 = queryWeight, product of:
               1.5098469 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017935613 = queryNorm
             0.3702305 = fieldWeight in 4546, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=4546)
         0.13389237 = weight(abstract_txt:rank in 4546) [ClassicSimilarity], result of:
           0.13389237 = score(doc=4546,freq=2.0), product of:
             0.1886483 = queryWeight, product of:
               1.6373394 = boost
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.017935613 = queryNorm
             0.709746 = fieldWeight in 4546, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.078125 = fieldNorm(doc=4546)
         0.041965652 = weight(abstract_txt:between in 4546) [ClassicSimilarity], result of:
           0.041965652 = score(doc=4546,freq=2.0), product of:
             0.10967011 = queryWeight, product of:
               1.765515 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.017935613 = queryNorm
             0.3826535 = fieldWeight in 4546, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.078125 = fieldNorm(doc=4546)
         0.08965067 = weight(abstract_txt:here in 4546) [ClassicSimilarity], result of:
           0.08965067 = score(doc=4546,freq=1.0), product of:
             0.20823725 = queryWeight, product of:
               2.1068668 = boost
               5.510679 = idf(docFreq=485, maxDocs=44218)
               0.017935613 = queryNorm
             0.4305218 = fieldWeight in 4546, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.510679 = idf(docFreq=485, maxDocs=44218)
               0.078125 = fieldNorm(doc=4546)
         0.53031284 = weight(abstract_txt:zipf's in 4546) [ClassicSimilarity], result of:
           0.53031284 = score(doc=4546,freq=3.0), product of:
             0.4125469 = queryWeight, product of:
               2.4213018 = boost
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.017935613 = queryNorm
             1.2854607 = fieldWeight in 4546, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.078125 = fieldNorm(doc=4546)
       0.24 = coord(6/25)
   

2. Losee, R.M.: Decisions in thesaurus construction and use (2007) 0.18

   0.17634702 = sum of:
     0.17634702 = product of:
       0.62981075 = sum of:
         0.055900108 = weight(abstract_txt:term in 924) [ClassicSimilarity], result of:
           0.055900108 = score(doc=924,freq=2.0), product of:
             0.10537981 = queryWeight, product of:
               1.2237451 = boost
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.017935613 = queryNorm
             0.53046316 = fieldWeight in 924, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.078125 = fieldNorm(doc=924)
         0.016798034 = weight(abstract_txt:that in 924) [ClassicSimilarity], result of:
           0.016798034 = score(doc=924,freq=2.0), product of:
             0.06416546 = queryWeight, product of:
               1.5098469 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017935613 = queryNorm
             0.26179248 = fieldWeight in 924, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=924)
         0.0126690045 = weight(abstract_txt:information in 924) [ClassicSimilarity], result of:
           0.0126690045 = score(doc=924,freq=1.0), product of:
             0.06698342 = queryWeight, product of:
               1.5426447 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.017935613 = queryNorm
             0.18913643 = fieldWeight in 924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.078125 = fieldNorm(doc=924)
         0.11573019 = weight(abstract_txt:phrases in 924) [ClassicSimilarity], result of:
           0.11573019 = score(doc=924,freq=1.0), product of:
             0.21567017 = queryWeight, product of:
               1.7506821 = boost
               6.8685737 = idf(docFreq=124, maxDocs=44218)
               0.017935613 = queryNorm
             0.5366073 = fieldWeight in 924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8685737 = idf(docFreq=124, maxDocs=44218)
               0.078125 = fieldNorm(doc=924)
         0.17543457 = weight(abstract_txt:theoretic in 924) [ClassicSimilarity], result of:
           0.17543457 = score(doc=924,freq=1.0), product of:
             0.28460056 = queryWeight, product of:
               2.0110836 = boost
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.017935613 = queryNorm
             0.61642385 = fieldWeight in 924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.078125 = fieldNorm(doc=924)
         0.08965067 = weight(abstract_txt:here in 924) [ClassicSimilarity], result of:
           0.08965067 = score(doc=924,freq=1.0), product of:
             0.20823725 = queryWeight, product of:
               2.1068668 = boost
               5.510679 = idf(docFreq=485, maxDocs=44218)
               0.017935613 = queryNorm
             0.4305218 = fieldWeight in 924, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.510679 = idf(docFreq=485, maxDocs=44218)
               0.078125 = fieldNorm(doc=924)
         0.16362818 = weight(abstract_txt:terms in 924) [ClassicSimilarity], result of:
           0.16362818 = score(doc=924,freq=3.0), product of:
             0.29902714 = queryWeight, product of:
               4.1228504 = boost
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.017935613 = queryNorm
             0.54720175 = fieldWeight in 924, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.078125 = fieldNorm(doc=924)
       0.28 = coord(7/25)
   

3. Wong, S.K.M.; Yao, Y.Y.: ¬An information-theoretic measure of term specifics (1992) 0.16

   0.16464943 = sum of:
     0.16464943 = product of:
       0.68603927 = sum of:
         0.1161862 = weight(abstract_txt:term in 4807) [ClassicSimilarity], result of:
           0.1161862 = score(doc=4807,freq=6.0), product of:
             0.10537981 = queryWeight, product of:
               1.2237451 = boost
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.017935613 = queryNorm
             1.102547 = fieldWeight in 4807, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.09375 = fieldNorm(doc=4807)
         0.020157643 = weight(abstract_txt:that in 4807) [ClassicSimilarity], result of:
           0.020157643 = score(doc=4807,freq=2.0), product of:
             0.06416546 = queryWeight, product of:
               1.5098469 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017935613 = queryNorm
             0.314151 = fieldWeight in 4807, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.09375 = fieldNorm(doc=4807)
         0.021500014 = weight(abstract_txt:information in 4807) [ClassicSimilarity], result of:
           0.021500014 = score(doc=4807,freq=2.0), product of:
             0.06698342 = queryWeight, product of:
               1.5426447 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.017935613 = queryNorm
             0.32097518 = fieldWeight in 4807, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.09375 = fieldNorm(doc=4807)
         0.035609033 = weight(abstract_txt:between in 4807) [ClassicSimilarity], result of:
           0.035609033 = score(doc=4807,freq=1.0), product of:
             0.10967011 = queryWeight, product of:
               1.765515 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.017935613 = queryNorm
             0.32469225 = fieldWeight in 4807, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.09375 = fieldNorm(doc=4807)
         0.19486406 = weight(abstract_txt:inverse in 4807) [ClassicSimilarity], result of:
           0.19486406 = score(doc=4807,freq=1.0), product of:
             0.27030826 = queryWeight, product of:
               1.9599361 = boost
               7.689554 = idf(docFreq=54, maxDocs=44218)
               0.017935613 = queryNorm
             0.7208957 = fieldWeight in 4807, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.689554 = idf(docFreq=54, maxDocs=44218)
               0.09375 = fieldNorm(doc=4807)
         0.2977223 = weight(abstract_txt:theoretic in 4807) [ClassicSimilarity], result of:
           0.2977223 = score(doc=4807,freq=2.0), product of:
             0.28460056 = queryWeight, product of:
               2.0110836 = boost
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.017935613 = queryNorm
             1.0461059 = fieldWeight in 4807, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.09375 = fieldNorm(doc=4807)
       0.24 = coord(6/25)
   

4. Aizawa, A.: ¬An information-theoretic perspective of tf-idf measures (2003) 0.16

   0.15928891 = sum of:
     0.15928891 = product of:
       0.6637038 = sum of:
         0.047432818 = weight(abstract_txt:term in 4155) [ClassicSimilarity], result of:
           0.047432818 = score(doc=4155,freq=1.0), product of:
             0.10537981 = queryWeight, product of:
               1.2237451 = boost
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.017935613 = queryNorm
             0.45011294 = fieldWeight in 4155, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.09375 = fieldNorm(doc=4155)
         0.020157643 = weight(abstract_txt:that in 4155) [ClassicSimilarity], result of:
           0.020157643 = score(doc=4155,freq=2.0), product of:
             0.06416546 = queryWeight, product of:
               1.5098469 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017935613 = queryNorm
             0.314151 = fieldWeight in 4155, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.09375 = fieldNorm(doc=4155)
         0.030405613 = weight(abstract_txt:information in 4155) [ClassicSimilarity], result of:
           0.030405613 = score(doc=4155,freq=4.0), product of:
             0.06698342 = queryWeight, product of:
               1.5426447 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.017935613 = queryNorm
             0.45392746 = fieldWeight in 4155, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.09375 = fieldNorm(doc=4155)
         0.19486406 = weight(abstract_txt:inverse in 4155) [ClassicSimilarity], result of:
           0.19486406 = score(doc=4155,freq=1.0), product of:
             0.27030826 = queryWeight, product of:
               1.9599361 = boost
               7.689554 = idf(docFreq=54, maxDocs=44218)
               0.017935613 = queryNorm
             0.7208957 = fieldWeight in 4155, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.689554 = idf(docFreq=54, maxDocs=44218)
               0.09375 = fieldNorm(doc=4155)
         0.21052147 = weight(abstract_txt:theoretic in 4155) [ClassicSimilarity], result of:
           0.21052147 = score(doc=4155,freq=1.0), product of:
             0.28460056 = queryWeight, product of:
               2.0110836 = boost
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.017935613 = queryNorm
             0.7397086 = fieldWeight in 4155, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.09375 = fieldNorm(doc=4155)
         0.16032222 = weight(abstract_txt:terms in 4155) [ClassicSimilarity], result of:
           0.16032222 = score(doc=4155,freq=2.0), product of:
             0.29902714 = queryWeight, product of:
               4.1228504 = boost
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.017935613 = queryNorm
             0.53614604 = fieldWeight in 4155, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.09375 = fieldNorm(doc=4155)
       0.24 = coord(6/25)
   

5. Egghe, L.: Zipfian and Lotkaian continuous concentration theory (2005) 0.15

   0.15421483 = sum of:
     0.15421483 = product of:
       0.7710741 = sum of:
         0.19713245 = weight(abstract_txt:zipf in 3678) [ClassicSimilarity], result of:
           0.19713245 = score(doc=3678,freq=3.0), product of:
             0.16928324 = queryWeight, product of:
               1.0967414 = boost
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.017935613 = queryNorm
             1.1645125 = fieldWeight in 3678, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.078125 = fieldNorm(doc=3678)
         0.016798034 = weight(abstract_txt:that in 3678) [ClassicSimilarity], result of:
           0.016798034 = score(doc=3678,freq=2.0), product of:
             0.06416546 = queryWeight, product of:
               1.5098469 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017935613 = queryNorm
             0.26179248 = fieldWeight in 3678, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=3678)
         0.029674198 = weight(abstract_txt:between in 3678) [ClassicSimilarity], result of:
           0.029674198 = score(doc=3678,freq=1.0), product of:
             0.10967011 = queryWeight, product of:
               1.765515 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.017935613 = queryNorm
             0.2705769 = fieldWeight in 3678, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.078125 = fieldNorm(doc=3678)
         0.43299866 = weight(abstract_txt:zipf's in 3678) [ClassicSimilarity], result of:
           0.43299866 = score(doc=3678,freq=2.0), product of:
             0.4125469 = queryWeight, product of:
               2.4213018 = boost
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.017935613 = queryNorm
             1.0495744 = fieldWeight in 3678, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.078125 = fieldNorm(doc=3678)
         0.09447078 = weight(abstract_txt:terms in 3678) [ClassicSimilarity], result of:
           0.09447078 = score(doc=3678,freq=1.0), product of:
             0.29902714 = queryWeight, product of:
               4.1228504 = boost
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.017935613 = queryNorm
             0.3159271 = fieldWeight in 3678, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.078125 = fieldNorm(doc=3678)
       0.2 = coord(5/25)