Document (#23771)

Author

Chung, Y.M.
Lee, J.Y.

Title

¬A corpus-based approach to comparative evaluation of statistical term association measures

Source

Journal of the American Society for Information Science and technology. 52(2001) no.4, S.283-296

Year

2001

Abstract

Statistical association measures have been widely applied in information retrieval research, usually employing a clustering of documents or terms on the basis of their relationships. Applications of the association measures for term clustering include automatic thesaurus construction and query expansion. This research evaluates the similarity of six association measures by comparing the relationship and behavior they demonstrate in various analyses of a test corpus. Analysis techniques include comparisons of highly ranked term pairs and term clusters, analyses of the correlation among the association measures using Pearson's correlation coefficient and MDS mapping, and an analysis of the impact of a term frequency on the association values by means of z-score. The major findings of the study are as follows: First, the most similar association measures are mutual information and Yule's coefficient of colligation Y, whereas cosine and Jaccard coefficients, as well as X**2 statistic and likelihood ratio, demonstrate quite similar behavior for terms with high frequency. Second, among all the measures, the X**2 statistic is the least affected by the frequency of terms. Third, although cosine and Jaccard coefficients tend to emphasize high frequency terms, mutual information and Yule's Y seem to overestimate rare terms

Theme

Automatisches Indexieren
Automatisches Klassifizieren

Similar documents (author)

1. Chung, T.M.: ¬A corpus comparison approach for terminology extraction (2003) 5.09

   5.085331 = sum of:
     5.085331 = weight(author_txt:chung in 73) [ClassicSimilarity], result of:
       5.085331 = fieldWeight in 73, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.13653 = idf(docFreq=33, maxDocs=42740)
         0.625 = fieldNorm(doc=73)
   

2. Chung, H.H.: User friendly audiovisual material cataloging at Westchester County Public Library System (2001) 5.09

   5.085331 = sum of:
     5.085331 = weight(author_txt:chung in 416) [ClassicSimilarity], result of:
       5.085331 = fieldWeight in 416, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.13653 = idf(docFreq=33, maxDocs=42740)
         0.625 = fieldNorm(doc=416)
   

3. Chung, Y.-K.: Characteristics of references in international classification systems literature (1995) 4.07

   4.068265 = sum of:
     4.068265 = weight(author_txt:chung in 3008) [ClassicSimilarity], result of:
       4.068265 = fieldWeight in 3008, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.13653 = idf(docFreq=33, maxDocs=42740)
         0.5 = fieldNorm(doc=3008)
   

4. Chung, Y.-K.: Bradford distribution and core authors in classification systems literature (1994) 4.07

   4.068265 = sum of:
     4.068265 = weight(author_txt:chung in 5135) [ClassicSimilarity], result of:
       4.068265 = fieldWeight in 5135, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.13653 = idf(docFreq=33, maxDocs=42740)
         0.5 = fieldNorm(doc=5135)
   

5. Chung, Y.-K.: Core international journals of classification systems : an application of Bradford's law (1994) 4.07

   4.068265 = sum of:
     4.068265 = weight(author_txt:chung in 5139) [ClassicSimilarity], result of:
       4.068265 = fieldWeight in 5139, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.13653 = idf(docFreq=33, maxDocs=42740)
         0.5 = fieldNorm(doc=5139)
   

Similar documents (content)

1. Eck, N.J. van; Waltman, L.: How to normalize cooccurrence data? : an analysis of some well-known similarity measures (2009) 0.31

   0.3081331 = sum of:
     0.3081331 = product of:
       1.283888 = sum of:
         0.0269773 = weight(abstract_txt:among in 4943) [ClassicSimilarity], result of:
           0.0269773 = score(doc=4943,freq=1.0), product of:
             0.07593711 = queryWeight, product of:
               1.1989708 = boost
               4.5473084 = idf(docFreq=1230, maxDocs=42740)
               0.013928073 = queryNorm
             0.35525846 = fieldWeight in 4943, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5473084 = idf(docFreq=1230, maxDocs=42740)
               0.078125 = fieldNorm(doc=4943)
         0.042379625 = weight(abstract_txt:behavior in 4943) [ClassicSimilarity], result of:
           0.042379625 = score(doc=4943,freq=1.0), product of:
             0.102618694 = queryWeight, product of:
               1.3937819 = boost
               5.286164 = idf(docFreq=587, maxDocs=42740)
               0.013928073 = queryNorm
             0.41298154 = fieldWeight in 4943, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.286164 = idf(docFreq=587, maxDocs=42740)
               0.078125 = fieldNorm(doc=4943)
         0.1904113 = weight(abstract_txt:cosine in 4943) [ClassicSimilarity], result of:
           0.1904113 = score(doc=4943,freq=2.0), product of:
             0.22177209 = queryWeight, product of:
               2.0489674 = boost
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.013928073 = queryNorm
             0.85859 = fieldWeight in 4943, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.078125 = fieldNorm(doc=4943)
         0.28510168 = weight(abstract_txt:jaccard in 4943) [ClassicSimilarity], result of:
           0.28510168 = score(doc=4943,freq=2.0), product of:
             0.29025403 = queryWeight, product of:
               2.3440707 = boost
               8.890302 = idf(docFreq=15, maxDocs=42740)
               0.013928073 = queryNorm
             0.98224884 = fieldWeight in 4943, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.890302 = idf(docFreq=15, maxDocs=42740)
               0.078125 = fieldNorm(doc=4943)
         0.4254592 = weight(abstract_txt:measures in 4943) [ClassicSimilarity], result of:
           0.4254592 = score(doc=4943,freq=7.0), product of:
             0.3790388 = queryWeight, product of:
               5.011381 = boost
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.013928073 = queryNorm
             1.1224687 = fieldWeight in 4943, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.078125 = fieldNorm(doc=4943)
         0.31355888 = weight(abstract_txt:association in 4943) [ClassicSimilarity], result of:
           0.31355888 = score(doc=4943,freq=3.0), product of:
             0.4101884 = queryWeight, product of:
               5.213235 = boost
               5.649175 = idf(docFreq=408, maxDocs=42740)
               0.013928073 = queryNorm
             0.76442647 = fieldWeight in 4943, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.649175 = idf(docFreq=408, maxDocs=42740)
               0.078125 = fieldNorm(doc=4943)
       0.24 = coord(6/25)
   

2. Alzahrani, S.; Palade, V.; Salim, N.; Abraham, A.: Using structural information and citation evidence to detect significant plagiarism cases in scientific publications (2012) 0.28

   0.2797458 = sum of:
     0.2797458 = product of:
       0.7770716 = sum of:
         0.02891628 = weight(abstract_txt:similar in 1983) [ClassicSimilarity], result of:
           0.02891628 = score(doc=1983,freq=1.0), product of:
             0.100883 = queryWeight, product of:
               1.3819444 = boost
               5.241268 = idf(docFreq=614, maxDocs=42740)
               0.013928073 = queryNorm
             0.28663185 = fieldWeight in 1983, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.241268 = idf(docFreq=614, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.03226684 = weight(abstract_txt:demonstrate in 1983) [ClassicSimilarity], result of:
           0.03226684 = score(doc=1983,freq=1.0), product of:
             0.10853272 = queryWeight, product of:
               1.4333819 = boost
               5.4363537 = idf(docFreq=505, maxDocs=42740)
               0.013928073 = queryNorm
             0.29730058 = fieldWeight in 1983, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4363537 = idf(docFreq=505, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.04841772 = weight(abstract_txt:statistical in 1983) [ClassicSimilarity], result of:
           0.04841772 = score(doc=1983,freq=2.0), product of:
             0.11290572 = queryWeight, product of:
               1.4619737 = boost
               5.544793 = idf(docFreq=453, maxDocs=42740)
               0.013928073 = queryNorm
             0.42883322 = fieldWeight in 1983, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.544793 = idf(docFreq=453, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.13124001 = weight(abstract_txt:coefficient in 1983) [ClassicSimilarity], result of:
           0.13124001 = score(doc=1983,freq=2.0), product of:
             0.21949463 = queryWeight, product of:
               2.0384192 = boost
               7.731065 = idf(docFreq=50, maxDocs=42740)
               0.013928073 = queryNorm
             0.597919 = fieldWeight in 1983, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.731065 = idf(docFreq=50, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.094248794 = weight(abstract_txt:cosine in 1983) [ClassicSimilarity], result of:
           0.094248794 = score(doc=1983,freq=1.0), product of:
             0.22177209 = queryWeight, product of:
               2.0489674 = boost
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.013928073 = queryNorm
             0.4249804 = fieldWeight in 1983, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.14111814 = weight(abstract_txt:jaccard in 1983) [ClassicSimilarity], result of:
           0.14111814 = score(doc=1983,freq=1.0), product of:
             0.29025403 = queryWeight, product of:
               2.3440707 = boost
               8.890302 = idf(docFreq=15, maxDocs=42740)
               0.013928073 = queryNorm
             0.48618838 = fieldWeight in 1983, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.890302 = idf(docFreq=15, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.08514065 = weight(abstract_txt:frequency in 1983) [ClassicSimilarity], result of:
           0.08514065 = score(doc=1983,freq=1.0), product of:
             0.2611105 = queryWeight, product of:
               3.1441898 = boost
               5.962447 = idf(docFreq=298, maxDocs=42740)
               0.013928073 = queryNorm
             0.32607132 = fieldWeight in 1983, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.962447 = idf(docFreq=298, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.056530915 = weight(abstract_txt:term in 1983) [ClassicSimilarity], result of:
           0.056530915 = score(doc=1983,freq=1.0), product of:
             0.21407226 = queryWeight, product of:
               3.1829643 = boost
               4.8287816 = idf(docFreq=928, maxDocs=42740)
               0.013928073 = queryNorm
             0.264074 = fieldWeight in 1983, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.8287816 = idf(docFreq=928, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
         0.15919226 = weight(abstract_txt:measures in 1983) [ClassicSimilarity], result of:
           0.15919226 = score(doc=1983,freq=2.0), product of:
             0.3790388 = queryWeight, product of:
               5.011381 = boost
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.013928073 = queryNorm
             0.41998935 = fieldWeight in 1983, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.0546875 = fieldNorm(doc=1983)
       0.36 = coord(9/25)
   

3. Schneider, J.W.; Borlund, P.: Matrix comparison, part 1 : motivation and important issues for measuring the resemblance between proximity measures or ordination results (2007) 0.24

   0.23673952 = sum of:
     0.23673952 = product of:
       0.739811 = sum of:
         0.050084468 = weight(abstract_txt:similar in 2585) [ClassicSimilarity], result of:
           0.050084468 = score(doc=2585,freq=3.0), product of:
             0.100883 = queryWeight, product of:
               1.3819444 = boost
               5.241268 = idf(docFreq=614, maxDocs=42740)
               0.013928073 = queryNorm
             0.4964609 = fieldWeight in 2585, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.241268 = idf(docFreq=614, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
         0.041953687 = weight(abstract_txt:behavior in 2585) [ClassicSimilarity], result of:
           0.041953687 = score(doc=2585,freq=2.0), product of:
             0.102618694 = queryWeight, product of:
               1.3937819 = boost
               5.286164 = idf(docFreq=587, maxDocs=42740)
               0.013928073 = queryNorm
             0.40883085 = fieldWeight in 2585, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.286164 = idf(docFreq=587, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
         0.03226684 = weight(abstract_txt:demonstrate in 2585) [ClassicSimilarity], result of:
           0.03226684 = score(doc=2585,freq=1.0), product of:
             0.10853272 = queryWeight, product of:
               1.4333819 = boost
               5.4363537 = idf(docFreq=505, maxDocs=42740)
               0.013928073 = queryNorm
             0.29730058 = fieldWeight in 2585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4363537 = idf(docFreq=505, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
         0.04841772 = weight(abstract_txt:statistical in 2585) [ClassicSimilarity], result of:
           0.04841772 = score(doc=2585,freq=2.0), product of:
             0.11290572 = queryWeight, product of:
               1.4619737 = boost
               5.544793 = idf(docFreq=453, maxDocs=42740)
               0.013928073 = queryNorm
             0.42883322 = fieldWeight in 2585, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.544793 = idf(docFreq=453, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
         0.08372672 = weight(abstract_txt:clustering in 2585) [ClassicSimilarity], result of:
           0.08372672 = score(doc=2585,freq=3.0), product of:
             0.14209932 = queryWeight, product of:
               1.6401274 = boost
               6.220473 = idf(docFreq=230, maxDocs=42740)
               0.013928073 = queryNorm
             0.58921266 = fieldWeight in 2585, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.220473 = idf(docFreq=230, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
         0.051414996 = weight(abstract_txt:correlation in 2585) [ClassicSimilarity], result of:
           0.051414996 = score(doc=2585,freq=1.0), product of:
             0.14806403 = queryWeight, product of:
               1.6741962 = boost
               6.3496847 = idf(docFreq=202, maxDocs=42740)
               0.013928073 = queryNorm
             0.34724838 = fieldWeight in 2585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3496847 = idf(docFreq=202, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
         0.094248794 = weight(abstract_txt:cosine in 2585) [ClassicSimilarity], result of:
           0.094248794 = score(doc=2585,freq=1.0), product of:
             0.22177209 = queryWeight, product of:
               2.0489674 = boost
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.013928073 = queryNorm
             0.4249804 = fieldWeight in 2585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
         0.33769777 = weight(abstract_txt:measures in 2585) [ClassicSimilarity], result of:
           0.33769777 = score(doc=2585,freq=9.0), product of:
             0.3790388 = queryWeight, product of:
               5.011381 = boost
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.013928073 = queryNorm
             0.8909319 = fieldWeight in 2585, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2585)
       0.32 = coord(8/25)
   

4. Egghe, L.: On the relation between the association strength and other similarity measures (2010) 0.21

   0.20754872 = sum of:
     0.20754872 = product of:
       1.2971796 = sum of:
         0.2665758 = weight(abstract_txt:cosine in 599) [ClassicSimilarity], result of:
           0.2665758 = score(doc=599,freq=2.0), product of:
             0.22177209 = queryWeight, product of:
               2.0489674 = boost
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.013928073 = queryNorm
             1.202026 = fieldWeight in 599, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.77107 = idf(docFreq=48, maxDocs=42740)
               0.109375 = fieldNorm(doc=599)
         0.28223628 = weight(abstract_txt:jaccard in 599) [ClassicSimilarity], result of:
           0.28223628 = score(doc=599,freq=1.0), product of:
             0.29025403 = queryWeight, product of:
               2.3440707 = boost
               8.890302 = idf(docFreq=15, maxDocs=42740)
               0.013928073 = queryNorm
             0.97237676 = fieldWeight in 599, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.890302 = idf(docFreq=15, maxDocs=42740)
               0.109375 = fieldNorm(doc=599)
         0.3899398 = weight(abstract_txt:measures in 599) [ClassicSimilarity], result of:
           0.3899398 = score(doc=599,freq=3.0), product of:
             0.3790388 = queryWeight, product of:
               5.011381 = boost
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.013928073 = queryNorm
             1.0287596 = fieldWeight in 599, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.109375 = fieldNorm(doc=599)
         0.35842764 = weight(abstract_txt:association in 599) [ClassicSimilarity], result of:
           0.35842764 = score(doc=599,freq=2.0), product of:
             0.4101884 = queryWeight, product of:
               5.213235 = boost
               5.649175 = idf(docFreq=408, maxDocs=42740)
               0.013928073 = queryNorm
             0.8738122 = fieldWeight in 599, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.649175 = idf(docFreq=408, maxDocs=42740)
               0.109375 = fieldNorm(doc=599)
       0.16 = coord(4/25)
   

5. Efron, M.: Linear time series models for term weighting in information retrieval (2010) 0.21

   0.20540127 = sum of:
     0.20540127 = product of:
       0.85583866 = sum of:
         0.042379625 = weight(abstract_txt:behavior in 689) [ClassicSimilarity], result of:
           0.042379625 = score(doc=689,freq=1.0), product of:
             0.102618694 = queryWeight, product of:
               1.3937819 = boost
               5.286164 = idf(docFreq=587, maxDocs=42740)
               0.013928073 = queryNorm
             0.41298154 = fieldWeight in 689, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.286164 = idf(docFreq=587, maxDocs=42740)
               0.078125 = fieldNorm(doc=689)
         0.06619911 = weight(abstract_txt:corpus in 689) [ClassicSimilarity], result of:
           0.06619911 = score(doc=689,freq=1.0), product of:
             0.13815178 = queryWeight, product of:
               1.6171855 = boost
               6.1334615 = idf(docFreq=251, maxDocs=42740)
               0.013928073 = queryNorm
             0.47917667 = fieldWeight in 689, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.1334615 = idf(docFreq=251, maxDocs=42740)
               0.078125 = fieldNorm(doc=689)
         0.096001975 = weight(abstract_txt:terms in 689) [ClassicSimilarity], result of:
           0.096001975 = score(doc=689,freq=4.0), product of:
             0.15133354 = queryWeight, product of:
               2.6762037 = boost
               4.05999 = idf(docFreq=2003, maxDocs=42740)
               0.013928073 = queryNorm
             0.6343734 = fieldWeight in 689, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.05999 = idf(docFreq=2003, maxDocs=42740)
               0.078125 = fieldNorm(doc=689)
         0.24325901 = weight(abstract_txt:frequency in 689) [ClassicSimilarity], result of:
           0.24325901 = score(doc=689,freq=4.0), product of:
             0.2611105 = queryWeight, product of:
               3.1441898 = boost
               5.962447 = idf(docFreq=298, maxDocs=42740)
               0.013928073 = queryNorm
             0.9316324 = fieldWeight in 689, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.962447 = idf(docFreq=298, maxDocs=42740)
               0.078125 = fieldNorm(doc=689)
         0.1805814 = weight(abstract_txt:term in 689) [ClassicSimilarity], result of:
           0.1805814 = score(doc=689,freq=5.0), product of:
             0.21407226 = queryWeight, product of:
               3.1829643 = boost
               4.8287816 = idf(docFreq=928, maxDocs=42740)
               0.013928073 = queryNorm
             0.8435535 = fieldWeight in 689, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.8287816 = idf(docFreq=928, maxDocs=42740)
               0.078125 = fieldNorm(doc=689)
         0.22741751 = weight(abstract_txt:measures in 689) [ClassicSimilarity], result of:
           0.22741751 = score(doc=689,freq=2.0), product of:
             0.3790388 = queryWeight, product of:
               5.011381 = boost
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.013928073 = queryNorm
             0.59998477 = fieldWeight in 689, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.4304423 = idf(docFreq=508, maxDocs=42740)
               0.078125 = fieldNorm(doc=689)
       0.24 = coord(6/25)