Search (2 results, page 1 of 1)

0.013040888 = product of:
  0.06520444 = sum of:
    0.06520444 = weight(_text_:index in 6976) [ClassicSimilarity], result of:
      0.06520444 = score(doc=6976,freq=2.0), product of:
        0.2250935 = queryWeight, product of:
          4.369764 = idf(docFreq=1520, maxDocs=44218)
          0.051511593 = queryNorm
        0.28967714 = fieldWeight in 6976, product of:
          1.4142135 = tf(freq=2.0), with freq of:
            2.0 = termFreq=2.0
          4.369764 = idf(docFreq=1520, maxDocs=44218)
          0.046875 = fieldNorm(doc=6976)
  0.2 = coord(1/5)

Abstract

There are regularities in the statistical information provided by natural language terms about neighboring terms. We find that when phrase rank increases, moving from common to less common phrases, the value of the expected mutual information measure (EMIM) between the terms regularly decreases. Luhn's model suggests that midrange terms are the best index terms and relevance discriminators. We suggest reasons for this principle based on the empirical relationships shown here between the rank of terms within phrases and the average mutual information between terms, which we refer to as the Inverse Representation- EMIM principle. We also suggest an Inverse EMIM term weight for indexing or retrieval applications that is consistent with Luhn's distribution. An information theoretic interpretation of Zipf's Law is provided. Using the regularity noted here, we suggest that Zipf's Law is a consequence of the statistical dependencies that exist between terms, described here using information theoretic concepts.

0.013040888 = product of:
  0.06520444 = sum of:
    0.06520444 = weight(_text_:index in 924) [ClassicSimilarity], result of:
      0.06520444 = score(doc=924,freq=2.0), product of:
        0.2250935 = queryWeight, product of:
          4.369764 = idf(docFreq=1520, maxDocs=44218)
          0.051511593 = queryNorm
        0.28967714 = fieldWeight in 924, product of:
          1.4142135 = tf(freq=2.0), with freq of:
            2.0 = termFreq=2.0
          4.369764 = idf(docFreq=1520, maxDocs=44218)
          0.046875 = fieldNorm(doc=924)
  0.2 = coord(1/5)

Abstract

A thesaurus and an ontology provide a set of structured terms, phrases, and metadata, often in a hierarchical arrangement, that may be used to index, search, and mine documents. We describe the decisions that should be made when including a term, deciding whether a term should be subdivided into its subclasses, or determining which of more than one set of possible subclasses should be used. Based on retrospective measurements or estimates of future performance when using thesaurus terms in document ordering, decisions are made so as to maximize performance. These decisions may be used in the automatic construction of a thesaurus. The evaluation of an existing thesaurus is described, consistent with the decision criteria developed here. These kinds of user-focused decision-theoretic techniques may be applied to other hierarchical applications, such as faceted classification systems used in information architecture or the use of hierarchical terms in "breadcrumb navigation".