Document (#32960)

Author

Kontostathis, A.
Pottenger, W.M.

Title

¬A framework for understanding Latent Semantic Indexing (LSI) performance

Source

Information processing and management. 42(2006) no.1, S.56-73

Year

2006

Abstract

In this paper we present a theoretical model for understanding the performance of Latent Semantic Indexing (LSI) search and retrieval application. Many models for understanding LSI have been proposed. Ours is the first to study the values produced by LSI in the term by dimension vectors. The framework presented here is based on term co-occurrence data. We show a strong correlation between second-order term co-occurrence and the values produced by the Singular Value Decomposition (SVD) algorithm that forms the foundation for LSI. We also present a mathematical proof that the SVD algorithm encapsulates term co-occurrence information.

Footnote

Beitrag innerhalb eines thematischen Schwerpunktes "Formal Methods for Information Retrieval"

Object

Latent Semantic Indexing

Similar documents (content)

1. Li, D.; Kwong, C.-P.; Lee, D.L.: Unified linear subspace approach to semantic analysis (2009) 0.26

   0.2590294 = sum of:
     0.2590294 = product of:
       0.80946684 = sum of:
         0.095855415 = weight(abstract_txt:decomposition in 3321) [ClassicSimilarity], result of:
           0.095855415 = score(doc=3321,freq=1.0), product of:
             0.193826 = queryWeight, product of:
               1.4501121 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.016892187 = queryNorm
             0.4945436 = fieldWeight in 3321, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
         0.11045881 = weight(abstract_txt:singular in 3321) [ClassicSimilarity], result of:
           0.11045881 = score(doc=3321,freq=1.0), product of:
             0.21304329 = queryWeight, product of:
               1.5203006 = boost
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.016892187 = queryNorm
             0.5184806 = fieldWeight in 3321, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
         0.03184323 = weight(abstract_txt:indexing in 3321) [ClassicSimilarity], result of:
           0.03184323 = score(doc=3321,freq=1.0), product of:
             0.11713563 = queryWeight, product of:
               1.594245 = boost
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.016892187 = queryNorm
             0.27184922 = fieldWeight in 3321, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
         0.08490437 = weight(abstract_txt:semantic in 3321) [ClassicSimilarity], result of:
           0.08490437 = score(doc=3321,freq=6.0), product of:
             0.1239502 = queryWeight, product of:
               1.6399634 = boost
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.016892187 = queryNorm
             0.6849878 = fieldWeight in 3321, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
         0.054331172 = weight(abstract_txt:performance in 3321) [ClassicSimilarity], result of:
           0.054331172 = score(doc=3321,freq=2.0), product of:
             0.13274974 = queryWeight, product of:
               1.697178 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.016892187 = queryNorm
             0.40927517 = fieldWeight in 3321, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
         0.054174524 = weight(abstract_txt:understanding in 3321) [ClassicSimilarity], result of:
           0.054174524 = score(doc=3321,freq=1.0), product of:
             0.19109006 = queryWeight, product of:
               2.4938781 = boost
               4.5360413 = idf(docFreq=1287, maxDocs=44218)
               0.016892187 = queryNorm
             0.28350258 = fieldWeight in 3321, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5360413 = idf(docFreq=1287, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
         0.22954018 = weight(abstract_txt:latent in 3321) [ClassicSimilarity], result of:
           0.22954018 = score(doc=3321,freq=3.0), product of:
             0.30307004 = queryWeight, product of:
               2.564378 = boost
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.016892187 = queryNorm
             0.7573833 = fieldWeight in 3321, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
         0.14835909 = weight(abstract_txt:term in 3321) [ClassicSimilarity], result of:
           0.14835909 = score(doc=3321,freq=3.0), product of:
             0.28544557 = queryWeight, product of:
               3.5195506 = boost
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.016892187 = queryNorm
             0.51974565 = fieldWeight in 3321, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.0625 = fieldNorm(doc=3321)
       0.32 = coord(8/25)
   

2. Zhu, W.Z.; Allen, R.B.: Document clustering using the LSI subspace signature model (2013) 0.25

   0.25469536 = sum of:
     0.25469536 = product of:
       0.90962625 = sum of:
         0.11981926 = weight(abstract_txt:decomposition in 690) [ClassicSimilarity], result of:
           0.11981926 = score(doc=690,freq=1.0), product of:
             0.193826 = queryWeight, product of:
               1.4501121 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.016892187 = queryNorm
             0.6181795 = fieldWeight in 690, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.078125 = fieldNorm(doc=690)
         0.1380735 = weight(abstract_txt:singular in 690) [ClassicSimilarity], result of:
           0.1380735 = score(doc=690,freq=1.0), product of:
             0.21304329 = queryWeight, product of:
               1.5203006 = boost
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.016892187 = queryNorm
             0.64810073 = fieldWeight in 690, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.078125 = fieldNorm(doc=690)
         0.05629141 = weight(abstract_txt:indexing in 690) [ClassicSimilarity], result of:
           0.05629141 = score(doc=690,freq=2.0), product of:
             0.11713563 = queryWeight, product of:
               1.594245 = boost
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.016892187 = queryNorm
             0.48056605 = fieldWeight in 690, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.078125 = fieldNorm(doc=690)
         0.07504556 = weight(abstract_txt:semantic in 690) [ClassicSimilarity], result of:
           0.07504556 = score(doc=690,freq=3.0), product of:
             0.1239502 = queryWeight, product of:
               1.6399634 = boost
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.016892187 = queryNorm
             0.6054493 = fieldWeight in 690, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.078125 = fieldNorm(doc=690)
         0.048022427 = weight(abstract_txt:performance in 690) [ClassicSimilarity], result of:
           0.048022427 = score(doc=690,freq=1.0), product of:
             0.13274974 = queryWeight, product of:
               1.697178 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.016892187 = queryNorm
             0.3617516 = fieldWeight in 690, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.078125 = fieldNorm(doc=690)
         0.28692523 = weight(abstract_txt:latent in 690) [ClassicSimilarity], result of:
           0.28692523 = score(doc=690,freq=3.0), product of:
             0.30307004 = queryWeight, product of:
               2.564378 = boost
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.016892187 = queryNorm
             0.9467291 = fieldWeight in 690, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.078125 = fieldNorm(doc=690)
         0.18544886 = weight(abstract_txt:term in 690) [ClassicSimilarity], result of:
           0.18544886 = score(doc=690,freq=3.0), product of:
             0.28544557 = queryWeight, product of:
               3.5195506 = boost
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.016892187 = queryNorm
             0.64968204 = fieldWeight in 690, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.078125 = fieldNorm(doc=690)
       0.28 = coord(7/25)
   

3. Berry, M.W.; Dumais, S.T.; O'Brien, G.W.: Using linear algebra for intelligent information retrieval (1995) 0.22

   0.224679 = sum of:
     0.224679 = product of:
       0.80242497 = sum of:
         0.11499603 = weight(abstract_txt:vectors in 2206) [ClassicSimilarity], result of:
           0.11499603 = score(doc=2206,freq=1.0), product of:
             0.18858892 = queryWeight, product of:
               1.4303873 = boost
               7.805067 = idf(docFreq=48, maxDocs=44218)
               0.016892187 = queryNorm
             0.6097709 = fieldWeight in 2206, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.805067 = idf(docFreq=48, maxDocs=44218)
               0.078125 = fieldNorm(doc=2206)
         0.11981926 = weight(abstract_txt:decomposition in 2206) [ClassicSimilarity], result of:
           0.11981926 = score(doc=2206,freq=1.0), product of:
             0.193826 = queryWeight, product of:
               1.4501121 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.016892187 = queryNorm
             0.6181795 = fieldWeight in 2206, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.078125 = fieldNorm(doc=2206)
         0.19526543 = weight(abstract_txt:singular in 2206) [ClassicSimilarity], result of:
           0.19526543 = score(doc=2206,freq=2.0), product of:
             0.21304329 = queryWeight, product of:
               1.5203006 = boost
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.016892187 = queryNorm
             0.91655284 = fieldWeight in 2206, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.078125 = fieldNorm(doc=2206)
         0.05629141 = weight(abstract_txt:indexing in 2206) [ClassicSimilarity], result of:
           0.05629141 = score(doc=2206,freq=2.0), product of:
             0.11713563 = queryWeight, product of:
               1.594245 = boost
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.016892187 = queryNorm
             0.48056605 = fieldWeight in 2206, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.078125 = fieldNorm(doc=2206)
         0.04332758 = weight(abstract_txt:semantic in 2206) [ClassicSimilarity], result of:
           0.04332758 = score(doc=2206,freq=1.0), product of:
             0.1239502 = queryWeight, product of:
               1.6399634 = boost
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.016892187 = queryNorm
             0.34955636 = fieldWeight in 2206, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.078125 = fieldNorm(doc=2206)
         0.16565636 = weight(abstract_txt:latent in 2206) [ClassicSimilarity], result of:
           0.16565636 = score(doc=2206,freq=1.0), product of:
             0.30307004 = queryWeight, product of:
               2.564378 = boost
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.016892187 = queryNorm
             0.5465943 = fieldWeight in 2206, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.078125 = fieldNorm(doc=2206)
         0.107068956 = weight(abstract_txt:term in 2206) [ClassicSimilarity], result of:
           0.107068956 = score(doc=2206,freq=1.0), product of:
             0.28544557 = queryWeight, product of:
               3.5195506 = boost
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.016892187 = queryNorm
             0.37509412 = fieldWeight in 2206, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.078125 = fieldNorm(doc=2206)
       0.28 = coord(7/25)
   

4. Rishel, T.; Perkins, L.A.; Yenduri, S.; Zand, F.: Determining the context of text using augmented latent semantic indexing (2007) 0.20

   0.19774716 = sum of:
     0.19774716 = product of:
       0.8239465 = sum of:
         0.11981926 = weight(abstract_txt:decomposition in 1316) [ClassicSimilarity], result of:
           0.11981926 = score(doc=1316,freq=1.0), product of:
             0.193826 = queryWeight, product of:
               1.4501121 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.016892187 = queryNorm
             0.6181795 = fieldWeight in 1316, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.078125 = fieldNorm(doc=1316)
         0.1380735 = weight(abstract_txt:singular in 1316) [ClassicSimilarity], result of:
           0.1380735 = score(doc=1316,freq=1.0), product of:
             0.21304329 = queryWeight, product of:
               1.5203006 = boost
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.016892187 = queryNorm
             0.64810073 = fieldWeight in 1316, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.078125 = fieldNorm(doc=1316)
         0.096883416 = weight(abstract_txt:semantic in 1316) [ClassicSimilarity], result of:
           0.096883416 = score(doc=1316,freq=5.0), product of:
             0.1239502 = queryWeight, product of:
               1.6399634 = boost
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.016892187 = queryNorm
             0.78163177 = fieldWeight in 1316, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.078125 = fieldNorm(doc=1316)
         0.048022427 = weight(abstract_txt:performance in 1316) [ClassicSimilarity], result of:
           0.048022427 = score(doc=1316,freq=1.0), product of:
             0.13274974 = queryWeight, product of:
               1.697178 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.016892187 = queryNorm
             0.3617516 = fieldWeight in 1316, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.078125 = fieldNorm(doc=1316)
         0.08983519 = weight(abstract_txt:algorithm in 1316) [ClassicSimilarity], result of:
           0.08983519 = score(doc=1316,freq=1.0), product of:
             0.2015434 = queryWeight, product of:
               2.0911965 = boost
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.016892187 = queryNorm
             0.44573617 = fieldWeight in 1316, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.078125 = fieldNorm(doc=1316)
         0.33131272 = weight(abstract_txt:latent in 1316) [ClassicSimilarity], result of:
           0.33131272 = score(doc=1316,freq=4.0), product of:
             0.30307004 = queryWeight, product of:
               2.564378 = boost
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.016892187 = queryNorm
             1.0931886 = fieldWeight in 1316, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.996407 = idf(docFreq=109, maxDocs=44218)
               0.078125 = fieldNorm(doc=1316)
       0.24 = coord(6/25)
   

5. Deerwester, S.C.; Dumais, S.T.; Landauer, T.K.; Furnas, G.W.; Harshman, R.A.: Indexing by latent semantic analysis (1990) 0.20

   0.19753748 = sum of:
     0.19753748 = product of:
       0.705491 = sum of:
         0.16262895 = weight(abstract_txt:vectors in 2399) [ClassicSimilarity], result of:
           0.16262895 = score(doc=2399,freq=2.0), product of:
             0.18858892 = queryWeight, product of:
               1.4303873 = boost
               7.805067 = idf(docFreq=48, maxDocs=44218)
               0.016892187 = queryNorm
             0.86234623 = fieldWeight in 2399, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.805067 = idf(docFreq=48, maxDocs=44218)
               0.078125 = fieldNorm(doc=2399)
         0.11981926 = weight(abstract_txt:decomposition in 2399) [ClassicSimilarity], result of:
           0.11981926 = score(doc=2399,freq=1.0), product of:
             0.193826 = queryWeight, product of:
               1.4501121 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.016892187 = queryNorm
             0.6181795 = fieldWeight in 2399, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.078125 = fieldNorm(doc=2399)
         0.1380735 = weight(abstract_txt:singular in 2399) [ClassicSimilarity], result of:
           0.1380735 = score(doc=2399,freq=1.0), product of:
             0.21304329 = queryWeight, product of:
               1.5203006 = boost
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.016892187 = queryNorm
             0.64810073 = fieldWeight in 2399, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.29569 = idf(docFreq=29, maxDocs=44218)
               0.078125 = fieldNorm(doc=2399)
         0.039804034 = weight(abstract_txt:indexing in 2399) [ClassicSimilarity], result of:
           0.039804034 = score(doc=2399,freq=1.0), product of:
             0.11713563 = queryWeight, product of:
               1.594245 = boost
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.016892187 = queryNorm
             0.3398115 = fieldWeight in 2399, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.3495874 = idf(docFreq=1551, maxDocs=44218)
               0.078125 = fieldNorm(doc=2399)
         0.04332758 = weight(abstract_txt:semantic in 2399) [ClassicSimilarity], result of:
           0.04332758 = score(doc=2399,freq=1.0), product of:
             0.1239502 = queryWeight, product of:
               1.6399634 = boost
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.016892187 = queryNorm
             0.34955636 = fieldWeight in 2399, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.078125 = fieldNorm(doc=2399)
         0.09476867 = weight(abstract_txt:values in 2399) [ClassicSimilarity], result of:
           0.09476867 = score(doc=2399,freq=1.0), product of:
             0.20885624 = queryWeight, product of:
               2.128797 = boost
               5.808009 = idf(docFreq=360, maxDocs=44218)
               0.016892187 = queryNorm
             0.45375073 = fieldWeight in 2399, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.808009 = idf(docFreq=360, maxDocs=44218)
               0.078125 = fieldNorm(doc=2399)
         0.107068956 = weight(abstract_txt:term in 2399) [ClassicSimilarity], result of:
           0.107068956 = score(doc=2399,freq=1.0), product of:
             0.28544557 = queryWeight, product of:
               3.5195506 = boost
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.016892187 = queryNorm
             0.37509412 = fieldWeight in 2399, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.8012047 = idf(docFreq=987, maxDocs=44218)
               0.078125 = fieldNorm(doc=2399)
       0.28 = coord(7/25)