Document (#39687)

Author

Brychcín, T.
Konopík, M.

Title

HPS: High precision stemmer

Source

Information processing and management. 51(2015) no.1, S.68-91

Year

2015

Abstract

Research into unsupervised ways of stemming has resulted, in the past few years, in the development of methods that are reliable and perform well. Our approach further shifts the boundaries of the state of the art by providing more accurate stemming results. The idea of the approach consists in building a stemmer in two stages. In the first stage, a stemming algorithm based upon clustering, which exploits the lexical and semantic information of words, is used to prepare large-scale training data for the second-stage algorithm. The second-stage algorithm uses a maximum entropy classifier. The stemming-specific features help the classifier decide when and how to stem a particular word. In our research, we have pursued the goal of creating a multi-purpose stemming tool. Its design opens up possibilities of solving non-traditional tasks such as approximating lemmas or improving language modeling. However, we still aim at very good results in the traditional task of information retrieval. The conducted tests reveal exceptional performance in all the above mentioned tasks. Our stemming method is compared with three state-of-the-art statistical algorithms and one rule-based algorithm. We used corpora in the Czech, Slovak, Polish, Hungarian, Spanish and English languages. In the tests, our algorithm excels in stemming previously unseen words (the words that are not present in the training set). Moreover, it was discovered that our approach demands very little text data for training when compared with competing unsupervised algorithms.

Content

Vgl.: doi: 10.1016/j.ipm.2014.08.006.

Footnote

Vgl. auch: http://liks.fav.zcu.cz/HPS/.

Theme

Computerlinguistik

Similar documents (content)

1. Savoy, J.: Searching strategies for the Hungarian language (2008) 0.33

   0.32800543 = sum of:
     0.32800543 = product of:
       1.171448 = sum of:
         0.033690676 = weight(abstract_txt:when in 2037) [ClassicSimilarity], result of:
           0.033690676 = score(doc=2037,freq=3.0), product of:
             0.06001857 = queryWeight, product of:
               1.015438 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.014248162 = queryNorm
             0.56133753 = fieldWeight in 2037, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.078125 = fieldNorm(doc=2037)
         0.11295161 = weight(abstract_txt:hungarian in 2037) [ClassicSimilarity], result of:
           0.11295161 = score(doc=2037,freq=1.0), product of:
             0.15389973 = queryWeight, product of:
               1.1497793 = boost
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.014248162 = queryNorm
             0.7339299 = fieldWeight in 2037, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.078125 = fieldNorm(doc=2037)
         0.03248081 = weight(abstract_txt:very in 2037) [ClassicSimilarity], result of:
           0.03248081 = score(doc=2037,freq=1.0), product of:
             0.084476806 = queryWeight, product of:
               1.2047011 = boost
               4.921521 = idf(docFreq=875, maxDocs=44218)
               0.014248162 = queryNorm
             0.38449383 = fieldWeight in 2037, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.921521 = idf(docFreq=875, maxDocs=44218)
               0.078125 = fieldNorm(doc=2037)
         0.059562173 = weight(abstract_txt:compared in 2037) [ClassicSimilarity], result of:
           0.059562173 = score(doc=2037,freq=3.0), product of:
             0.087752506 = queryWeight, product of:
               1.2278359 = boost
               5.0160327 = idf(docFreq=796, maxDocs=44218)
               0.014248162 = queryNorm
             0.6787518 = fieldWeight in 2037, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.0160327 = idf(docFreq=796, maxDocs=44218)
               0.078125 = fieldNorm(doc=2037)
         0.037192058 = weight(abstract_txt:approach in 2037) [ClassicSimilarity], result of:
           0.037192058 = score(doc=2037,freq=3.0), product of:
             0.07338541 = queryWeight, product of:
               1.3751848 = boost
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.014248162 = queryNorm
             0.5068045 = fieldWeight in 2037, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.078125 = fieldNorm(doc=2037)
         0.21300408 = weight(abstract_txt:stemmer in 2037) [ClassicSimilarity], result of:
           0.21300408 = score(doc=2037,freq=1.0), product of:
             0.29596806 = queryWeight, product of:
               2.2549295 = boost
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.014248162 = queryNorm
             0.71968603 = fieldWeight in 2037, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.078125 = fieldNorm(doc=2037)
         0.68256664 = weight(abstract_txt:stemming in 2037) [ClassicSimilarity], result of:
           0.68256664 = score(doc=2037,freq=3.0), product of:
             0.67722327 = queryWeight, product of:
               6.38132 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.014248162 = queryNorm
             1.0078901 = fieldWeight in 2037, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.078125 = fieldNorm(doc=2037)
       0.28 = coord(7/25)
   

2. Fox, B.; Fox, C.J.: Efficient stemmer generation (2002) 0.23

   0.23465236 = sum of:
     0.23465236 = product of:
       1.4665773 = sum of:
         0.07094011 = weight(abstract_txt:algorithms in 2585) [ClassicSimilarity], result of:
           0.07094011 = score(doc=2585,freq=1.0), product of:
             0.11363064 = queryWeight, product of:
               1.397199 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.014248162 = queryNorm
             0.6243044 = fieldWeight in 2585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.109375 = fieldNorm(doc=2585)
         0.6668082 = weight(abstract_txt:stemmer in 2585) [ClassicSimilarity], result of:
           0.6668082 = score(doc=2585,freq=5.0), product of:
             0.29596806 = queryWeight, product of:
               2.2549295 = boost
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.014248162 = queryNorm
             2.2529736 = fieldWeight in 2585, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.109375 = fieldNorm(doc=2585)
         0.17711705 = weight(abstract_txt:algorithm in 2585) [ClassicSimilarity], result of:
           0.17711705 = score(doc=2585,freq=1.0), product of:
             0.28382748 = queryWeight, product of:
               3.4914658 = boost
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.014248162 = queryNorm
             0.62403065 = fieldWeight in 2585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.109375 = fieldNorm(doc=2585)
         0.55171204 = weight(abstract_txt:stemming in 2585) [ClassicSimilarity], result of:
           0.55171204 = score(doc=2585,freq=1.0), product of:
             0.67722327 = queryWeight, product of:
               6.38132 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.014248162 = queryNorm
             0.8146679 = fieldWeight in 2585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.109375 = fieldNorm(doc=2585)
       0.16 = coord(4/25)
   

3. Martins, A.L.; Souza, R.R.; Ribeiro de Mello, H.: ¬The use of noun phrases in information retrieval : proposing a mechanism for automatic classification (2014) 0.23

   0.22633283 = sum of:
     0.22633283 = product of:
       0.6287023 = sum of:
         0.024071766 = weight(abstract_txt:compared in 1441) [ClassicSimilarity], result of:
           0.024071766 = score(doc=1441,freq=1.0), product of:
             0.087752506 = queryWeight, product of:
               1.2278359 = boost
               5.0160327 = idf(docFreq=796, maxDocs=44218)
               0.014248162 = queryNorm
             0.27431428 = fieldWeight in 1441, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0160327 = idf(docFreq=796, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.024180718 = weight(abstract_txt:second in 1441) [ClassicSimilarity], result of:
           0.024180718 = score(doc=1441,freq=1.0), product of:
             0.08801709 = queryWeight, product of:
               1.2296855 = boost
               5.023589 = idf(docFreq=790, maxDocs=44218)
               0.014248162 = queryNorm
             0.27472752 = fieldWeight in 1441, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.023589 = idf(docFreq=790, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.026251739 = weight(abstract_txt:tasks in 1441) [ClassicSimilarity], result of:
           0.026251739 = score(doc=1441,freq=1.0), product of:
             0.09297359 = queryWeight, product of:
               1.263835 = boost
               5.1630983 = idf(docFreq=687, maxDocs=44218)
               0.014248162 = queryNorm
             0.28235695 = fieldWeight in 1441, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1630983 = idf(docFreq=687, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.015030992 = weight(abstract_txt:approach in 1441) [ClassicSimilarity], result of:
           0.015030992 = score(doc=1441,freq=1.0), product of:
             0.07338541 = queryWeight, product of:
               1.3751848 = boost
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.014248162 = queryNorm
             0.20482263 = fieldWeight in 1441, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.06052008 = weight(abstract_txt:tests in 1441) [ClassicSimilarity], result of:
           0.06052008 = score(doc=1441,freq=2.0), product of:
             0.12877865 = queryWeight, product of:
               1.4874161 = boost
               6.0764866 = idf(docFreq=275, maxDocs=44218)
               0.014248162 = queryNorm
             0.46995428 = fieldWeight in 1441, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.0764866 = idf(docFreq=275, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.038222145 = weight(abstract_txt:training in 1441) [ClassicSimilarity], result of:
           0.038222145 = score(doc=1441,freq=1.0), product of:
             0.13671872 = queryWeight, product of:
               1.8770256 = boost
               5.112096 = idf(docFreq=723, maxDocs=44218)
               0.014248162 = queryNorm
             0.27956775 = fieldWeight in 1441, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.112096 = idf(docFreq=723, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.07601024 = weight(abstract_txt:words in 1441) [ClassicSimilarity], result of:
           0.07601024 = score(doc=1441,freq=3.0), product of:
             0.14990829 = queryWeight, product of:
               1.9654818 = boost
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.014248162 = queryNorm
             0.507045 = fieldWeight in 1441, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.088558525 = weight(abstract_txt:algorithm in 1441) [ClassicSimilarity], result of:
           0.088558525 = score(doc=1441,freq=1.0), product of:
             0.28382748 = queryWeight, product of:
               3.4914658 = boost
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.014248162 = queryNorm
             0.31201532 = fieldWeight in 1441, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
         0.27585602 = weight(abstract_txt:stemming in 1441) [ClassicSimilarity], result of:
           0.27585602 = score(doc=1441,freq=1.0), product of:
             0.67722327 = queryWeight, product of:
               6.38132 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.014248162 = queryNorm
             0.40733394 = fieldWeight in 1441, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1441)
       0.36 = coord(9/25)
   

4. Frakes, W.B.: Stemming algorithms (1992) 0.20

   0.20195244 = sum of:
     0.20195244 = product of:
       1.682937 = sum of:
         0.08107441 = weight(abstract_txt:algorithms in 3503) [ClassicSimilarity], result of:
           0.08107441 = score(doc=3503,freq=1.0), product of:
             0.11363064 = queryWeight, product of:
               1.397199 = boost
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.014248162 = queryNorm
             0.7134907 = fieldWeight in 3503, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.707926 = idf(docFreq=398, maxDocs=44218)
               0.125 = fieldNorm(doc=3503)
         0.3408065 = weight(abstract_txt:stemmer in 3503) [ClassicSimilarity], result of:
           0.3408065 = score(doc=3503,freq=1.0), product of:
             0.29596806 = queryWeight, product of:
               2.2549295 = boost
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.014248162 = queryNorm
             1.1514976 = fieldWeight in 3503, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.125 = fieldNorm(doc=3503)
         1.2610561 = weight(abstract_txt:stemming in 3503) [ClassicSimilarity], result of:
           1.2610561 = score(doc=3503,freq=4.0), product of:
             0.67722327 = queryWeight, product of:
               6.38132 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.014248162 = queryNorm
             1.862098 = fieldWeight in 3503, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.125 = fieldNorm(doc=3503)
       0.12 = coord(3/25)
   

5. Kraaij, W.; Pohlmann, R.: Evaluation of a Dutch stemming algorithm (1995) 0.20

   0.20022003 = sum of:
     0.20022003 = product of:
       1.2513752 = sum of:
         0.08866015 = weight(abstract_txt:words in 5798) [ClassicSimilarity], result of:
           0.08866015 = score(doc=5798,freq=2.0), product of:
             0.14990829 = queryWeight, product of:
               1.9654818 = boost
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.014248162 = queryNorm
             0.5914293 = fieldWeight in 5798, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.078125 = fieldNorm(doc=5798)
         0.30123323 = weight(abstract_txt:stemmer in 5798) [ClassicSimilarity], result of:
           0.30123323 = score(doc=5798,freq=2.0), product of:
             0.29596806 = queryWeight, product of:
               2.2549295 = boost
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.014248162 = queryNorm
             1.0177897 = fieldWeight in 5798, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.078125 = fieldNorm(doc=5798)
         0.17891523 = weight(abstract_txt:algorithm in 5798) [ClassicSimilarity], result of:
           0.17891523 = score(doc=5798,freq=2.0), product of:
             0.28382748 = queryWeight, product of:
               3.4914658 = boost
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.014248162 = queryNorm
             0.63036615 = fieldWeight in 5798, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.078125 = fieldNorm(doc=5798)
         0.68256664 = weight(abstract_txt:stemming in 5798) [ClassicSimilarity], result of:
           0.68256664 = score(doc=5798,freq=3.0), product of:
             0.67722327 = queryWeight, product of:
               6.38132 = boost
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.014248162 = queryNorm
             1.0078901 = fieldWeight in 5798, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.448392 = idf(docFreq=69, maxDocs=44218)
               0.078125 = fieldNorm(doc=5798)
       0.16 = coord(4/25)