-
Suominen, O.; Koskenniemi, I.: Annif Analyzer Shootout : comparing text lemmatization methods for automated subject indexing (2022)
0.06
0.05560436 = product of:
0.11120872 = sum of:
0.11120872 = product of:
0.22241744 = sum of:
0.22241744 = weight(_text_:toolkit in 658) [ClassicSimilarity], result of:
0.22241744 = score(doc=658,freq=4.0), product of:
0.3736465 = queryWeight, product of:
7.61935 = idf(docFreq=58, maxDocs=44218)
0.049039155 = queryNorm
0.5952617 = fieldWeight in 658, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
7.61935 = idf(docFreq=58, maxDocs=44218)
0.0390625 = fieldNorm(doc=658)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- Automated text classification is an important function for many AI systems relevant to libraries, including automated subject indexing and classification. When implemented using the traditional natural language processing (NLP) paradigm, one key part of the process is the normalization of words using stemming or lemmatization, which reduces the amount of linguistic variation and often improves the quality of classification. In this paper, we compare the output of seven different text lemmatization algorithms as well as two baseline methods. We measure how the choice of method affects the quality of text classification using example corpora in three languages. The experiments have been performed using the open source Annif toolkit for automated subject indexing and classification, but should generalize also to other NLP toolkits and similar text classification tasks. The results show that lemmatization methods in most cases outperform baseline methods in text classification particularly for Finnish and Swedish text, but not English, where baseline methods are most effective. The differences between lemmatization methods are quite small. The systematic comparison will help optimize text classification pipelines and inform the further development of the Annif toolkit to incorporate a wider choice of normalization methods.
-
Lowe, D.B.; Dollinger, I.; Koster, T.; Herbert, B.E.: Text mining for type of research classification (2021)
0.05
0.047181867 = product of:
0.094363734 = sum of:
0.094363734 = product of:
0.18872747 = sum of:
0.18872747 = weight(_text_:toolkit in 720) [ClassicSimilarity], result of:
0.18872747 = score(doc=720,freq=2.0), product of:
0.3736465 = queryWeight, product of:
7.61935 = idf(docFreq=58, maxDocs=44218)
0.049039155 = queryNorm
0.5050963 = fieldWeight in 720, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
7.61935 = idf(docFreq=58, maxDocs=44218)
0.046875 = fieldNorm(doc=720)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- This project brought together undergraduate students in Computer Science with librarians to mine abstracts of articles from the Texas A&M University Libraries' institutional repository, OAKTrust, in order to probe the creation of new metadata to improve discovery and use. The mining operation task consisted simply of classifying the articles into two categories of research type: basic research ("for understanding," "curiosity-based," or "knowledge-based") and applied research ("use-based"). These categories are fundamental especially for funders but are also important to researchers. The mining-to-classification steps took several iterations, but ultimately, we achieved good results with the toolkit BERT (Bidirectional Encoder Representations from Transformers). The project and its workflows represent a preview of what may lie ahead in the future of crafting metadata using text mining techniques to enhance discoverability.
-
Voorhees, E.M.: Implementing agglomerative hierarchic clustering algorithms for use in document retrieval (1986)
0.03
0.026576513 = product of:
0.053153027 = sum of:
0.053153027 = product of:
0.10630605 = sum of:
0.10630605 = weight(_text_:22 in 402) [ClassicSimilarity], result of:
0.10630605 = score(doc=402,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.61904186 = fieldWeight in 402, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.125 = fieldNorm(doc=402)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Information processing and management. 22(1986) no.6, S.465-476
-
Fuhr, N.; Niewelt, B.: ¬Ein Retrievaltest mit automatisch indexierten Dokumenten (1984)
0.02
0.023254449 = product of:
0.046508897 = sum of:
0.046508897 = product of:
0.093017794 = sum of:
0.093017794 = weight(_text_:22 in 262) [ClassicSimilarity], result of:
0.093017794 = score(doc=262,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.5416616 = fieldWeight in 262, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=262)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 20.10.2000 12:22:23
-
Hlava, M.M.K.: Automatic indexing : comparing rule-based and statistics-based indexing systems (2005)
0.02
0.023254449 = product of:
0.046508897 = sum of:
0.046508897 = product of:
0.093017794 = sum of:
0.093017794 = weight(_text_:22 in 6265) [ClassicSimilarity], result of:
0.093017794 = score(doc=6265,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.5416616 = fieldWeight in 6265, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=6265)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Information outlook. 9(2005) no.8, S.22-23
-
Fuhr, N.: Ranking-Experimente mit gewichteter Indexierung (1986)
0.02
0.019932386 = product of:
0.03986477 = sum of:
0.03986477 = product of:
0.07972954 = sum of:
0.07972954 = weight(_text_:22 in 58) [ClassicSimilarity], result of:
0.07972954 = score(doc=58,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.46428138 = fieldWeight in 58, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.09375 = fieldNorm(doc=58)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 14. 6.2015 22:12:44
-
Hauer, M.: Automatische Indexierung (2000)
0.02
0.019932386 = product of:
0.03986477 = sum of:
0.03986477 = product of:
0.07972954 = sum of:
0.07972954 = weight(_text_:22 in 5887) [ClassicSimilarity], result of:
0.07972954 = score(doc=5887,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.46428138 = fieldWeight in 5887, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.09375 = fieldNorm(doc=5887)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Wissen in Aktion: Wege des Knowledge Managements. 22. Online-Tagung der DGI, Frankfurt am Main, 2.-4.5.2000. Proceedings. Hrsg.: R. Schmidt
-
Fuhr, N.: Rankingexperimente mit gewichteter Indexierung (1986)
0.02
0.019932386 = product of:
0.03986477 = sum of:
0.03986477 = product of:
0.07972954 = sum of:
0.07972954 = weight(_text_:22 in 2051) [ClassicSimilarity], result of:
0.07972954 = score(doc=2051,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.46428138 = fieldWeight in 2051, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.09375 = fieldNorm(doc=2051)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 14. 6.2015 22:12:56
-
Hauer, M.: Tiefenindexierung im Bibliothekskatalog : 17 Jahre intelligentCAPTURE (2019)
0.02
0.019932386 = product of:
0.03986477 = sum of:
0.03986477 = product of:
0.07972954 = sum of:
0.07972954 = weight(_text_:22 in 5629) [ClassicSimilarity], result of:
0.07972954 = score(doc=5629,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.46428138 = fieldWeight in 5629, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.09375 = fieldNorm(doc=5629)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- B.I.T.online. 22(2019) H.2, S.163-166
-
Biebricher, N.; Fuhr, N.; Lustig, G.; Schwantner, M.; Knorz, G.: ¬The automatic indexing system AIR/PHYS : from research to application (1988)
0.02
0.01661032 = product of:
0.03322064 = sum of:
0.03322064 = product of:
0.06644128 = sum of:
0.06644128 = weight(_text_:22 in 1952) [ClassicSimilarity], result of:
0.06644128 = score(doc=1952,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.38690117 = fieldWeight in 1952, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=1952)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 16. 8.1998 12:51:22
-
Kutschekmanesch, S.; Lutes, B.; Moelle, K.; Thiel, U.; Tzeras, K.: Automated multilingual indexing : a synthesis of rule-based and thesaurus-based methods (1998)
0.02
0.01661032 = product of:
0.03322064 = sum of:
0.03322064 = product of:
0.06644128 = sum of:
0.06644128 = weight(_text_:22 in 4157) [ClassicSimilarity], result of:
0.06644128 = score(doc=4157,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.38690117 = fieldWeight in 4157, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=4157)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Information und Märkte: 50. Deutscher Dokumentartag 1998, Kongreß der Deutschen Gesellschaft für Dokumentation e.V. (DGD), Rheinische Friedrich-Wilhelms-Universität Bonn, 22.-24. September 1998. Hrsg. von Marlies Ockenfeld u. Gerhard J. Mantwill
-
Tsareva, P.V.: Algoritmy dlya raspoznavaniya pozitivnykh i negativnykh vkhozdenii deskriptorov v tekst i protsedura avtomaticheskoi klassifikatsii tekstov (1999)
0.02
0.01661032 = product of:
0.03322064 = sum of:
0.03322064 = product of:
0.06644128 = sum of:
0.06644128 = weight(_text_:22 in 374) [ClassicSimilarity], result of:
0.06644128 = score(doc=374,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.38690117 = fieldWeight in 374, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=374)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 1. 4.2002 10:22:41
-
Stankovic, R. et al.: Indexing of textual databases based on lexical resources : a case study for Serbian (2016)
0.02
0.01661032 = product of:
0.03322064 = sum of:
0.03322064 = product of:
0.06644128 = sum of:
0.06644128 = weight(_text_:22 in 2759) [ClassicSimilarity], result of:
0.06644128 = score(doc=2759,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.38690117 = fieldWeight in 2759, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=2759)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 1. 2.2016 18:25:22
-
Tsujii, J.-I.: Automatic acquisition of semantic collocation from corpora (1995)
0.01
0.013288257 = product of:
0.026576513 = sum of:
0.026576513 = product of:
0.053153027 = sum of:
0.053153027 = weight(_text_:22 in 4709) [ClassicSimilarity], result of:
0.053153027 = score(doc=4709,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.30952093 = fieldWeight in 4709, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=4709)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 31. 7.1996 9:22:19
-
Riloff, E.: ¬An empirical study of automated dictionary construction for information extraction in three domains (1996)
0.01
0.013288257 = product of:
0.026576513 = sum of:
0.026576513 = product of:
0.053153027 = sum of:
0.053153027 = weight(_text_:22 in 6752) [ClassicSimilarity], result of:
0.053153027 = score(doc=6752,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.30952093 = fieldWeight in 6752, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=6752)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 6. 3.1997 16:22:15
-
Lepsky, K.; Vorhauer, J.: Lingo - ein open source System für die Automatische Indexierung deutschsprachiger Dokumente (2006)
0.01
0.013288257 = product of:
0.026576513 = sum of:
0.026576513 = product of:
0.053153027 = sum of:
0.053153027 = weight(_text_:22 in 3581) [ClassicSimilarity], result of:
0.053153027 = score(doc=3581,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.30952093 = fieldWeight in 3581, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=3581)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 24. 3.2006 12:22:02
-
Probst, M.; Mittelbach, J.: Maschinelle Indexierung in der Sacherschließung wissenschaftlicher Bibliotheken (2006)
0.01
0.013288257 = product of:
0.026576513 = sum of:
0.026576513 = product of:
0.053153027 = sum of:
0.053153027 = weight(_text_:22 in 1755) [ClassicSimilarity], result of:
0.053153027 = score(doc=1755,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.30952093 = fieldWeight in 1755, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=1755)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 22. 3.2008 12:35:19
-
Glaesener, L.: Automatisches Indexieren einer informationswissenschaftlichen Datenbank mit Mehrwortgruppen (2012)
0.01
0.013288257 = product of:
0.026576513 = sum of:
0.026576513 = product of:
0.053153027 = sum of:
0.053153027 = weight(_text_:22 in 401) [ClassicSimilarity], result of:
0.053153027 = score(doc=401,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.30952093 = fieldWeight in 401, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=401)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 11. 9.2012 19:43:22
-
Hodges, P.R.: Keyword in title indexes : effectiveness of retrieval in computer searches (1983)
0.01
0.011627224 = product of:
0.023254449 = sum of:
0.023254449 = product of:
0.046508897 = sum of:
0.046508897 = weight(_text_:22 in 5001) [ClassicSimilarity], result of:
0.046508897 = score(doc=5001,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.2708308 = fieldWeight in 5001, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0546875 = fieldNorm(doc=5001)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 14. 3.1996 13:22:21
-
Bordoni, L.; Pazienza, M.T.: Documents automatic indexing in an environmental domain (1997)
0.01
0.011627224 = product of:
0.023254449 = sum of:
0.023254449 = product of:
0.046508897 = sum of:
0.046508897 = weight(_text_:22 in 530) [ClassicSimilarity], result of:
0.046508897 = score(doc=530,freq=2.0), product of:
0.17172676 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.049039155 = queryNorm
0.2708308 = fieldWeight in 530, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0546875 = fieldNorm(doc=530)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- International forum on information and documentation. 22(1997) no.1, S.17-28
