Dunning, T.: Statistical identification of language (1994)
0.00
0.003754243 = product of:
0.011262729 = sum of:
0.011262729 = weight(_text_:a in 3627) [ClassicSimilarity], result of:
0.011262729 = score(doc=3627,freq=16.0), product of:
0.05209492 = queryWeight, product of:
1.153047 = idf(docFreq=37942, maxDocs=44218)
0.045180224 = queryNorm
0.2161963 = fieldWeight in 3627, product of:
4.0 = tf(freq=16.0), with freq of:
16.0 = termFreq=16.0
1.153047 = idf(docFreq=37942, maxDocs=44218)
0.046875 = fieldNorm(doc=3627)
0.33333334 = coord(1/3)
- Abstract
- A statistically based program has been written which learns to distinguish between languages. The amount of training text that such a program needs is surprisingly small, and the amount of text needed to make an identification is also quite small. The program incorporates no linguistic presuppositions other than the assumption that text can be encoded as a string of bytes. Such a program can be used to determine which language small bits of text are in. It also shows a potential for what might be called 'statistical philology' in that it may be applied directly to phonetic transcriptions to help elucidate family trees among language dialects. A variant of this program has been shown to be useful as a quality control in biochemistry. In this application, genetic sequences are assumed to be expressions in a language peculiar to the organism from which the sequence is taken. Thus language identification becomes species identification.
