Robertson, S.: Understanding inverse document frequency : on theoretical arguments for IDF (2004)
0.00
0.004743375 = product of:
0.033203624 = sum of:
0.012233062 = weight(_text_:information in 4421) [ClassicSimilarity], result of:
0.012233062 = score(doc=4421,freq=6.0), product of:
0.052020688 = queryWeight, product of:
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.029633347 = queryNorm
0.23515764 = fieldWeight in 4421, product of:
2.4494898 = tf(freq=6.0), with freq of:
6.0 = termFreq=6.0
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.0546875 = fieldNorm(doc=4421)
0.020970564 = weight(_text_:retrieval in 4421) [ClassicSimilarity], result of:
0.020970564 = score(doc=4421,freq=2.0), product of:
0.08963835 = queryWeight, product of:
3.024915 = idf(docFreq=5836, maxDocs=44218)
0.029633347 = queryNorm
0.23394634 = fieldWeight in 4421, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.024915 = idf(docFreq=5836, maxDocs=44218)
0.0546875 = fieldNorm(doc=4421)
0.14285715 = coord(2/14)
- Abstract
- The term-weighting function known as IDF was proposed in 1972, and has since been extremely widely used, usually as part of a TF*IDF function. It is often described as a heuristic, and many papers have been written (some based on Shannon's Information Theory) seeking to establish some theoretical basis for it. Some of these attempts are reviewed, and it is shown that the Information Theory approaches are problematic, but that there are good theoretical justifications of both IDF and TF*IDF in the traditional probabilistic model of information retrieval.
Bodoff, D.; Robertson, S.: ¬A new unified probabilistic model (2004)
0.00
0.003790876 = product of:
0.02653613 = sum of:
0.00856136 = weight(_text_:information in 2129) [ClassicSimilarity], result of:
0.00856136 = score(doc=2129,freq=4.0), product of:
0.052020688 = queryWeight, product of:
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.029633347 = queryNorm
0.16457605 = fieldWeight in 2129, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.046875 = fieldNorm(doc=2129)
0.01797477 = weight(_text_:retrieval in 2129) [ClassicSimilarity], result of:
0.01797477 = score(doc=2129,freq=2.0), product of:
0.08963835 = queryWeight, product of:
3.024915 = idf(docFreq=5836, maxDocs=44218)
0.029633347 = queryNorm
0.20052543 = fieldWeight in 2129, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.024915 = idf(docFreq=5836, maxDocs=44218)
0.046875 = fieldNorm(doc=2129)
0.14285715 = coord(2/14)
- Abstract
- This paper proposes a new unified probabilistic model. Two previous models, Robertson et al.'s "Model 0" and "Model 3," each have strengths and weaknesses. The strength of Model 0 not found in Model 3, is that it does not require relevance data about the particular document or query, and, related to that, its probability estimates are straightforward. The strength of Model 3 not found in Model 0 is that it can utilize feedback information about the particular document and query in question. In this paper we introduce a new unified probabilistic model that combines these strengths: the expression of its probabilities is straightforward, it does not require that data must be available for the particular document or query in question, but it can utilize such specific data if it is available. The model is one way to resolve the difficulty of combining two marginal views in probabilistic retrieval.
- Source
- Journal of the American Society for Information Science and Technology. 55(2004) no.6, S.471-487
