-
Chu, H.: Factors affecting relevance judgment : a report from TREC Legal track (2011)
0.06
0.058798738 = product of:
0.117597476 = sum of:
0.117597476 = sum of:
0.0872787 = weight(_text_:2007 in 4540) [ClassicSimilarity], result of:
0.0872787 = score(doc=4540,freq=6.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.43196514 = fieldWeight in 4540, product of:
2.4494898 = tf(freq=6.0), with freq of:
6.0 = termFreq=6.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.0390625 = fieldNorm(doc=4540)
0.030318772 = weight(_text_:22 in 4540) [ClassicSimilarity], result of:
0.030318772 = score(doc=4540,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.19345059 = fieldWeight in 4540, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=4540)
0.5 = coord(1/2)
- Abstract
- Purpose - This study intends to identify factors that affect relevance judgment of retrieved information as part of the 2007 TREC Legal track interactive task. Design/methodology/approach - Data were gathered and analyzed from the participants of the 2007 TREC Legal track interactive task using a questionnaire which includes not only a list of 80 relevance factors identified in prior research, but also a space for expressing their thoughts on relevance judgment in the process. Findings - This study finds that topicality remains a primary criterion, out of various options, for determining relevance, while specificity of the search request, task, or retrieved results also helps greatly in relevance judgment. Research limitations/implications - Relevance research should focus on the topicality and specificity of what is being evaluated as well as conducted in real environments. Practical implications - If multiple relevance factors are presented to assessors, the total number in a list should be below ten to take account of the limited processing capacity of human beings' short-term memory. Otherwise, the assessors might either completely ignore or inadequately consider some of the relevance factors when making judgment decisions. Originality/value - This study presents a method for reducing the artificiality of relevance research design, an apparent limitation in many related studies. Specifically, relevance judgment was made in this research as part of the 2007 TREC Legal track interactive task rather than a study devised for the sake of it. The assessors also served as searchers so that their searching experience would facilitate their subsequent relevance judgments.
- Date
- 12. 7.2011 18:29:22
-
Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011)
0.04
0.040354572 = product of:
0.080709144 = sum of:
0.080709144 = sum of:
0.050390374 = weight(_text_:2007 in 4197) [ClassicSimilarity], result of:
0.050390374 = score(doc=4197,freq=2.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.24939516 = fieldWeight in 4197, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.0390625 = fieldNorm(doc=4197)
0.030318772 = weight(_text_:22 in 4197) [ClassicSimilarity], result of:
0.030318772 = score(doc=4197,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.19345059 = fieldWeight in 4197, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=4197)
0.5 = coord(1/2)
- Abstract
- The Initiative for the Evaluation of XML retrieval (INEX) provides a TREC-like platform for evaluating content-oriented XML retrieval systems. Since 2007, INEX has been using a set of precision-recall based metrics for its ad hoc tasks. The authors investigate the reliability and robustness of these focused retrieval measures, and of the INEX pooling method. They explore four specific questions: How reliable are the metrics when assessments are incomplete, or when query sets are small? What is the minimum pool/query-set size that can be used to reliably evaluate systems? Can the INEX collections be used to fairly evaluate "new" systems that did not participate in the pooling process? And, for a fixed amount of assessment effort, would this effort be better spent in thoroughly judging a few queries, or in judging many queries relatively superficially? The authors' findings validate properties of precision-recall-based metrics observed in document retrieval settings. Early precision measures are found to be more error-prone and less stable under incomplete judgments and small topic-set sizes. They also find that system rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced, and confirm that the INEX collections remain usable when evaluating nonparticipating systems. Finally, they observe that for a fixed amount of effort, judging shallow pools for many queries is better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. This result confirms the effectiveness of pooling methods.
- Date
- 22. 1.2011 14:20:56
-
Hierl, S.: Bezugsrahmen für die Evaluation von Information Retrieval Systemen mit Visualisierungskomponenten (2007)
0.03
0.031869676 = product of:
0.06373935 = sum of:
0.06373935 = product of:
0.1274787 = sum of:
0.1274787 = weight(_text_:2007 in 3040) [ClassicSimilarity], result of:
0.1274787 = score(doc=3040,freq=5.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.6309255 = fieldWeight in 3040, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.0625 = fieldNorm(doc=3040)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- B.I.T.online. 10(2007) H.2, S.xxx-xxx
- Year
- 2007
-
Beall, J.; Kafadar, K.: Measuring typographical errors' impact on retrieval in bibliographic databases (2007)
0.02
0.023902256 = product of:
0.047804512 = sum of:
0.047804512 = product of:
0.095609024 = sum of:
0.095609024 = weight(_text_:2007 in 261) [ClassicSimilarity], result of:
0.095609024 = score(doc=261,freq=5.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.47319412 = fieldWeight in 261, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.046875 = fieldNorm(doc=261)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Cataloging and classification quarterly. 44(2007) nos.3/4, S.197-211
- Year
- 2007
-
MacFarlane, A.: Evaluation of web search for the information practitioner (2007)
0.02
0.023902256 = product of:
0.047804512 = sum of:
0.047804512 = product of:
0.095609024 = sum of:
0.095609024 = weight(_text_:2007 in 817) [ClassicSimilarity], result of:
0.095609024 = score(doc=817,freq=5.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.47319412 = fieldWeight in 817, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.046875 = fieldNorm(doc=817)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Aslib proceedings. 59(2007) no.4/5, S.352-366
- Year
- 2007
-
Fuhr, N.; Niewelt, B.: ¬Ein Retrievaltest mit automatisch indexierten Dokumenten (1984)
0.02
0.021223139 = product of:
0.042446278 = sum of:
0.042446278 = product of:
0.084892556 = sum of:
0.084892556 = weight(_text_:22 in 262) [ClassicSimilarity], result of:
0.084892556 = score(doc=262,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.5416616 = fieldWeight in 262, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=262)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 20.10.2000 12:22:23
-
Tomaiuolo, N.G.; Parker, J.: Maximizing relevant retrieval : keyword and natural language searching (1998)
0.02
0.021223139 = product of:
0.042446278 = sum of:
0.042446278 = product of:
0.084892556 = sum of:
0.084892556 = weight(_text_:22 in 6418) [ClassicSimilarity], result of:
0.084892556 = score(doc=6418,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.5416616 = fieldWeight in 6418, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=6418)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Online. 22(1998) no.6, S.57-58
-
Voorhees, E.M.; Harman, D.: Overview of the Sixth Text REtrieval Conference (TREC-6) (2000)
0.02
0.021223139 = product of:
0.042446278 = sum of:
0.042446278 = product of:
0.084892556 = sum of:
0.084892556 = weight(_text_:22 in 6438) [ClassicSimilarity], result of:
0.084892556 = score(doc=6438,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.5416616 = fieldWeight in 6438, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=6438)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 11. 8.2001 16:22:19
-
Dalrymple, P.W.: Retrieval by reformulation in two library catalogs : toward a cognitive model of searching behavior (1990)
0.02
0.021223139 = product of:
0.042446278 = sum of:
0.042446278 = product of:
0.084892556 = sum of:
0.084892556 = weight(_text_:22 in 5089) [ClassicSimilarity], result of:
0.084892556 = score(doc=5089,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.5416616 = fieldWeight in 5089, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=5089)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 22. 7.2006 18:43:54
-
Ruthven, I.; Baillie, M.; Elsweiler, D.: ¬The relative effects of knowledge, interest and confidence in assessing relevance (2007)
0.02
0.019918546 = product of:
0.039837092 = sum of:
0.039837092 = product of:
0.079674184 = sum of:
0.079674184 = weight(_text_:2007 in 835) [ClassicSimilarity], result of:
0.079674184 = score(doc=835,freq=5.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.39432842 = fieldWeight in 835, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.0390625 = fieldNorm(doc=835)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of documentation. 63(2007) no.4, S.482-504
- Year
- 2007
-
Hemminger, B.M.; Saelim, B.; Sullivan, P.F.; Vision, T.J.: Comparison of full-text searching to metadata searching for genes in two biomedical literature cohorts (2007)
0.02
0.019918546 = product of:
0.039837092 = sum of:
0.039837092 = product of:
0.079674184 = sum of:
0.079674184 = weight(_text_:2007 in 1327) [ClassicSimilarity], result of:
0.079674184 = score(doc=1327,freq=5.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.39432842 = fieldWeight in 1327, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.0390625 = fieldNorm(doc=1327)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the American Society for Information Science and Technology. 58(2007) no.14, S.2341-2352
- Year
- 2007
-
Vechtomova, O.: Facet-based opinion retrieval from blogs (2010)
0.02
0.01763663 = product of:
0.03527326 = sum of:
0.03527326 = product of:
0.07054652 = sum of:
0.07054652 = weight(_text_:2007 in 4225) [ClassicSimilarity], result of:
0.07054652 = score(doc=4225,freq=2.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.34915322 = fieldWeight in 4225, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.0546875 = fieldNorm(doc=4225)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- The paper presents methods of retrieving blog posts containing opinions about an entity expressed in the query. The methods use a lexicon of subjective words and phrases compiled from manually and automatically developed resources. One of the methods uses the Kullback-Leibler divergence to weight subjective words occurring near query terms in documents, another uses proximity between the occurrences of query terms and subjective words in documents, and the third combines both factors. Methods of structuring queries into facets, facet expansion using Wikipedia, and a facet-based retrieval are also investigated in this work. The methods were evaluated using the TREC 2007 and 2008 Blog track topics, and proved to be highly effective.
-
Mansourian, Y.; Ford, N.: Web searchers' attributions of success and failure: an empirical study (2007)
0.02
0.015934838 = product of:
0.031869676 = sum of:
0.031869676 = product of:
0.06373935 = sum of:
0.06373935 = weight(_text_:2007 in 840) [ClassicSimilarity], result of:
0.06373935 = score(doc=840,freq=5.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.31546274 = fieldWeight in 840, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.03125 = fieldNorm(doc=840)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of documentation. 63(2007) no.5, S.659-679
- Year
- 2007
-
Mansourian, Y.; Ford, N.: Search persistence and failure on the web : a "bounded rationality" and "satisficing" analysis (2007)
0.02
0.015934838 = product of:
0.031869676 = sum of:
0.031869676 = product of:
0.06373935 = sum of:
0.06373935 = weight(_text_:2007 in 841) [ClassicSimilarity], result of:
0.06373935 = score(doc=841,freq=5.0), product of:
0.20205033 = queryWeight, product of:
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.044755515 = queryNorm
0.31546274 = fieldWeight in 841, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
4.514535 = idf(docFreq=1315, maxDocs=44218)
0.03125 = fieldNorm(doc=841)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of documentation. 63(2007) no.5, S.680-701
- Year
- 2007
-
Allan, J.; Callan, J.P.; Croft, W.B.; Ballesteros, L.; Broglio, J.; Xu, J.; Shu, H.: INQUERY at TREC-5 (1997)
0.02
0.015159386 = product of:
0.030318772 = sum of:
0.030318772 = product of:
0.060637545 = sum of:
0.060637545 = weight(_text_:22 in 3103) [ClassicSimilarity], result of:
0.060637545 = score(doc=3103,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.38690117 = fieldWeight in 3103, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=3103)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 27. 2.1999 20:55:22
-
Ng, K.B.; Loewenstern, D.; Basu, C.; Hirsh, H.; Kantor, P.B.: Data fusion of machine-learning methods for the TREC5 routing tak (and other work) (1997)
0.02
0.015159386 = product of:
0.030318772 = sum of:
0.030318772 = product of:
0.060637545 = sum of:
0.060637545 = weight(_text_:22 in 3107) [ClassicSimilarity], result of:
0.060637545 = score(doc=3107,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.38690117 = fieldWeight in 3107, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=3107)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 27. 2.1999 20:59:22
-
Saracevic, T.: On a method for studying the structure and nature of requests in information retrieval (1983)
0.02
0.015159386 = product of:
0.030318772 = sum of:
0.030318772 = product of:
0.060637545 = sum of:
0.060637545 = weight(_text_:22 in 2417) [ClassicSimilarity], result of:
0.060637545 = score(doc=2417,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.38690117 = fieldWeight in 2417, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=2417)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Pages
- S.22-25
-
Rijsbergen, C.J. van: ¬A test for the separation of relevant and non-relevant documents in experimental retrieval collections (1973)
0.01
0.012127508 = product of:
0.024255017 = sum of:
0.024255017 = product of:
0.048510034 = sum of:
0.048510034 = weight(_text_:22 in 5002) [ClassicSimilarity], result of:
0.048510034 = score(doc=5002,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.30952093 = fieldWeight in 5002, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=5002)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 19. 3.1996 11:22:12
-
Sanderson, M.: ¬The Reuters test collection (1996)
0.01
0.012127508 = product of:
0.024255017 = sum of:
0.024255017 = product of:
0.048510034 = sum of:
0.048510034 = weight(_text_:22 in 6971) [ClassicSimilarity], result of:
0.048510034 = score(doc=6971,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.30952093 = fieldWeight in 6971, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=6971)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Information retrieval: new systems and current research. Proceedings of the 16th Research Colloquium of the British Computer Society Information Retrieval Specialist Group, Drymen, Scotland, 22-23 Mar 94. Ed.: R. Leon
-
Lespinasse, K.: TREC: une conference pour l'evaluation des systemes de recherche d'information (1997)
0.01
0.012127508 = product of:
0.024255017 = sum of:
0.024255017 = product of:
0.048510034 = sum of:
0.048510034 = weight(_text_:22 in 744) [ClassicSimilarity], result of:
0.048510034 = score(doc=744,freq=2.0), product of:
0.15672618 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.044755515 = queryNorm
0.30952093 = fieldWeight in 744, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=744)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Date
- 1. 8.1996 22:01:00
