-
Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011)
0.02
0.017729571 = product of:
0.035459142 = sum of:
0.035459142 = sum of:
0.0042540217 = weight(_text_:s in 4197) [ClassicSimilarity], result of:
0.0042540217 = score(doc=4197,freq=4.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.08494043 = fieldWeight in 4197, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=4197)
0.03120512 = weight(_text_:22 in 4197) [ClassicSimilarity], result of:
0.03120512 = score(doc=4197,freq=2.0), product of:
0.16130796 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.046063907 = queryNorm
0.19345059 = fieldWeight in 4197, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=4197)
0.5 = coord(1/2)
- Date
- 22. 1.2011 14:20:56
- Source
- Journal of the American Society for Information Science and Technology. 62(2011) no.2, S.375-394
-
Chu, H.: Factors affecting relevance judgment : a report from TREC Legal track (2011)
0.02
0.017106583 = product of:
0.034213167 = sum of:
0.034213167 = sum of:
0.0030080476 = weight(_text_:s in 4540) [ClassicSimilarity], result of:
0.0030080476 = score(doc=4540,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 4540, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=4540)
0.03120512 = weight(_text_:22 in 4540) [ClassicSimilarity], result of:
0.03120512 = score(doc=4540,freq=2.0), product of:
0.16130796 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.046063907 = queryNorm
0.19345059 = fieldWeight in 4540, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=4540)
0.5 = coord(1/2)
- Date
- 12. 7.2011 18:29:22
- Source
- Journal of documentation. 67(2011) no.2, S.264-278
-
Wildemuth, B.; Freund, L.; Toms, E.G.: Untangling search task complexity and difficulty in the context of interactive information retrieval studies (2014)
0.02
0.017106583 = product of:
0.034213167 = sum of:
0.034213167 = sum of:
0.0030080476 = weight(_text_:s in 1786) [ClassicSimilarity], result of:
0.0030080476 = score(doc=1786,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 1786, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=1786)
0.03120512 = weight(_text_:22 in 1786) [ClassicSimilarity], result of:
0.03120512 = score(doc=1786,freq=2.0), product of:
0.16130796 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.046063907 = queryNorm
0.19345059 = fieldWeight in 1786, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=1786)
0.5 = coord(1/2)
- Date
- 6. 4.2015 19:31:22
- Source
- Journal of documentation. 70(2014) no.6, S.1118-1140
-
Ravana, S.D.; Taheri, M.S.; Rajagopal, P.: Document-based approach to improve the accuracy of pairwise comparison in evaluating information retrieval systems (2015)
0.02
0.017106583 = product of:
0.034213167 = sum of:
0.034213167 = sum of:
0.0030080476 = weight(_text_:s in 2587) [ClassicSimilarity], result of:
0.0030080476 = score(doc=2587,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 2587, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=2587)
0.03120512 = weight(_text_:22 in 2587) [ClassicSimilarity], result of:
0.03120512 = score(doc=2587,freq=2.0), product of:
0.16130796 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.046063907 = queryNorm
0.19345059 = fieldWeight in 2587, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=2587)
0.5 = coord(1/2)
- Date
- 20. 1.2015 18:30:22
- Source
- Aslib journal of information management. 67(2015) no.4, S.408-421
-
Rajagopal, P.; Ravana, S.D.; Koh, Y.S.; Balakrishnan, V.: Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment (2019)
0.02
0.017106583 = product of:
0.034213167 = sum of:
0.034213167 = sum of:
0.0030080476 = weight(_text_:s in 5287) [ClassicSimilarity], result of:
0.0030080476 = score(doc=5287,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 5287, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=5287)
0.03120512 = weight(_text_:22 in 5287) [ClassicSimilarity], result of:
0.03120512 = score(doc=5287,freq=2.0), product of:
0.16130796 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.046063907 = queryNorm
0.19345059 = fieldWeight in 5287, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0390625 = fieldNorm(doc=5287)
0.5 = coord(1/2)
- Date
- 20. 1.2015 18:30:22
- Source
- Aslib journal of information management. 71(2019) no.1, S.2-17
-
Järvelin, K.: Evaluation (2011)
0.00
0.0021056333 = product of:
0.0042112665 = sum of:
0.0042112665 = product of:
0.008422533 = sum of:
0.008422533 = weight(_text_:s in 548) [ClassicSimilarity], result of:
0.008422533 = score(doc=548,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.16817348 = fieldWeight in 548, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.109375 = fieldNorm(doc=548)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Pages
- S.113-138
-
Bashir, S.; Rauber, A.: On the relationship between query characteristics and IR functions retrieval bias (2011)
0.00
0.0010635054 = product of:
0.0021270108 = sum of:
0.0021270108 = product of:
0.0042540217 = sum of:
0.0042540217 = weight(_text_:s in 4628) [ClassicSimilarity], result of:
0.0042540217 = score(doc=4628,freq=4.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.08494043 = fieldWeight in 4628, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=4628)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the American Society for Information Science and Technology. 62(2011) no.8, S.1515-1532
-
Vakkari, P.; Huuskonen, S.: Search effort degrades search output but improves task outcome (2012)
0.00
0.0010635054 = product of:
0.0021270108 = sum of:
0.0021270108 = product of:
0.0042540217 = sum of:
0.0042540217 = weight(_text_:s in 46) [ClassicSimilarity], result of:
0.0042540217 = score(doc=46,freq=4.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.08494043 = fieldWeight in 46, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=46)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the American Society for Information Science and Technology. 63(2012) no.4, S.657-670
-
Al-Maskari, A.; Sanderson, M.: ¬A review of factors influencing user satisfaction in information retrieval (2010)
0.00
0.0010528166 = product of:
0.0021056333 = sum of:
0.0021056333 = product of:
0.0042112665 = sum of:
0.0042112665 = weight(_text_:s in 3447) [ClassicSimilarity], result of:
0.0042112665 = score(doc=3447,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.08408674 = fieldWeight in 3447, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0546875 = fieldNorm(doc=3447)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the American Society for Information Science and Technology. 61(2010) no.5, S.859-868
-
Vechtomova, O.: Facet-based opinion retrieval from blogs (2010)
0.00
0.0010528166 = product of:
0.0021056333 = sum of:
0.0021056333 = product of:
0.0042112665 = sum of:
0.0042112665 = weight(_text_:s in 4225) [ClassicSimilarity], result of:
0.0042112665 = score(doc=4225,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.08408674 = fieldWeight in 4225, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0546875 = fieldNorm(doc=4225)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Information processing and management. 46(2010) no.1, S.71-88
-
Munkelt, J.: Erstellung einer DNB-Retrieval-Testkollektion (2018)
0.00
0.0010528166 = product of:
0.0021056333 = sum of:
0.0021056333 = product of:
0.0042112665 = sum of:
0.0042112665 = weight(_text_:s in 4310) [ClassicSimilarity], result of:
0.0042112665 = score(doc=4310,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.08408674 = fieldWeight in 4310, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0546875 = fieldNorm(doc=4310)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Pages
- II, 79 S
-
Schultz Jr., W.N.; Braddy, L.: ¬A librarian-centered study of perceptions of subject terms and controlled vocabulary (2017)
0.00
0.0010528166 = product of:
0.0021056333 = sum of:
0.0021056333 = product of:
0.0042112665 = sum of:
0.0042112665 = weight(_text_:s in 5156) [ClassicSimilarity], result of:
0.0042112665 = score(doc=5156,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.08408674 = fieldWeight in 5156, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0546875 = fieldNorm(doc=5156)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Cataloging and classification quarterly. 55(2017) no.7/8, S.456-466
-
Colace, F.; Santo, M. de; Greco, L.; Napoletano, P.: Improving relevance feedback-based query expansion by the use of a weighted word pairs approach (2015)
0.00
9.0241426E-4 = product of:
0.0018048285 = sum of:
0.0018048285 = product of:
0.003609657 = sum of:
0.003609657 = weight(_text_:s in 2263) [ClassicSimilarity], result of:
0.003609657 = score(doc=2263,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.072074346 = fieldWeight in 2263, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046875 = fieldNorm(doc=2263)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the Association for Information Science and Technology. 66(2015) no.11, S.2223-2234
-
Li, J.; Zhang, P.; Song, D.; Wu, Y.: Understanding an enriched multidimensional user relevance model by analyzing query logs (2017)
0.00
9.0241426E-4 = product of:
0.0018048285 = sum of:
0.0018048285 = product of:
0.003609657 = sum of:
0.003609657 = weight(_text_:s in 3961) [ClassicSimilarity], result of:
0.003609657 = score(doc=3961,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.072074346 = fieldWeight in 3961, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046875 = fieldNorm(doc=3961)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the Association for Information Science and Technology. 68(2017) no.12, S.2743-2754
-
Borlund, P.: ¬A study of the use of simulated work task situations in interactive information retrieval evaluations : a meta-evaluation (2016)
0.00
8.508044E-4 = product of:
0.0017016088 = sum of:
0.0017016088 = product of:
0.0034032175 = sum of:
0.0034032175 = weight(_text_:s in 2880) [ClassicSimilarity], result of:
0.0034032175 = score(doc=2880,freq=4.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.06795235 = fieldWeight in 2880, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.03125 = fieldNorm(doc=2880)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- Purpose - The purpose of this paper is to report a study of how the test instrument of a simulated work task situation is used in empirical evaluations of interactive information retrieval (IIR) and reported in the research literature. In particular, the author is interested to learn whether the requirements of how to employ simulated work task situations are followed, and whether these requirements call for further highlighting and refinement. Design/methodology/approach - In order to study how simulated work task situations are used, the research literature in question is identified. This is done partly via citation analysis by use of Web of Science®, and partly by systematic search of online repositories. On this basis, 67 individual publications were identified and they constitute the sample of analysis. Findings - The analysis reveals a need for clarifications of how to use simulated work task situations in IIR evaluations. In particular, with respect to the design and creation of realistic simulated work task situations. There is a lack of tailoring of the simulated work task situations to the test participants. Likewise, the requirement to include the test participants' personal information needs is neglected. Further, there is a need to add and emphasise a requirement to depict the used simulated work task situations when reporting the IIR studies. Research limitations/implications - Insight about the use of simulated work task situations has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies. Originality/value - Simulated work task situations are widely used in IIR studies, and the present study is the first comprehensive study of the intended and unintended use of this test instrument since its introduction in the late 1990's. The paper addresses the need to carefully design and tailor simulated work task situations to suit the test participants in order to obtain the intended authentic and realistic IIR under study.
- Source
- Journal of documentation. 72(2016) no.3, S.394-413
-
Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018)
0.00
8.508044E-4 = product of:
0.0017016088 = sum of:
0.0017016088 = product of:
0.0034032175 = sum of:
0.0034032175 = weight(_text_:s in 5092) [ClassicSimilarity], result of:
0.0034032175 = score(doc=5092,freq=4.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.06795235 = fieldWeight in 5092, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.03125 = fieldNorm(doc=5092)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- While test collections provide the cornerstone for Cranfield-based evaluation of information retrieval (IR) systems, it has become practically infeasible to rely on traditional pooling techniques to construct test collections at the scale of today's massive document collections (e.g., ClueWeb12's 700M+ Webpages). This has motivated a flurry of studies proposing more cost-effective yet reliable IR evaluation methods. In this paper, we propose a new intelligent topic selection method which reduces the number of search topics (and thereby costly human relevance judgments) needed for reliable IR evaluation. To rigorously assess our method, we integrate previously disparate lines of research on intelligent topic selection and deep vs. shallow judging (i.e., whether it is more cost-effective to collect many relevance judgments for a few topics or a few judgments for many topics). While prior work on intelligent topic selection has never been evaluated against shallow judging baselines, prior work on deep vs. shallow judging has largely argued for shallowed judging, but assuming random topic selection. We argue that for evaluating any topic selection method, ultimately one must ask whether it is actually useful to select topics, or should one simply perform shallow judging over many topics? In seeking a rigorous answer to this over-arching question, we conduct a comprehensive investigation over a set of relevant factors never previously studied together: 1) method of topic selection; 2) the effect of topic familiarity on human judging speed; and 3) how different topic generation processes (requiring varying human effort) impact (i) budget utilization and (ii) the resultant quality of judgments. Experiments on NIST TREC Robust 2003 and Robust 2004 test collections show that not only can we reliably evaluate IR systems with fewer topics, but also that: 1) when topics are intelligently selected, deep judging is often more cost-effective than shallow judging in evaluation reliability; and 2) topic familiarity and topic generation costs greatly impact the evaluation cost vs. reliability trade-off. Our findings challenge conventional wisdom in showing that deep judging is often preferable to shallow judging when topics are selected intelligently.
- Source
- Information processing and management. 54(2018) no.1, S.37-59
-
Thornley, C.V.; Johnson, A.C.; Smeaton, A.F.; Lee, H.: ¬The scholarly impact of TRECVid (2003-2009) (2011)
0.00
7.520119E-4 = product of:
0.0015040238 = sum of:
0.0015040238 = product of:
0.0030080476 = sum of:
0.0030080476 = weight(_text_:s in 4363) [ClassicSimilarity], result of:
0.0030080476 = score(doc=4363,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 4363, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=4363)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the American Society for Information Science and Technology. 62(2011) no.4, S.613-627
-
Kelly, D.; Sugimoto, C.R.: ¬A systematic review of interactive information retrieval evaluation studies, 1967-2006 (2013)
0.00
7.520119E-4 = product of:
0.0015040238 = sum of:
0.0015040238 = product of:
0.0030080476 = sum of:
0.0030080476 = weight(_text_:s in 684) [ClassicSimilarity], result of:
0.0030080476 = score(doc=684,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 684, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=684)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the American Society for Information Science and Technology. 64(2013) no.4, S.745-770
-
Lu, K.; Kipp, M.E.I.: Understanding the retrieval effectiveness of collaborative tags and author keywords in different retrieval environments : an experimental study on medical collections (2014)
0.00
7.520119E-4 = product of:
0.0015040238 = sum of:
0.0015040238 = product of:
0.0030080476 = sum of:
0.0030080476 = weight(_text_:s in 1215) [ClassicSimilarity], result of:
0.0030080476 = score(doc=1215,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 1215, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=1215)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of the Association for Information Science and Technology. 65(2014) no.3, S.483-500
-
Ruthven, I.: Relevance behaviour in TREC (2014)
0.00
7.520119E-4 = product of:
0.0015040238 = sum of:
0.0015040238 = product of:
0.0030080476 = sum of:
0.0030080476 = weight(_text_:s in 1785) [ClassicSimilarity], result of:
0.0030080476 = score(doc=1785,freq=2.0), product of:
0.05008241 = queryWeight, product of:
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.046063907 = queryNorm
0.060061958 = fieldWeight in 1785, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
1.0872376 = idf(docFreq=40523, maxDocs=44218)
0.0390625 = fieldNorm(doc=1785)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Source
- Journal of documentation. 70(2014) no.6, S.1098-1117
