-
Ruthven, I.; Baillie, M.; Azzopardi, L.; Bierig, R.; Nicol, E.; Sweeney, S.; Yaciki, M.: Contextual factors affecting the utility of surrogates within exploratory search (2008)
0.00
7.280176E-4 = product of:
0.010920263 = sum of:
0.010920263 = product of:
0.021840526 = sum of:
0.021840526 = weight(_text_:information in 2042) [ClassicSimilarity], result of:
0.021840526 = score(doc=2042,freq=20.0), product of:
0.050870337 = queryWeight, product of:
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.028978055 = queryNorm
0.42933714 = fieldWeight in 2042, product of:
4.472136 = tf(freq=20.0), with freq of:
20.0 = termFreq=20.0
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.0546875 = fieldNorm(doc=2042)
0.5 = coord(1/2)
0.06666667 = coord(1/15)
- Abstract
- In this paper we investigate how information surrogates might be useful in exploratory search and what information it is useful for a surrogate to contain. By comparing assessments based on artificially created information surrogates, we investigate the effect of the source of information, the quality of an information source and the date of information upon the assessment process. We also investigate how varying levels of topical knowledge, assessor confidence and prior expectation affect the assessment of information surrogates. We show that both types of contextual information affect how the information surrogates are judged and what actions are performed as a result of the surrogates.
- Source
- Information processing and management. 44(2008) no.2, S.437-462
-
Baillie, M.; Azzopardi, L.; Ruthven, I.: Evaluating epistemic uncertainty under incomplete assessments (2008)
0.00
3.4178712E-4 = product of:
0.0051268064 = sum of:
0.0051268064 = product of:
0.010253613 = sum of:
0.010253613 = weight(_text_:information in 2065) [ClassicSimilarity], result of:
0.010253613 = score(doc=2065,freq=6.0), product of:
0.050870337 = queryWeight, product of:
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.028978055 = queryNorm
0.20156369 = fieldWeight in 2065, product of:
2.4494898 = tf(freq=6.0), with freq of:
6.0 = termFreq=6.0
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.046875 = fieldNorm(doc=2065)
0.5 = coord(1/2)
0.06666667 = coord(1/15)
- Abstract
- The thesis of this study is to propose an extended methodology for laboratory based Information Retrieval evaluation under incomplete relevance assessments. This new methodology aims to identify potential uncertainty during system comparison that may result from incompleteness. The adoption of this methodology is advantageous, because the detection of epistemic uncertainty - the amount of knowledge (or ignorance) we have about the estimate of a system's performance - during the evaluation process can guide and direct researchers when evaluating new systems over existing and future test collections. Across a series of experiments we demonstrate how this methodology can lead towards a finer grained analysis of systems. In particular, we show through experimentation how the current practice in Information Retrieval evaluation of using a measurement depth larger than the pooling depth increases uncertainty during system comparison.
- Source
- Information processing and management. 44(2008) no.2, S.811-837
-
Ruthven, I.; Baillie, M.; Elsweiler, D.: ¬The relative effects of knowledge, interest and confidence in assessing relevance (2007)
0.00
2.3255666E-4 = product of:
0.0034883497 = sum of:
0.0034883497 = product of:
0.0069766995 = sum of:
0.0069766995 = weight(_text_:information in 835) [ClassicSimilarity], result of:
0.0069766995 = score(doc=835,freq=4.0), product of:
0.050870337 = queryWeight, product of:
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.028978055 = queryNorm
0.13714671 = fieldWeight in 835, product of:
2.0 = tf(freq=4.0), with freq of:
4.0 = termFreq=4.0
1.7554779 = idf(docFreq=20772, maxDocs=44218)
0.0390625 = fieldNorm(doc=835)
0.5 = coord(1/2)
0.06666667 = coord(1/15)
- Abstract
- Purpose - The purpose of this paper is to examine how different aspects of an assessor's context, in particular their knowledge of a search topic, their interest in the search topic and their confidence in assessing relevance for a topic, affect the relevance judgements made and the assessor's ability to predict which documents they will assess as being relevant. Design/methodology/approach - The study was conducted as part of the Text REtrieval Conference (TREC) HARD track. Using a specially constructed questionnaire information was sought on TREC assessors' personal context and, using the TREC assessments gathered, the responses were correlated to the questionnaire questions and the final relevance decisions. Findings - This study found that each of the three factors (interest, knowledge and confidence) had an affect on how many documents were assessed as relevant and the balance between how many documents were marked as marginally or highly relevant. Also these factors are shown to affect an assessors' ability to predict what information they will finally mark as being relevant. Research limitations/implications - The major limitation is that the research is conducted within the TREC initiative. This means that we can report on results but cannot report on discussions with the assessors. The research implications are numerous but mainly on the effect of personal context on the outcomes of a user study. Practical implications - One major consequence is that we should take more account of how we construct search tasks for IIR evaluation to create tasks that are interesting and relevant to experimental subjects. Originality/value - Examining different search variables within one study to compare the relative effects on these variables on the search outcomes.
