-
Sanderson, M.: ¬The Reuters test collection (1996)
0.05
0.046974547 = product of:
0.14092363 = sum of:
0.14092363 = sum of:
0.08962556 = weight(_text_:group in 6971) [ClassicSimilarity], result of:
0.08962556 = score(doc=6971,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.40912446 = fieldWeight in 6971, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.0625 = fieldNorm(doc=6971)
0.05129807 = weight(_text_:22 in 6971) [ClassicSimilarity], result of:
0.05129807 = score(doc=6971,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.30952093 = fieldWeight in 6971, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.0625 = fieldNorm(doc=6971)
0.33333334 = coord(1/3)
- Source
- Information retrieval: new systems and current research. Proceedings of the 16th Research Colloquium of the British Computer Society Information Retrieval Specialist Group, Drymen, Scotland, 22-23 Mar 94. Ed.: R. Leon
-
Crestani, F.; Rijsbergen, C.J. van: Information retrieval by imaging (1996)
0.04
0.035230905 = product of:
0.105692714 = sum of:
0.105692714 = sum of:
0.06721917 = weight(_text_:group in 6967) [ClassicSimilarity], result of:
0.06721917 = score(doc=6967,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.30684334 = fieldWeight in 6967, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.046875 = fieldNorm(doc=6967)
0.03847355 = weight(_text_:22 in 6967) [ClassicSimilarity], result of:
0.03847355 = score(doc=6967,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.23214069 = fieldWeight in 6967, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.046875 = fieldNorm(doc=6967)
0.33333334 = coord(1/3)
- Source
- Information retrieval: new systems and current research. Proceedings of the 16th Research Colloquium of the British Computer Society Information Retrieval Specialist Group, Drymen, Scotland, 22-23 Mar 94. Ed.: R. Leon
-
Sanderson, M.; Ruthven, I.: Report on the Glasgow IR group (glair4) submission (1997)
0.02
0.02240639 = product of:
0.06721917 = sum of:
0.06721917 = product of:
0.13443834 = sum of:
0.13443834 = weight(_text_:group in 3088) [ClassicSimilarity], result of:
0.13443834 = score(doc=3088,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.6136867 = fieldWeight in 3088, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.09375 = fieldNorm(doc=3088)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
-
Zhang, X.: Collaborative relevance judgment : a group consensus method for evaluating user search performance (2002)
0.02
0.02240639 = product of:
0.06721917 = sum of:
0.06721917 = product of:
0.13443834 = sum of:
0.13443834 = weight(_text_:group in 250) [ClassicSimilarity], result of:
0.13443834 = score(doc=250,freq=8.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.6136867 = fieldWeight in 250, product of:
2.828427 = tf(freq=8.0), with freq of:
8.0 = termFreq=8.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.046875 = fieldNorm(doc=250)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- Relevance judgment has traditionally been considered a personal and subjective matter. A user's search and the search result are treated as an isolated event. To consider the collaborative nature of information retrieval (IR) in a group/organization or even societal context, this article proposes a method that measures relevance based on group/peer consensus. The method can be used in IR experiments. In this method, the relevance of a document is decided by group consensus, or more specifically, by the number of users (or experiment participants) who retrieve it for the same search question. The more users who retrieve it, the more relevant the document will be considered. A user's search performance can be measured by a relevance score based on this notion. The article reports the results of an experiment using this method to compare the search performance of different types of users. Related issues with the method and future directions are also discussed
-
Fuhr, N.; Niewelt, B.: ¬Ein Retrievaltest mit automatisch indexierten Dokumenten (1984)
0.01
0.014961937 = product of:
0.04488581 = sum of:
0.04488581 = product of:
0.08977162 = sum of:
0.08977162 = weight(_text_:22 in 262) [ClassicSimilarity], result of:
0.08977162 = score(doc=262,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.5416616 = fieldWeight in 262, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=262)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Date
- 20.10.2000 12:22:23
-
Tomaiuolo, N.G.; Parker, J.: Maximizing relevant retrieval : keyword and natural language searching (1998)
0.01
0.014961937 = product of:
0.04488581 = sum of:
0.04488581 = product of:
0.08977162 = sum of:
0.08977162 = weight(_text_:22 in 6418) [ClassicSimilarity], result of:
0.08977162 = score(doc=6418,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.5416616 = fieldWeight in 6418, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=6418)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Source
- Online. 22(1998) no.6, S.57-58
-
Voorhees, E.M.; Harman, D.: Overview of the Sixth Text REtrieval Conference (TREC-6) (2000)
0.01
0.014961937 = product of:
0.04488581 = sum of:
0.04488581 = product of:
0.08977162 = sum of:
0.08977162 = weight(_text_:22 in 6438) [ClassicSimilarity], result of:
0.08977162 = score(doc=6438,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.5416616 = fieldWeight in 6438, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=6438)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Date
- 11. 8.2001 16:22:19
-
Dalrymple, P.W.: Retrieval by reformulation in two library catalogs : toward a cognitive model of searching behavior (1990)
0.01
0.014961937 = product of:
0.04488581 = sum of:
0.04488581 = product of:
0.08977162 = sum of:
0.08977162 = weight(_text_:22 in 5089) [ClassicSimilarity], result of:
0.08977162 = score(doc=5089,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.5416616 = fieldWeight in 5089, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.109375 = fieldNorm(doc=5089)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Date
- 22. 7.2006 18:43:54
-
Janes, J.W.; McKinney, R.: Relevance judgements of actual users and secondary judges : a comparative study (1992)
0.01
0.013070395 = product of:
0.039211184 = sum of:
0.039211184 = product of:
0.07842237 = sum of:
0.07842237 = weight(_text_:group in 4276) [ClassicSimilarity], result of:
0.07842237 = score(doc=4276,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.35798392 = fieldWeight in 4276, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.0546875 = fieldNorm(doc=4276)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- Examines judgements of relevance of document representations to query statements made by people other than the the originators of the queries. A small group of graduate students in the School of Information and Library Studies and undergraduates of Michigan Univ. judges sets of documents that had been retrieved for and judged by real users for a previous study. The assessment of relevance, by the secondary judges, were analysed by themselves and in comparison with the users' assessments. The judges performed reasonably well but some important differences were identified. Secondary judges use the various fields of document records in different ways than users and have a higher threshold of relevance
-
Wu, C.-J.: Experiments on using the Dublin Core to reduce the retrieval error ratio (1998)
0.01
0.013070395 = product of:
0.039211184 = sum of:
0.039211184 = product of:
0.07842237 = sum of:
0.07842237 = weight(_text_:group in 5201) [ClassicSimilarity], result of:
0.07842237 = score(doc=5201,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.35798392 = fieldWeight in 5201, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.0546875 = fieldNorm(doc=5201)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- In order to test the power of metadata on information retrieval, an experiment was designed and conducted on a group of 7 graduate students using the Dublin Core as the cataloguing metadata. Results show that, on average, the retrieval error rate is only 2.9 per cent for the MES system (http://140.136.85.194), which utilizes the Dublin Core to describe the documents on the World Wide Web, in contrast to 20.7 per cent for the 7 famous search engines including HOTBOT, GAIS, LYCOS, EXCITE, INFOSEEK, YAHOO, and OCTOPUS. The very low error rate indicates that the users can use the information of the Dublin Core to decide whether to retrieve the documents or not
-
Meadows, C.J.: ¬A study of user performance and attitudes with information retrieval interfaces (1995)
0.01
0.011203195 = product of:
0.033609584 = sum of:
0.033609584 = product of:
0.06721917 = sum of:
0.06721917 = weight(_text_:group in 2674) [ClassicSimilarity], result of:
0.06721917 = score(doc=2674,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.30684334 = fieldWeight in 2674, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.046875 = fieldNorm(doc=2674)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- Reports on a project undertaken to compare the behaviour of 2 types of users with 2 types of information retrieval interfaces. The user types were search process specialists and subject matter domain specialists with no prior online database search experience. The interfaces were native DIALOG, which uses a procedural language, and OAK, a largely menu based, hence non procedural language interface communicating with DIALOG. 3 types of data were recorded: logs automatically recorded by computer moitoring of all searches, results of structured interviews with subjects at the time of the searches, and results of focus group discussions after all project tasks were completed. The type of user was determined by a combination of prior training, objective in searching, and subject domain knowledge. The results show that the type of interface does affect performance and users adapt their behaviour to interfaces differently. Different combinations of search experience and domain knowledge will lead to different behaviour in use of an information retrieval system. Different kinds of users can best be served with different kinds of interfaces
-
Shafique, M.; Chaudhry, A.S.: Intelligent agent-based online information retrieval (1995)
0.01
0.011203195 = product of:
0.033609584 = sum of:
0.033609584 = product of:
0.06721917 = sum of:
0.06721917 = weight(_text_:group in 3851) [ClassicSimilarity], result of:
0.06721917 = score(doc=3851,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.30684334 = fieldWeight in 3851, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.046875 = fieldNorm(doc=3851)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- Describes an intelligent agent based information retrieval model. The relevance matrix used by the intelligent agent consists of rows and columns; rows represent the documents and columns are used for keywords. Entries represent predetermined weights of keywords in documents. The search/query vector is constructed by the intelligent agent through explicit interaction with the user, using an interactive query refinement techniques. With manipulation of the relevance matrix against the search vector, the agent uses the manipulated information to filter the document representations and retrieve the most relevant documents, consequently improving the retrieval performance. Work is in progress on an experiment to compare the retrieval results from a conventional retrieval model and an intelligent agent based retrieval model. A test document collection on artificial intelligence has been selected as a sample. Retrieval tests are being carried out on a selected group of researchers using the 2 retrieval systems. Results will be compared to assess the retrieval performance using precision and recall matrices
-
Shaw, W.M.; Burgin, R.; Howell, P.: Performance standards and evaluations in IR test collections : vector-space and other retrieval models (1997)
0.01
0.011203195 = product of:
0.033609584 = sum of:
0.033609584 = product of:
0.06721917 = sum of:
0.06721917 = weight(_text_:group in 7259) [ClassicSimilarity], result of:
0.06721917 = score(doc=7259,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.30684334 = fieldWeight in 7259, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.046875 = fieldNorm(doc=7259)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- Computes low performance standards for each query and for the group of queries in 13 traditional and 4 TREC test collections. Predicted by the hypergeometric distribution, the standards represent the highest level of retrieval effectiveness attributable to chance. Compares operational levels of performance for vector-space, ad-hoc-feature-based, probabilistic, and other retrieval models to the standards. The effectiveness of these techniques in small, traditional test collections, can be explained by retrieving a few more relevant documents for most queries than expected by chance. The effectiveness of retrieval techniques in the larger TREC test collections can only be explained by retrieving many more relevant documents for most queries than expected by chance. The discrepancy between deviations form chance in traditional and TREC test collections is due to a decrease in performance standards for large test collections, not to an increase in operational performance. The next generation of information retrieval systems would be enhanced by abandoning uninformative performance summaries and focusing on effectiveness and improvements in effectiveness of individual queries
-
Cavanagh, A.K.: ¬A comparison of the retrieval performance of multi-disciplinary table-of-contents databases with conventional specialised databases (1997)
0.01
0.011203195 = product of:
0.033609584 = sum of:
0.033609584 = product of:
0.06721917 = sum of:
0.06721917 = weight(_text_:group in 770) [ClassicSimilarity], result of:
0.06721917 = score(doc=770,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.30684334 = fieldWeight in 770, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.046875 = fieldNorm(doc=770)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- In an endeavour to compare retrieval performance and periodical overlap in a biological field, the same topic was searched on 5 Table of Contents (ToC) databases and 3 specialised biological databases. Performance was assessed in terms of precision and recall. The ToC databases in general had higher precision in that most material found was relevant. They were less satisfactory in recall where some located fewer than 50% of identified high relevance articles. Subject specific databases had overall better recall but lower precision with many more false drops and items of low relevance occuring. These differences were associated with variations in indexing practice and policy and searching capabilities of the various databases. In a further comparison, it was found that the electronic databases, as a group, identified only 75% of the articles known from independent source to have been published in the field
-
Vakkari, P.; Sormunen, E.: ¬The influence of relevance levels an the effectiveness of interactive information retrieval (2004)
0.01
0.011203195 = product of:
0.033609584 = sum of:
0.033609584 = product of:
0.06721917 = sum of:
0.06721917 = weight(_text_:group in 2884) [ClassicSimilarity], result of:
0.06721917 = score(doc=2884,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.30684334 = fieldWeight in 2884, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.046875 = fieldNorm(doc=2884)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- In this paper, we focus an the effect of graded relevance an the results of interactive information retrieval (IR) experiments based an assigned search tasks in a test collection. A group of 26 subjects searched for four Text REtrieval Conference (TREC) topics using automatic and interactive query expansion based an relevance feedback. The TREC- and user-suggested pools of relevant documents were reassessed an a four-level relevance scale. The results show that the users could identify nearly all highly relevant documents and about half of the marginal ones. Users also selected a fair number of irrelevant documents for query expansion. The findings suggest that the effectiveness of query expansion is closely related to the searchers' success in retrieving and identifying highly relevant documents for feedback. The implications of the results an interpreting the findings of past experiments with liberal relevance thresholds are also discussed.
-
Allan, J.; Callan, J.P.; Croft, W.B.; Ballesteros, L.; Broglio, J.; Xu, J.; Shu, H.: INQUERY at TREC-5 (1997)
0.01
0.010687098 = product of:
0.032061294 = sum of:
0.032061294 = product of:
0.06412259 = sum of:
0.06412259 = weight(_text_:22 in 3103) [ClassicSimilarity], result of:
0.06412259 = score(doc=3103,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.38690117 = fieldWeight in 3103, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=3103)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Date
- 27. 2.1999 20:55:22
-
Ng, K.B.; Loewenstern, D.; Basu, C.; Hirsh, H.; Kantor, P.B.: Data fusion of machine-learning methods for the TREC5 routing tak (and other work) (1997)
0.01
0.010687098 = product of:
0.032061294 = sum of:
0.032061294 = product of:
0.06412259 = sum of:
0.06412259 = weight(_text_:22 in 3107) [ClassicSimilarity], result of:
0.06412259 = score(doc=3107,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.38690117 = fieldWeight in 3107, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=3107)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Date
- 27. 2.1999 20:59:22
-
Saracevic, T.: On a method for studying the structure and nature of requests in information retrieval (1983)
0.01
0.010687098 = product of:
0.032061294 = sum of:
0.032061294 = product of:
0.06412259 = sum of:
0.06412259 = weight(_text_:22 in 2417) [ClassicSimilarity], result of:
0.06412259 = score(doc=2417,freq=2.0), product of:
0.16573377 = queryWeight, product of:
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.047327764 = queryNorm
0.38690117 = fieldWeight in 2417, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
3.5018296 = idf(docFreq=3622, maxDocs=44218)
0.078125 = fieldNorm(doc=2417)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Pages
- S.22-25
-
Nelson, M.J.: ¬The effect of query characteristics on retrieval results in the TREC retrieval tests (1995)
0.01
0.009335997 = product of:
0.028007988 = sum of:
0.028007988 = product of:
0.056015976 = sum of:
0.056015976 = weight(_text_:group in 3608) [ClassicSimilarity], result of:
0.056015976 = score(doc=3608,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.2557028 = fieldWeight in 3608, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.0390625 = fieldNorm(doc=3608)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- There have been 3 Text Retrieval Conferences (TREC) organized by the National Insitute of Standards and Technology (NIST) over the last 3 years which have compared retrieval results on fairly large databases (at least 1 gigabyte). The queries (called topics), relevance judgements and databases were all provided by NIST. The main goal of the tests was to compare various retrieval algorithms using various measures of retrieval effectiveness. When Tague-Sutcliffe performed an analysis of variance on the average precision there is a large group of systems at the top of the ranking which are not significantly different. In addition the queries contribute more to the mean square the systems. To gather further insight into the results, this research investigates the variations in query properties as a partial explanation for the variation in retrieval scores. For each topic statement for the queries, the length (number of content words), langth of various parts and total number of relevant documents are correlated with the average precision
-
Borlund, P.: Experimental components for the evaluation of interactive information retrieval systems (2000)
0.01
0.009335997 = product of:
0.028007988 = sum of:
0.028007988 = product of:
0.056015976 = sum of:
0.056015976 = weight(_text_:group in 4549) [ClassicSimilarity], result of:
0.056015976 = score(doc=4549,freq=2.0), product of:
0.21906674 = queryWeight, product of:
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.047327764 = queryNorm
0.2557028 = fieldWeight in 4549, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.628715 = idf(docFreq=1173, maxDocs=44218)
0.0390625 = fieldNorm(doc=4549)
0.5 = coord(1/2)
0.33333334 = coord(1/3)
- Abstract
- This paper presents a set of basic components which constitutes the experimental setting intended for the evaluation of interactive information retrieval (IIR) systems, the aim of which is to facilitate evaluation of IIR systems in a way which is as close as possible to realistic IR processes. The experimental settings consists of 3 components: (1) the involvement of potential users as test persons; (2) the application of dynamic and individual information needs; and (3) the use of multidimensionsal and dynamic relevance judgements. Hidden under the information need component is the essential central sub-component, the simulated work task situation, the tool that triggers the (simulated) dynamic information need. This paper also reports on the empirical findings of the meta-evaluation of the application of this sub-component, the purpose of which is to discover whether the application of simulated work task situations to future evaluation of IIR systems can be recommended. Investigations are carried out to dertermine whether any search behavioural differences exist between test persons' treatment of their own real information needs versus simulated information needs. The hypothesis is that if no difference exist one can correctly substitute real information needs with simulated information needs through the application of simulated work task situations. The empirical results of the meta-evaluation provide positive evidence for the application of simulated work task situations to the evaluation of IIR systems. The results also indicate that tailoring work task situations to the group of test persons is important in motivating them. Furthermore, the results of the evaluation show that different versions of semantic openness of the simulated situations make no difference to the test persons' search treatment
