Search (45 results, page 1 of 3)

0.048897676 = product of:
  0.09779535 = sum of:
    0.09779535 = sum of:
      0.06741798 = weight(_text_:e.g in 1786) [ClassicSimilarity], result of:
        0.06741798 = score(doc=1786,freq=2.0), product of:
          0.23393378 = queryWeight, product of:
            5.2168427 = idf(docFreq=651, maxDocs=44218)
            0.044842023 = queryNorm
          0.28819257 = fieldWeight in 1786, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            5.2168427 = idf(docFreq=651, maxDocs=44218)
            0.0390625 = fieldNorm(doc=1786)
      0.030377375 = weight(_text_:22 in 1786) [ClassicSimilarity], result of:
        0.030377375 = score(doc=1786,freq=2.0), product of:
          0.15702912 = queryWeight, product of:
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.044842023 = queryNorm
          0.19345059 = fieldWeight in 1786, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.0390625 = fieldNorm(doc=1786)
  0.5 = coord(1/2)

Date

6. 4.2015 19:31:22

0.02359629 = product of:
  0.04719258 = sum of:
    0.04719258 = product of:
      0.09438516 = sum of:
        0.09438516 = weight(_text_:e.g in 7435) [ClassicSimilarity], result of:
          0.09438516 = score(doc=7435,freq=2.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.40346956 = fieldWeight in 7435, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.0546875 = fieldNorm(doc=7435)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Describes the comparative approach to system evaluation used in this research project which delegated the administartion of an online retrieval test to an experimental online catalogue to produce data for evaluating the effectiveness of a new subject access design. Describes the methods enlisted to sort out problem test administration, e.g. to identify out-of-scope queries, incomplete system administration, and suspect post-search questionnaire responses. Covers how w the researchers handled problem search administrations and what actions they would use to reduce or eliminate the occurrence of such administrations in future online retrieval tests that delegate control of retrieval tests to online systems

0.023354271 = product of:
  0.046708543 = sum of:
    0.046708543 = product of:
      0.093417086 = sum of:
        0.093417086 = weight(_text_:e.g in 840) [ClassicSimilarity], result of:
          0.093417086 = score(doc=840,freq=6.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.39933133 = fieldWeight in 840, product of:
              2.4494898 = tf(freq=6.0), with freq of:
                6.0 = termFreq=6.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.03125 = fieldNorm(doc=840)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Purpose - This paper reports the findings of a study designed to explore web searchers' perceptions of the causes of their search failure and success. In particular, it seeks to discover the extent to which the constructs locus of control and attribution theory might provide useful frameworks for understanding searchers' perceptions. Design/methodology/approach - A combination of inductive and deductive approaches were employed. Perceptions of failed and successful searches were derived from the inductive analysis of using open-ended qualitative interviews with a sample of 37 biologists at the University of Sheffield. These perceptions were classified into "internal" and "external" attributions, and the relationships between these categories and "successful" and "failed" searches were analysed deductively to test the extent to which they might be explainable using locus of control and attribution theory interpretive frameworks. Findings - All searchers were readily able to recall "successful" and "unsuccessful" searches. In a large majority of cases (82.4 per cent), they clearly attributed each search to either internal (e.g. ability or effort) or external (e.g. luck or information not being available) factors. The pattern of such relationships was analysed, and mapped onto those that would be predicted by locus of control and attribution theory. The authors conclude that the potential of these theoretical frameworks to illuminate one's understanding of web searching, and associated training, merits further systematic study. Research limitations/implications - The findings are based on a relatively small sample of academic and research staff in a particular subject area. Importantly, also, the study can at best provide a prima facie case for further systematic study since, although the patterns of attribution behaviour accord with those predictable by locus of control and attribution theory, data relating to the predictive elements of these theories (e.g. levels of confidence and achievement) were not available. This issue is discussed, and recommendations made for further work. Originality/value - The findings provide some empirical support for the notion that locus of control and attribution theory might - subject to the limitations noted above - be potentially useful theoretical frameworks for helping us better understand web-based information seeking. If so, they could have implications particularly for better understanding of searchers' motivations, and for the design and development of more effective search training programmes.

0.021264162 = product of:
  0.042528324 = sum of:
    0.042528324 = product of:
      0.08505665 = sum of:
        0.08505665 = weight(_text_:22 in 262) [ClassicSimilarity], result of:
          0.08505665 = score(doc=262,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.5416616 = fieldWeight in 262, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.109375 = fieldNorm(doc=262)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Date

20.10.2000 12:22:23

0.021264162 = product of:
  0.042528324 = sum of:
    0.042528324 = product of:
      0.08505665 = sum of:
        0.08505665 = weight(_text_:22 in 6418) [ClassicSimilarity], result of:
          0.08505665 = score(doc=6418,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.5416616 = fieldWeight in 6418, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.109375 = fieldNorm(doc=6418)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Source

Online. 22(1998) no.6, S.57-58

0.021264162 = product of:
  0.042528324 = sum of:
    0.042528324 = product of:
      0.08505665 = sum of:
        0.08505665 = weight(_text_:22 in 6438) [ClassicSimilarity], result of:
          0.08505665 = score(doc=6438,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.5416616 = fieldWeight in 6438, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.109375 = fieldNorm(doc=6438)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Date

11. 8.2001 16:22:19

0.021264162 = product of:
  0.042528324 = sum of:
    0.042528324 = product of:
      0.08505665 = sum of:
        0.08505665 = weight(_text_:22 in 5089) [ClassicSimilarity], result of:
          0.08505665 = score(doc=5089,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.5416616 = fieldWeight in 5089, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.109375 = fieldNorm(doc=5089)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Date

22. 7.2006 18:43:54

0.020225393 = product of:
  0.040450785 = sum of:
    0.040450785 = product of:
      0.08090157 = sum of:
        0.08090157 = weight(_text_:e.g in 3961) [ClassicSimilarity], result of:
          0.08090157 = score(doc=3961,freq=2.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.34583107 = fieldWeight in 3961, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.046875 = fieldNorm(doc=3961)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Modeling multidimensional relevance in information retrieval (IR) has attracted much attention in recent years. However, most existing studies are conducted through relatively small-scale user studies, which may not reflect a real-world and natural search scenario. In this article, we propose to study the multidimensional user relevance model (MURM) on large scale query logs, which record users' various search behaviors (e.g., query reformulations, clicks and dwelling time, etc.) in natural search settings. We advance an existing MURM model (including five dimensions: topicality, novelty, reliability, understandability, and scope) by providing two additional dimensions, that is, interest and habit. The two new dimensions represent personalized relevance judgment on retrieved documents. Further, for each dimension in the enriched MURM model, a set of computable features are formulated. By conducting extensive document ranking experiments on Bing's query logs and TREC session Track data, we systematically investigated the impact of each dimension on retrieval performance and gained a series of insightful findings which may bring benefits for the design of future IR systems.

0.016854495 = product of:
  0.03370899 = sum of:
    0.03370899 = product of:
      0.06741798 = sum of:
        0.06741798 = weight(_text_:e.g in 2462) [ClassicSimilarity], result of:
          0.06741798 = score(doc=2462,freq=2.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.28819257 = fieldWeight in 2462, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.0390625 = fieldNorm(doc=2462)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

The field of information retrieval (IR) traditionally addressed the problem of retrieving text documents from large collections by full text indexing of words. It has always been characterised by a strong focus on evaluation to compare the performance of alternative designs. the emergence into widespread use both of multimedia and of interactive user interfaces has extensive implications for this field and the evaluation methods on which it depends. discusses what we currently understand about those implications. The 'system' being measured must be expanded to include the human users, whose behaviour has a large effect on overall retrieval success, which now depends upon sessions of many retrieval cycles, rather than a single transaction. Multimedia raise issues not only of how users might specify a query in the same medium (e.g. sketch the kind of picture they want), but of cross-medium retrieval. Current explorations in IR evaluation show diversity along at least 2 dimensions. One is that between comprehensive models that have a place for every possible relevant factor, and lightweight methods. The other is that between highly standardised workbench tests avoiding human users vs. workplace studies

0.016854495 = product of:
  0.03370899 = sum of:
    0.03370899 = product of:
      0.06741798 = sum of:
        0.06741798 = weight(_text_:e.g in 3700) [ClassicSimilarity], result of:
          0.06741798 = score(doc=3700,freq=2.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.28819257 = fieldWeight in 3700, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.0390625 = fieldNorm(doc=3700)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Purpose This paper demonstrates how to apply traditional information retrieval evaluation methods based on standards from the Text REtrieval Conference (TREC) and web search evaluation to all types of modern library information systems including online public access catalogs, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources. Design/methodology/approach We apply conventional procedures from information retrieval evaluation to the library information system context considering the specific characteristics of modern library materials. Findings We introduce a framework consisting of five parts: (1) search queries, (2) search results, (3) assessors, (4) testing, and (5) data analysis. We show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs. printed monographs and what issues need to be considered for retrieval tests in the library context. Practical implications The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context. Originality/value Although a considerable amount of research has been done on information retrieval evaluation, and standards for conducting retrieval effectiveness studies do exist, to our knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century library information systems. We demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

0.016854495 = product of:
  0.03370899 = sum of:
    0.03370899 = product of:
      0.06741798 = sum of:
        0.06741798 = weight(_text_:e.g in 5049) [ClassicSimilarity], result of:
          0.06741798 = score(doc=5049,freq=2.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.28819257 = fieldWeight in 5049, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.0390625 = fieldNorm(doc=5049)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Information Retrieval (IR) develops complex systems, constituted of several components, which aim at returning and optimally ranking the most relevant documents in response to user queries. In this context, experimental evaluation plays a central role, since it allows for measuring IR systems effectiveness, increasing the understanding of their functioning, and better directing the efforts for improving them. Current evaluation methodologies are limited by two major factors: (i) IR systems are evaluated as "black boxes", since it is not possible to decompose the contributions of the different components, e.g., stop lists, stemmers, and IR models; (ii) given that it is not possible to predict the effectiveness of an IR system, both academia and industry need to explore huge numbers of systems, originated by large combinatorial compositions of their components, to understand how they perform and how these components interact together. We propose a Combinatorial visuaL Analytics system for Information Retrieval Evaluation (CLAIRE) which allows for exploring and making sense of the performances of a large amount of IR systems, in order to quickly and intuitively grasp which system configurations are preferred, what are the contributions of the different components and how these components interact together. The CLAIRE system is then validated against use cases based on several test collections using a wide set of systems, generated by a combinatorial composition of several off-the-shelf components, representing the most common denominator almost always present in English IR systems. In particular, we validate the findings enabled by CLAIRE with respect to consolidated deep statistical analyses and we show that the CLAIRE system allows the generation of new insights, which were not detectable with traditional approaches.

0.0151886875 = product of:
  0.030377375 = sum of:
    0.030377375 = product of:
      0.06075475 = sum of:
        0.06075475 = weight(_text_:22 in 3103) [ClassicSimilarity], result of:
          0.06075475 = score(doc=3103,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.38690117 = fieldWeight in 3103, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.078125 = fieldNorm(doc=3103)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Date

27. 2.1999 20:55:22

0.0151886875 = product of:
  0.030377375 = sum of:
    0.030377375 = product of:
      0.06075475 = sum of:
        0.06075475 = weight(_text_:22 in 3107) [ClassicSimilarity], result of:
          0.06075475 = score(doc=3107,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.38690117 = fieldWeight in 3107, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.078125 = fieldNorm(doc=3107)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Date

27. 2.1999 20:59:22

0.0151886875 = product of:
  0.030377375 = sum of:
    0.030377375 = product of:
      0.06075475 = sum of:
        0.06075475 = weight(_text_:22 in 2417) [ClassicSimilarity], result of:
          0.06075475 = score(doc=2417,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.38690117 = fieldWeight in 2417, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.078125 = fieldNorm(doc=2417)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Pages

S.22-25

0.014596419 = product of:
  0.029192839 = sum of:
    0.029192839 = product of:
      0.058385678 = sum of:
        0.058385678 = weight(_text_:e.g in 6299) [ClassicSimilarity], result of:
          0.058385678 = score(doc=6299,freq=6.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.24958208 = fieldWeight in 6299, product of:
              2.4494898 = tf(freq=6.0), with freq of:
                6.0 = termFreq=6.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.01953125 = fieldNorm(doc=6299)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Footnote

Rez. in: Machine translation review: 1999, no.10, S.26-27 (D. Lewis): "Cross Language Information Retrieval (CLIR) addresses the growing need to access large volumes of data across language boundaries. The typical requirement is for the user to input a free form query, usually a brief description of a topic, into a search or retrieval engine which returns a list, in ranked order, of documents or web pages that are relevant to the topic. The search engine matches the terms in the query to indexed terms, usually keywords previously derived from the target documents. Unlike monolingual information retrieval, CLIR requires query terms in one language to be matched to indexed terms in another. Matching can be done by bilingual dictionary lookup, full machine translation, or by applying statistical methods. A query's success is measured in terms of recall (how many potentially relevant target documents are found) and precision (what proportion of documents found are relevant). Issues in CLIR are how to translate query terms into index terms, how to eliminate alternative translations (e.g. to decide that French 'traitement' in a query means 'treatment' and not 'salary'), and how to rank or weight translation alternatives that are retained (e.g. how to order the French terms 'aventure', 'business', 'affaire', and 'liaison' as relevant translations of English 'affair'). Grefenstette provides a lucid and useful overview of the field and the problems. The volume brings together a number of experiments and projects in CLIR. Mark Davies (New Mexico State University) describes Recuerdo, a Spanish retrieval engine which reduces translation ambiguities by scanning indexes for parallel texts; it also uses either a bilingual dictionary or direct equivalents from a parallel corpus in order to compare results for queries on parallel texts. Lisa Ballesteros and Bruce Croft (University of Massachusetts) use a 'local feedback' technique which automatically enhances a query by adding extra terms to it both before and after translation; such terms can be derived from documents known to be relevant to the query.
The retrieved output from a query including the phrase 'big rockets' may be, for instance, a sentence containing 'giant rocket' which is semantically ranked above 'military ocket'. David Hull (Xerox Research Centre, Grenoble) describes an implementation of a weighted Boolean model for Spanish-English CLIR. Users construct Boolean-type queries, weighting each term in the query, which is then translated by an on-line dictionary before being applied to the database. Comparisons with the performance of unweighted free-form queries ('vector space' models) proved encouraging. Two contributions consider the evaluation of CLIR systems. In order to by-pass the time-consuming and expensive process of assembling a standard collection of documents and of user queries against which the performance of an CLIR system is manually assessed, Páriac Sheridan et al (ETH Zurich) propose a method based on retrieving 'seed documents'. This involves identifying a unique document in a database (the 'seed document') and, for a number of queries, measuring how fast it is retrieved. The authors have also assembled a large database of multilingual news documents for testing purposes. By storing the (fairly short) documents in a structured form tagged with descriptor codes (e.g. for topic, country and area), the test suite is easily expanded while remaining consistent for the purposes of testing. Douglas Ouard and Bonne Dorr (University of Maryland) describe an evaluation methodology which appears to apply LSI techniques in order to filter and rank incoming documents designed for testing CLIR systems. The volume provides the reader an excellent overview of several projects in CLIR. It is well supported with references and is intended as a secondary text for researchers and practitioners. It highlights the need for a good, general tutorial introduction to the field."

0.013483594 = product of:
  0.026967188 = sum of:
    0.026967188 = product of:
      0.053934377 = sum of:
        0.053934377 = weight(_text_:e.g in 3960) [ClassicSimilarity], result of:
          0.053934377 = score(doc=3960,freq=2.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.23055404 = fieldWeight in 3960, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.03125 = fieldNorm(doc=3960)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Purpose - This paper aims to discuss and test the claim that utilization of the personalization techniques can be valuable to improve the efficiency of collaborative information retrieval (CIR) systems. Design/methodology/approach - A new personalized CIR system, called PERCIRS, is presented based on the user profile similarity calculation (UPSC) formulas. To this aim, the paper proposes several UPSC formulas as well as two techniques to evaluate them. As the proposed CIR system is personalized, it could not be evaluated by Cranfield, like evaluation techniques (e.g. TREC). Hence, this paper proposes a new user-centric mechanism, which enables PERCIRS to be evaluated. This mechanism is generic and can be used to evaluate any other personalized IR system. Findings - The results show that among the proposed UPSC formulas in this paper, the (query-document)-graph based formula is the most effective. After integrating this formula into PERCIRS and comparing it with nine other IR systems, it is concluded that the results of the system are better than the other IR systems. In addition, the paper shows that the complexity of the system is less that the complexity of the other CIR systems. Research limitations/implications - This system asks the users to explicitly rank the returned documents, while explicit ranking is still not widespread enough. However it believes that the users should actively participate in the IR process in order to aptly satisfy their needs to information. Originality/value - The value of this paper lies in combining collaborative and personalized IR, as well as introducing a mechanism which enables the personalized IR system to be evaluated. The proposed evaluation mechanism is very valuable for developers of personalized IR systems. The paper also introduces some significant user profile similarity calculation formulas, and two techniques to evaluate them. These formulas can also be used to find the user's community in the social networks.

0.013483594 = product of:
  0.026967188 = sum of:
    0.026967188 = product of:
      0.053934377 = sum of:
        0.053934377 = weight(_text_:e.g in 5092) [ClassicSimilarity], result of:
          0.053934377 = score(doc=5092,freq=2.0), product of:
            0.23393378 = queryWeight, product of:
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.044842023 = queryNorm
            0.23055404 = fieldWeight in 5092, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              5.2168427 = idf(docFreq=651, maxDocs=44218)
              0.03125 = fieldNorm(doc=5092)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

While test collections provide the cornerstone for Cranfield-based evaluation of information retrieval (IR) systems, it has become practically infeasible to rely on traditional pooling techniques to construct test collections at the scale of today's massive document collections (e.g., ClueWeb12's 700M+ Webpages). This has motivated a flurry of studies proposing more cost-effective yet reliable IR evaluation methods. In this paper, we propose a new intelligent topic selection method which reduces the number of search topics (and thereby costly human relevance judgments) needed for reliable IR evaluation. To rigorously assess our method, we integrate previously disparate lines of research on intelligent topic selection and deep vs. shallow judging (i.e., whether it is more cost-effective to collect many relevance judgments for a few topics or a few judgments for many topics). While prior work on intelligent topic selection has never been evaluated against shallow judging baselines, prior work on deep vs. shallow judging has largely argued for shallowed judging, but assuming random topic selection. We argue that for evaluating any topic selection method, ultimately one must ask whether it is actually useful to select topics, or should one simply perform shallow judging over many topics? In seeking a rigorous answer to this over-arching question, we conduct a comprehensive investigation over a set of relevant factors never previously studied together: 1) method of topic selection; 2) the effect of topic familiarity on human judging speed; and 3) how different topic generation processes (requiring varying human effort) impact (i) budget utilization and (ii) the resultant quality of judgments. Experiments on NIST TREC Robust 2003 and Robust 2004 test collections show that not only can we reliably evaluate IR systems with fewer topics, but also that: 1) when topics are intelligently selected, deep judging is often more cost-effective than shallow judging in evaluation reliability; and 2) topic familiarity and topic generation costs greatly impact the evaluation cost vs. reliability trade-off. Our findings challenge conventional wisdom in showing that deep judging is often preferable to shallow judging when topics are selected intelligently.

0.01215095 = product of:
  0.0243019 = sum of:
    0.0243019 = product of:
      0.0486038 = sum of:
        0.0486038 = weight(_text_:22 in 5002) [ClassicSimilarity], result of:
          0.0486038 = score(doc=5002,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.30952093 = fieldWeight in 5002, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0625 = fieldNorm(doc=5002)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Date

19. 3.1996 11:22:12

0.01215095 = product of:
  0.0243019 = sum of:
    0.0243019 = product of:
      0.0486038 = sum of:
        0.0486038 = weight(_text_:22 in 6971) [ClassicSimilarity], result of:
          0.0486038 = score(doc=6971,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.30952093 = fieldWeight in 6971, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0625 = fieldNorm(doc=6971)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Source

Information retrieval: new systems and current research. Proceedings of the 16th Research Colloquium of the British Computer Society Information Retrieval Specialist Group, Drymen, Scotland, 22-23 Mar 94. Ed.: R. Leon

0.01215095 = product of:
  0.0243019 = sum of:
    0.0243019 = product of:
      0.0486038 = sum of:
        0.0486038 = weight(_text_:22 in 744) [ClassicSimilarity], result of:
          0.0486038 = score(doc=744,freq=2.0), product of:
            0.15702912 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044842023 = queryNorm
            0.30952093 = fieldWeight in 744, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0625 = fieldNorm(doc=744)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Date

1. 8.1996 22:01:00