Search (42 results, page 1 of 3)

0.018646898 = product of:
  0.037293795 = sum of:
    0.037293795 = sum of:
      0.003647848 = weight(_text_:a in 328) [ClassicSimilarity], result of:
        0.003647848 = score(doc=328,freq=2.0), product of:
          0.04772363 = queryWeight, product of:
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.041389145 = queryNorm
          0.07643694 = fieldWeight in 328, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.046875 = fieldNorm(doc=328)
      0.033645947 = weight(_text_:22 in 328) [ClassicSimilarity], result of:
        0.033645947 = score(doc=328,freq=2.0), product of:
          0.14493774 = queryWeight, product of:
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.041389145 = queryNorm
          0.23214069 = fieldWeight in 328, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.046875 = fieldNorm(doc=328)
  0.5 = coord(1/2)

Date

22. 7.2012 19:25:54

Type

a

0.018578956 = product of:
  0.03715791 = sum of:
    0.03715791 = sum of:
      0.00911962 = weight(_text_:a in 4197) [ClassicSimilarity], result of:
        0.00911962 = score(doc=4197,freq=18.0), product of:
          0.04772363 = queryWeight, product of:
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.041389145 = queryNorm
          0.19109234 = fieldWeight in 4197, product of:
            4.2426405 = tf(freq=18.0), with freq of:
              18.0 = termFreq=18.0
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.0390625 = fieldNorm(doc=4197)
      0.028038291 = weight(_text_:22 in 4197) [ClassicSimilarity], result of:
        0.028038291 = score(doc=4197,freq=2.0), product of:
          0.14493774 = queryWeight, product of:
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.041389145 = queryNorm
          0.19345059 = fieldWeight in 4197, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.0390625 = fieldNorm(doc=4197)
  0.5 = coord(1/2)

Abstract

The Initiative for the Evaluation of XML retrieval (INEX) provides a TREC-like platform for evaluating content-oriented XML retrieval systems. Since 2007, INEX has been using a set of precision-recall based metrics for its ad hoc tasks. The authors investigate the reliability and robustness of these focused retrieval measures, and of the INEX pooling method. They explore four specific questions: How reliable are the metrics when assessments are incomplete, or when query sets are small? What is the minimum pool/query-set size that can be used to reliably evaluate systems? Can the INEX collections be used to fairly evaluate "new" systems that did not participate in the pooling process? And, for a fixed amount of assessment effort, would this effort be better spent in thoroughly judging a few queries, or in judging many queries relatively superficially? The authors' findings validate properties of precision-recall-based metrics observed in document retrieval settings. Early precision measures are found to be more error-prone and less stable under incomplete judgments and small topic-set sizes. They also find that system rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced, and confirm that the INEX collections remain usable when evaluating nonparticipating systems. Finally, they observe that for a fixed amount of effort, judging shallow pools for many queries is better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. This result confirms the effectiveness of pooling methods.

Date

22. 1.2011 14:20:56

Type

a

0.018578956 = product of:
  0.03715791 = sum of:
    0.03715791 = sum of:
      0.00911962 = weight(_text_:a in 4540) [ClassicSimilarity], result of:
        0.00911962 = score(doc=4540,freq=18.0), product of:
          0.04772363 = queryWeight, product of:
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.041389145 = queryNorm
          0.19109234 = fieldWeight in 4540, product of:
            4.2426405 = tf(freq=18.0), with freq of:
              18.0 = termFreq=18.0
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.0390625 = fieldNorm(doc=4540)
      0.028038291 = weight(_text_:22 in 4540) [ClassicSimilarity], result of:
        0.028038291 = score(doc=4540,freq=2.0), product of:
          0.14493774 = queryWeight, product of:
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.041389145 = queryNorm
          0.19345059 = fieldWeight in 4540, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.0390625 = fieldNorm(doc=4540)
  0.5 = coord(1/2)

Abstract

Purpose - This study intends to identify factors that affect relevance judgment of retrieved information as part of the 2007 TREC Legal track interactive task. Design/methodology/approach - Data were gathered and analyzed from the participants of the 2007 TREC Legal track interactive task using a questionnaire which includes not only a list of 80 relevance factors identified in prior research, but also a space for expressing their thoughts on relevance judgment in the process. Findings - This study finds that topicality remains a primary criterion, out of various options, for determining relevance, while specificity of the search request, task, or retrieved results also helps greatly in relevance judgment. Research limitations/implications - Relevance research should focus on the topicality and specificity of what is being evaluated as well as conducted in real environments. Practical implications - If multiple relevance factors are presented to assessors, the total number in a list should be below ten to take account of the limited processing capacity of human beings' short-term memory. Otherwise, the assessors might either completely ignore or inadequately consider some of the relevance factors when making judgment decisions. Originality/value - This study presents a method for reducing the artificiality of relevance research design, an apparent limitation in many related studies. Specifically, relevance judgment was made in this research as part of the 2007 TREC Legal track interactive task rather than a study devised for the sake of it. The assessors also served as searchers so that their searching experience would facilitate their subsequent relevance judgments.

Date

12. 7.2011 18:29:22

Type

a

0.018318176 = product of:
  0.036636353 = sum of:
    0.036636353 = sum of:
      0.00859806 = weight(_text_:a in 2587) [ClassicSimilarity], result of:
        0.00859806 = score(doc=2587,freq=16.0), product of:
          0.04772363 = queryWeight, product of:
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.041389145 = queryNorm
          0.18016359 = fieldWeight in 2587, product of:
            4.0 = tf(freq=16.0), with freq of:
              16.0 = termFreq=16.0
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.0390625 = fieldNorm(doc=2587)
      0.028038291 = weight(_text_:22 in 2587) [ClassicSimilarity], result of:
        0.028038291 = score(doc=2587,freq=2.0), product of:
          0.14493774 = queryWeight, product of:
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.041389145 = queryNorm
          0.19345059 = fieldWeight in 2587, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.0390625 = fieldNorm(doc=2587)
  0.5 = coord(1/2)

Abstract

Purpose The purpose of this paper is to propose a method to have more accurate results in comparing performance of the paired information retrieval (IR) systems with reference to the current method, which is based on the mean effectiveness scores of the systems across a set of identified topics/queries. Design/methodology/approach Based on the proposed approach, instead of the classic method of using a set of topic scores, the documents level scores are considered as the evaluation unit. These document scores are the defined document's weight, which play the role of the mean average precision (MAP) score of the systems as a significance test's statics. The experiments were conducted using the TREC 9 Web track collection. Findings The p-values generated through the two types of significance tests, namely the Student's t-test and Mann-Whitney show that by using the document level scores as an evaluation unit, the difference between IR systems is more significant compared with utilizing topic scores. Originality/value Utilizing a suitable test collection is a primary prerequisite for IR systems comparative evaluation. However, in addition to reusable test collections, having an accurate statistical testing is a necessity for these evaluations. The findings of this study will assist IR researchers to evaluate their retrieval systems and algorithms more accurately.

Date

20. 1.2015 18:30:22

Type

a

0.017742215 = product of:
  0.03548443 = sum of:
    0.03548443 = sum of:
      0.0074461387 = weight(_text_:a in 5287) [ClassicSimilarity], result of:
        0.0074461387 = score(doc=5287,freq=12.0), product of:
          0.04772363 = queryWeight, product of:
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.041389145 = queryNorm
          0.15602624 = fieldWeight in 5287, product of:
            3.4641016 = tf(freq=12.0), with freq of:
              12.0 = termFreq=12.0
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.0390625 = fieldNorm(doc=5287)
      0.028038291 = weight(_text_:22 in 5287) [ClassicSimilarity], result of:
        0.028038291 = score(doc=5287,freq=2.0), product of:
          0.14493774 = queryWeight, product of:
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.041389145 = queryNorm
          0.19345059 = fieldWeight in 5287, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.0390625 = fieldNorm(doc=5287)
  0.5 = coord(1/2)

Abstract

Purpose The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges' involvement. Then the study determines the variation in system rankings due to low effort relevance judgment in evaluating retrieval systems at different depth of evaluation. Design/methodology/approach Effort-based relevance judgments are generated using a proposed boxplot approach for simple document features, HTML features and readability features. The boxplot approach is a simple yet repeatable approach in classifying documents' effort while ensuring outlier scores do not skew the grading of the entire set of documents. Findings The retrieval systems evaluation using low effort relevance judgments has a stronger influence on shallow depth of evaluation compared to deeper depth. It is proved that difference in the system rankings is due to low effort documents and not the number of relevant documents. Originality/value Hence, it is crucial to evaluate retrieval systems at shallow depth using low effort relevance judgments.

Date

20. 1.2015 18:30:22

Type

a

0.015539082 = product of:
  0.031078164 = sum of:
    0.031078164 = sum of:
      0.003039873 = weight(_text_:a in 1786) [ClassicSimilarity], result of:
        0.003039873 = score(doc=1786,freq=2.0), product of:
          0.04772363 = queryWeight, product of:
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.041389145 = queryNorm
          0.06369744 = fieldWeight in 1786, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            1.153047 = idf(docFreq=37942, maxDocs=44218)
            0.0390625 = fieldNorm(doc=1786)
      0.028038291 = weight(_text_:22 in 1786) [ClassicSimilarity], result of:
        0.028038291 = score(doc=1786,freq=2.0), product of:
          0.14493774 = queryWeight, product of:
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.041389145 = queryNorm
          0.19345059 = fieldWeight in 1786, product of:
            1.4142135 = tf(freq=2.0), with freq of:
              2.0 = termFreq=2.0
            3.5018296 = idf(docFreq=3622, maxDocs=44218)
            0.0390625 = fieldNorm(doc=1786)
  0.5 = coord(1/2)

Date

6. 4.2015 19:31:22

Type

a

0.0026061484 = product of:
  0.0052122967 = sum of:
    0.0052122967 = product of:
      0.010424593 = sum of:
        0.010424593 = weight(_text_:a in 3447) [ClassicSimilarity], result of:
          0.010424593 = score(doc=3447,freq=12.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.21843673 = fieldWeight in 3447, product of:
              3.4641016 = tf(freq=12.0), with freq of:
                12.0 = termFreq=12.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0546875 = fieldNorm(doc=3447)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

The authors investigate factors influencing user satisfaction in information retrieval. It is evident from this study that user satisfaction is a subjective variable, which can be influenced by several factors such as system effectiveness, user effectiveness, user effort, and user characteristics and expectations. Therefore, information retrieval evaluators should consider all these factors in obtaining user satisfaction and in using it as a criterion of system effectiveness. Previous studies have conflicting conclusions on the relationship between user satisfaction and system effectiveness; this study has substantiated these findings and supports using user satisfaction as a criterion of system effectiveness.

Type

a

0.00252053 = product of:
  0.00504106 = sum of:
    0.00504106 = product of:
      0.01008212 = sum of:
        0.01008212 = weight(_text_:a in 3868) [ClassicSimilarity], result of:
          0.01008212 = score(doc=3868,freq=22.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.21126054 = fieldWeight in 3868, product of:
              4.690416 = tf(freq=22.0), with freq of:
                22.0 = termFreq=22.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=3868)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Gross et al. (2015) have demonstrated that about a quarter of hits would typically be lost to keyword searchers if contemporary academic library catalogs dropped their controlled subject headings. This paper reports on an analysis of the loss levels that would result if a bibliographic database, namely the Australian Education Index (AEI), were missing the subject descriptors and identifiers assigned by its professional indexers, employing the methodology developed by Gross and Taylor (2005), and later by Gross et al. (2015). The results indicate that AEI users would lose a similar proportion of hits per query to that experienced by library catalog users: on average, 27% of the resources found by a sample of keyword queries on the AEI database would not have been found without the subject indexing, based on the Australian Thesaurus of Education Descriptors (ATED). The paper also discusses the methodological limitations of these studies, pointing out that real-life users might still find some of the resources missed by a particular query through follow-up searches, while additional resources might also be found through iterative searching on the subject vocabulary. The paper goes on to describe a new research design, based on a before - and - after experiment, which addresses some of these limitations. It is argued that this alternative design will provide a more realistic picture of the value that professionally assigned subject indexing and controlled subject vocabularies can add to literature searching of a more scholarly and thorough kind.

Type

a

0.0024318986 = product of:
  0.004863797 = sum of:
    0.004863797 = product of:
      0.009727594 = sum of:
        0.009727594 = weight(_text_:a in 43) [ClassicSimilarity], result of:
          0.009727594 = score(doc=43,freq=8.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.20383182 = fieldWeight in 43, product of:
              2.828427 = tf(freq=8.0), with freq of:
                8.0 = termFreq=8.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0625 = fieldNorm(doc=43)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

We present the TREC Open Search track, which represents a new evaluation paradigm for information retrieval. It offers the possibility for researchers to evaluate their approaches in a live setting, with real, unsuspecting users of an existing search engine. The first edition of the track focuses on the academic search domain and features the ad-hoc scientific literature search task. We report on experiments with three different academic search engines: Cite-SeerX, SSOAR, and Microsoft Academic Search.

Type

a

0.0024128247 = product of:
  0.0048256493 = sum of:
    0.0048256493 = product of:
      0.009651299 = sum of:
        0.009651299 = weight(_text_:a in 2263) [ClassicSimilarity], result of:
          0.009651299 = score(doc=2263,freq=14.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.20223314 = fieldWeight in 2263, product of:
              3.7416575 = tf(freq=14.0), with freq of:
                14.0 = termFreq=14.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.046875 = fieldNorm(doc=2263)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

In this article, the use of a new term extraction method for query expansion (QE) in text retrieval is investigated. The new method expands the initial query with a structured representation made of weighted word pairs (WWP) extracted from a set of training documents (relevance feedback). Standard text retrieval systems can handle a WWP structure through custom Boolean weighted models. We experimented with both the explicit and pseudorelevance feedback schemas and compared the proposed term extraction method with others in the literature, such as KLD and RM3. Evaluations have been conducted on a number of test collections (Text REtrivel Conference [TREC]-6, -7, -8, -9, and -10). Results demonstrated that the QE method based on this new structure outperforms the baseline.

Type

a

0.0024032309 = product of:
  0.0048064617 = sum of:
    0.0048064617 = product of:
      0.0096129235 = sum of:
        0.0096129235 = weight(_text_:a in 4674) [ClassicSimilarity], result of:
          0.0096129235 = score(doc=4674,freq=20.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.20142901 = fieldWeight in 4674, product of:
              4.472136 = tf(freq=20.0), with freq of:
                20.0 = termFreq=20.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=4674)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

In information retrieval evaluation, pooling is a well-known technique to extract a sample of documents to be assessed for relevance. Given the pooled documents, a number of studies have proposed different prioritization methods to adjudicate documents for judgment. These methods follow different strategies to reduce the assessment effort. However, there is no clear guidance on how many relevance judgments are required for creating a reliable test collection. In this article we investigate and further develop methods to determine when to stop making relevance judgments. We propose a highly diversified set of stopping methods and provide a comprehensive analysis of the usefulness of the resulting test collections. Some of the stopping methods introduced here combine innovative estimates of recall with time series models used in Financial Trading. Experimental results on several representative collections show that some stopping methods can reduce up to 95% of the assessment effort and still produce a robust test collection. We demonstrate that the reduced set of judgments can be reliably employed to compare search systems using disparate effectiveness metrics such as Average Precision, NDCG, P@100, and Rank Biased Precision. With all these measures, the correlations found between full pool rankings and reduced pool rankings is very high.

Type

a

0.002279905 = product of:
  0.00455981 = sum of:
    0.00455981 = product of:
      0.00911962 = sum of:
        0.00911962 = weight(_text_:a in 4039) [ClassicSimilarity], result of:
          0.00911962 = score(doc=4039,freq=18.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.19109234 = fieldWeight in 4039, product of:
              4.2426405 = tf(freq=18.0), with freq of:
                18.0 = termFreq=18.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=4039)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

A non-negligible fraction of user queries end up with very few or even no matching results in leading commercial web search engines. In this work, we provide a detailed characterization of such queries and show that search engines try to improve such queries by showing the results of related queries. Through a user study, we show that these query suggestions are usually perceived as relevant. Also, through a query log analysis, we show that the users are dissatisfied after submitting a query that match no results at least 88.5% of the time. As a first step towards solving these no-answer queries, we devised a large number of features that can be used to identify such queries and built machine-learning models. These models can be useful for scenarios such as the mobile- or meta-search, where identifying a query that will retrieve no results at the client device (i.e., even before submitting it to the search engine) may yield gains in terms of the bandwidth usage, power consumption, and/or monetary costs. Experiments over query logs indicate that, despite the heavy skew in class sizes, our models achieve good prediction quality, with accuracy (in terms of area under the curve) up to 0.95.

Type

a

0.002279905 = product of:
  0.00455981 = sum of:
    0.00455981 = product of:
      0.00911962 = sum of:
        0.00911962 = weight(_text_:a in 4300) [ClassicSimilarity], result of:
          0.00911962 = score(doc=4300,freq=18.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.19109234 = fieldWeight in 4300, product of:
              4.2426405 = tf(freq=18.0), with freq of:
                18.0 = termFreq=18.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=4300)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Gross et al. (2015) have demonstrated that about a quarter of hits would typically be lost to keyword searchers if contemporary academic library catalogs dropped their controlled subject headings. This article reports on an investigation of the search value that subject descriptors and identifiers assigned by professional indexers add to a bibliographic database, namely the Australian Education Index (AEI). First, a similar methodology to that developed by Gross et al. (2015) was applied, with keyword searches representing a range of educational topics run on the AEI database with and without its subject indexing. The results indicated that AEI users would also lose, on average, about a quarter of hits per query. Second, an alternative research design was applied in which an experienced literature searcher was asked to find resources on a set of educational topics on an AEI database stripped of its subject indexing and then asked to search for additional resources on the same topics after the subject indexing had been reinserted. In this study, the proportion of additional resources that would have been lost had it not been for the subject indexing was again found to be about a quarter of the total resources found for each topic, on average.

Type

a

0.0021279112 = product of:
  0.0042558224 = sum of:
    0.0042558224 = product of:
      0.008511645 = sum of:
        0.008511645 = weight(_text_:a in 548) [ClassicSimilarity], result of:
          0.008511645 = score(doc=548,freq=2.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.17835285 = fieldWeight in 548, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.109375 = fieldNorm(doc=548)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Type

a

0.0021279112 = product of:
  0.0042558224 = sum of:
    0.0042558224 = product of:
      0.008511645 = sum of:
        0.008511645 = weight(_text_:a in 5156) [ClassicSimilarity], result of:
          0.008511645 = score(doc=5156,freq=8.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.17835285 = fieldWeight in 5156, product of:
              2.828427 = tf(freq=8.0), with freq of:
                8.0 = termFreq=8.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0546875 = fieldNorm(doc=5156)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Controlled vocabulary and subject headings in OPAC records have proven to be useful in improving search results. The authors used a survey to gather information about librarian opinions and professional use of controlled vocabulary. Data from a range of backgrounds and expertise were examined, including academic and public libraries, and technical services as well as public services professionals. Responses overall demonstrated positive opinions of the value of controlled vocabulary, including in reference interactions as well as during bibliographic instruction sessions. Results are also examined based upon factors such as age and type of librarian.

Type

a

0.0021060861 = product of:
  0.0042121722 = sum of:
    0.0042121722 = product of:
      0.0084243445 = sum of:
        0.0084243445 = weight(_text_:a in 3097) [ClassicSimilarity], result of:
          0.0084243445 = score(doc=3097,freq=6.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.17652355 = fieldWeight in 3097, product of:
              2.4494898 = tf(freq=6.0), with freq of:
                6.0 = termFreq=6.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0625 = fieldNorm(doc=3097)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Das Projekt "Aufbau einer Infrastruktur für Information Retrieval-Evaluationen" (AIIRE) bietet eine Softwareinfrastruktur zur Unterstützung von Information Retrieval-Evaluationen (IR-Evaluationen). Die Infrastruktur basiert auf einem Tool-Kit, das bei GESIS im Rahmen des DFG-Projekts IRM entwickelt wurde. Ziel ist es, ein System zu bieten, das zur Forschung und Lehre am Fachbereich Media für IR-Evaluationen genutzt werden kann. This paper describes some aspects of a project called "Aufbau einer Infrastruktur für Information Retrieval-Evaluationen" (AIIRE). Its goal is to build a software-infrastructure which supports the evaluation of information retrieval algorithms.

Type

a

0.0020106873 = product of:
  0.0040213745 = sum of:
    0.0040213745 = product of:
      0.008042749 = sum of:
        0.008042749 = weight(_text_:a in 684) [ClassicSimilarity], result of:
          0.008042749 = score(doc=684,freq=14.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.1685276 = fieldWeight in 684, product of:
              3.7416575 = tf(freq=14.0), with freq of:
                14.0 = termFreq=14.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=684)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

With the increasing number and diversity of search tools available, interest in the evaluation of search systems, particularly from a user perspective, has grown among researchers. More researchers are designing and evaluating interactive information retrieval (IIR) systems and beginning to innovate in evaluation methods. Maturation of a research specialty relies on the ability to replicate research, provide standards for measurement and analysis, and understand past endeavors. This article presents a historical overview of 40 years of IIR evaluation studies using the method of systematic review. A total of 2,791 journal and conference units were manually examined and 127 articles were selected for analysis in this study, based on predefined inclusion and exclusion criteria. These articles were systematically coded using features such as author, publication date, sources and references, and properties of the research method used in the articles, such as number of subjects, tasks, corpora, and measures. Results include data describing the growth of IIR studies over time, the most frequently occurring and cited authors and sources, and the most common types of corpora and measures used. An additional product of this research is a bibliography of IIR evaluation research that can be used by students, teachers, and those new to the area. To the authors' knowledge, this is the first historical, systematic characterization of the IIR evaluation literature, including the documentation of methods and measures used by researchers in this specialty.

Type

a

0.0020106873 = product of:
  0.0040213745 = sum of:
    0.0040213745 = product of:
      0.008042749 = sum of:
        0.008042749 = weight(_text_:a in 3700) [ClassicSimilarity], result of:
          0.008042749 = score(doc=3700,freq=14.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.1685276 = fieldWeight in 3700, product of:
              3.7416575 = tf(freq=14.0), with freq of:
                14.0 = termFreq=14.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=3700)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Purpose This paper demonstrates how to apply traditional information retrieval evaluation methods based on standards from the Text REtrieval Conference (TREC) and web search evaluation to all types of modern library information systems including online public access catalogs, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources. Design/methodology/approach We apply conventional procedures from information retrieval evaluation to the library information system context considering the specific characteristics of modern library materials. Findings We introduce a framework consisting of five parts: (1) search queries, (2) search results, (3) assessors, (4) testing, and (5) data analysis. We show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs. printed monographs and what issues need to be considered for retrieval tests in the library context. Practical implications The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context. Originality/value Although a considerable amount of research has been done on information retrieval evaluation, and standards for conducting retrieval effectiveness studies do exist, to our knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century library information systems. We demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

Type

a

0.0020106873 = product of:
  0.0040213745 = sum of:
    0.0040213745 = product of:
      0.008042749 = sum of:
        0.008042749 = weight(_text_:a in 5049) [ClassicSimilarity], result of:
          0.008042749 = score(doc=5049,freq=14.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.1685276 = fieldWeight in 5049, product of:
              3.7416575 = tf(freq=14.0), with freq of:
                14.0 = termFreq=14.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=5049)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Information Retrieval (IR) develops complex systems, constituted of several components, which aim at returning and optimally ranking the most relevant documents in response to user queries. In this context, experimental evaluation plays a central role, since it allows for measuring IR systems effectiveness, increasing the understanding of their functioning, and better directing the efforts for improving them. Current evaluation methodologies are limited by two major factors: (i) IR systems are evaluated as "black boxes", since it is not possible to decompose the contributions of the different components, e.g., stop lists, stemmers, and IR models; (ii) given that it is not possible to predict the effectiveness of an IR system, both academia and industry need to explore huge numbers of systems, originated by large combinatorial compositions of their components, to understand how they perform and how these components interact together. We propose a Combinatorial visuaL Analytics system for Information Retrieval Evaluation (CLAIRE) which allows for exploring and making sense of the performances of a large amount of IR systems, in order to quickly and intuitively grasp which system configurations are preferred, what are the contributions of the different components and how these components interact together. The CLAIRE system is then validated against use cases based on several test collections using a wide set of systems, generated by a combinatorial composition of several off-the-shelf components, representing the most common denominator almost always present in English IR systems. In particular, we validate the findings enabled by CLAIRE with respect to consolidated deep statistical analyses and we show that the CLAIRE system allows the generation of new insights, which were not detectable with traditional approaches.

Type

a

0.0020106873 = product of:
  0.0040213745 = sum of:
    0.0040213745 = product of:
      0.008042749 = sum of:
        0.008042749 = weight(_text_:a in 5098) [ClassicSimilarity], result of:
          0.008042749 = score(doc=5098,freq=14.0), product of:
            0.04772363 = queryWeight, product of:
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.041389145 = queryNorm
            0.1685276 = fieldWeight in 5098, product of:
              3.7416575 = tf(freq=14.0), with freq of:
                14.0 = termFreq=14.0
              1.153047 = idf(docFreq=37942, maxDocs=44218)
              0.0390625 = fieldNorm(doc=5098)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Evaluating Information Retrieval systems is crucial to making progress in search technologies. Evaluation is often based on assembling reference collections consisting of documents, queries and relevance judgments done by humans. In large-scale environments, exhaustively judging relevance becomes infeasible. Instead, only a pool of documents is judged for relevance. By selectively choosing documents from the pool we can optimize the number of judgments required to identify a given number of relevant documents. We argue that this iterative selection process can be naturally modeled as a reinforcement learning problem and propose innovative and formal adjudication methods based on multi-armed bandits. Casting document judging as a multi-armed bandit problem is not only theoretically appealing, but also leads to highly effective adjudication methods. Under this bandit allocation framework, we consider stationary and non-stationary models and propose seven new document adjudication methods (five stationary methods and two non-stationary variants). Our paper also reports a series of experiments performed to thoroughly compare our new methods against current adjudication methods. This comparative study includes existing methods designed for pooling-based evaluation and existing methods designed for metasearch. Our experiments show that our theoretically grounded adjudication methods can substantially minimize the assessment effort.

Type

a