Search (7 results, page 1 of 1)

  • × language_ss:"e"
  • × theme_ss:"Retrievalstudien"
  • × year_i:[2010 TO 2020}
  1. Thornley, C.V.; Johnson, A.C.; Smeaton, A.F.; Lee, H.: ¬The scholarly impact of TRECVid (2003-2009) (2011) 0.02 
    0.016490098 = product of:
  0.032980196 = sum of:
    0.032980196 = product of:
      0.06596039 = sum of:
        0.06596039 = weight(_text_:2003 in 4363) [ClassicSimilarity], result of:
          0.06596039 = score(doc=4363,freq=4.0), product of:
            0.19453894 = queryWeight, product of:
              4.339969 = idf(docFreq=1566, maxDocs=44218)
              0.044824958 = queryNorm
            0.3390601 = fieldWeight in 4363, product of:
              2.0 = tf(freq=4.0), with freq of:
                4.0 = termFreq=4.0
              4.339969 = idf(docFreq=1566, maxDocs=44218)
              0.0390625 = fieldNorm(doc=4363)
      0.5 = coord(1/2)
  0.5 = coord(1/2)
    Abstract
    This paper reports on an investigation into the scholarly impact of the TRECVid (Text Retrieval and Evaluation Conference, Video Retrieval Evaluation) benchmarking conferences between 2003 and 2009. The contribution of TRECVid to research in video retrieval is assessed by analyzing publication content to show the development of techniques and approaches over time and by analyzing publication impact through publication numbers and citation analysis. Popular conference and journal venues for TRECVid publications are identified in terms of number of citations received. For a selection of participants at different career stages, the relative importance of TRECVid publications in terms of citations vis à vis their other publications is investigated. TRECVid, as an evaluation conference, provides data on which research teams 'scored' highly against the evaluation criteria and the relationship between 'top scoring' teams at TRECVid and the 'top scoring' papers in terms of citations is analyzed. A strong relationship was found between 'success' at TRECVid and 'success' at citations both for high scoring and low scoring teams. The implications of the study in terms of the value of TRECVid as a research activity, and the value of bibliometric analysis as a research evaluation tool, are discussed.
  2. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.01 
    0.009328208 = product of:
  0.018656416 = sum of:
    0.018656416 = product of:
      0.03731283 = sum of:
        0.03731283 = weight(_text_:2003 in 5092) [ClassicSimilarity], result of:
          0.03731283 = score(doc=5092,freq=2.0), product of:
            0.19453894 = queryWeight, product of:
              4.339969 = idf(docFreq=1566, maxDocs=44218)
              0.044824958 = queryNorm
            0.19180135 = fieldWeight in 5092, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              4.339969 = idf(docFreq=1566, maxDocs=44218)
              0.03125 = fieldNorm(doc=5092)
      0.5 = coord(1/2)
  0.5 = coord(1/2)
    Abstract
    While test collections provide the cornerstone for Cranfield-based evaluation of information retrieval (IR) systems, it has become practically infeasible to rely on traditional pooling techniques to construct test collections at the scale of today's massive document collections (e.g., ClueWeb12's 700M+ Webpages). This has motivated a flurry of studies proposing more cost-effective yet reliable IR evaluation methods. In this paper, we propose a new intelligent topic selection method which reduces the number of search topics (and thereby costly human relevance judgments) needed for reliable IR evaluation. To rigorously assess our method, we integrate previously disparate lines of research on intelligent topic selection and deep vs. shallow judging (i.e., whether it is more cost-effective to collect many relevance judgments for a few topics or a few judgments for many topics). While prior work on intelligent topic selection has never been evaluated against shallow judging baselines, prior work on deep vs. shallow judging has largely argued for shallowed judging, but assuming random topic selection. We argue that for evaluating any topic selection method, ultimately one must ask whether it is actually useful to select topics, or should one simply perform shallow judging over many topics? In seeking a rigorous answer to this over-arching question, we conduct a comprehensive investigation over a set of relevant factors never previously studied together: 1) method of topic selection; 2) the effect of topic familiarity on human judging speed; and 3) how different topic generation processes (requiring varying human effort) impact (i) budget utilization and (ii) the resultant quality of judgments. Experiments on NIST TREC Robust 2003 and Robust 2004 test collections show that not only can we reliably evaluate IR systems with fewer topics, but also that: 1) when topics are intelligently selected, deep judging is often more cost-effective than shallow judging in evaluation reliability; and 2) topic familiarity and topic generation costs greatly impact the evaluation cost vs. reliability trade-off. Our findings challenge conventional wisdom in showing that deep judging is often preferable to shallow judging when topics are selected intelligently.
  3. Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011) 0.01 
    0.007591454 = product of:
  0.015182908 = sum of:
    0.015182908 = product of:
      0.030365815 = sum of:
        0.030365815 = weight(_text_:22 in 4197) [ClassicSimilarity], result of:
          0.030365815 = score(doc=4197,freq=2.0), product of:
            0.15696937 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044824958 = queryNorm
            0.19345059 = fieldWeight in 4197, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0390625 = fieldNorm(doc=4197)
      0.5 = coord(1/2)
  0.5 = coord(1/2)
    Date
    22. 1.2011 14:20:56
  4. Chu, H.: Factors affecting relevance judgment : a report from TREC Legal track (2011) 0.01 
    0.007591454 = product of:
  0.015182908 = sum of:
    0.015182908 = product of:
      0.030365815 = sum of:
        0.030365815 = weight(_text_:22 in 4540) [ClassicSimilarity], result of:
          0.030365815 = score(doc=4540,freq=2.0), product of:
            0.15696937 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044824958 = queryNorm
            0.19345059 = fieldWeight in 4540, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0390625 = fieldNorm(doc=4540)
      0.5 = coord(1/2)
  0.5 = coord(1/2)
    Date
    12. 7.2011 18:29:22
  5. Wildemuth, B.; Freund, L.; Toms, E.G.: Untangling search task complexity and difficulty in the context of interactive information retrieval studies (2014) 0.01 
    0.007591454 = product of:
  0.015182908 = sum of:
    0.015182908 = product of:
      0.030365815 = sum of:
        0.030365815 = weight(_text_:22 in 1786) [ClassicSimilarity], result of:
          0.030365815 = score(doc=1786,freq=2.0), product of:
            0.15696937 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044824958 = queryNorm
            0.19345059 = fieldWeight in 1786, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0390625 = fieldNorm(doc=1786)
      0.5 = coord(1/2)
  0.5 = coord(1/2)
    Date
    6. 4.2015 19:31:22
  6. Ravana, S.D.; Taheri, M.S.; Rajagopal, P.: Document-based approach to improve the accuracy of pairwise comparison in evaluating information retrieval systems (2015) 0.01 
    0.007591454 = product of:
  0.015182908 = sum of:
    0.015182908 = product of:
      0.030365815 = sum of:
        0.030365815 = weight(_text_:22 in 2587) [ClassicSimilarity], result of:
          0.030365815 = score(doc=2587,freq=2.0), product of:
            0.15696937 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044824958 = queryNorm
            0.19345059 = fieldWeight in 2587, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0390625 = fieldNorm(doc=2587)
      0.5 = coord(1/2)
  0.5 = coord(1/2)
    Date
    20. 1.2015 18:30:22
  7. Rajagopal, P.; Ravana, S.D.; Koh, Y.S.; Balakrishnan, V.: Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment (2019) 0.01 
    0.007591454 = product of:
  0.015182908 = sum of:
    0.015182908 = product of:
      0.030365815 = sum of:
        0.030365815 = weight(_text_:22 in 5287) [ClassicSimilarity], result of:
          0.030365815 = score(doc=5287,freq=2.0), product of:
            0.15696937 = queryWeight, product of:
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.044824958 = queryNorm
            0.19345059 = fieldWeight in 5287, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              3.5018296 = idf(docFreq=3622, maxDocs=44218)
              0.0390625 = fieldNorm(doc=5287)
      0.5 = coord(1/2)
  0.5 = coord(1/2)
    Date
    20. 1.2015 18:30:22

Authors