Document (#42093)

Author
Kutlu, M.
Elsayed, T.
Lease, M.
Title
Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging
Source
Information processing and management. 54(2018) no.1, S.37-59
Year
2018
Abstract
While test collections provide the cornerstone for Cranfield-based evaluation of information retrieval (IR) systems, it has become practically infeasible to rely on traditional pooling techniques to construct test collections at the scale of today's massive document collections (e.g., ClueWeb12's 700M+ Webpages). This has motivated a flurry of studies proposing more cost-effective yet reliable IR evaluation methods. In this paper, we propose a new intelligent topic selection method which reduces the number of search topics (and thereby costly human relevance judgments) needed for reliable IR evaluation. To rigorously assess our method, we integrate previously disparate lines of research on intelligent topic selection and deep vs. shallow judging (i.e., whether it is more cost-effective to collect many relevance judgments for a few topics or a few judgments for many topics). While prior work on intelligent topic selection has never been evaluated against shallow judging baselines, prior work on deep vs. shallow judging has largely argued for shallowed judging, but assuming random topic selection. We argue that for evaluating any topic selection method, ultimately one must ask whether it is actually useful to select topics, or should one simply perform shallow judging over many topics? In seeking a rigorous answer to this over-arching question, we conduct a comprehensive investigation over a set of relevant factors never previously studied together: 1) method of topic selection; 2) the effect of topic familiarity on human judging speed; and 3) how different topic generation processes (requiring varying human effort) impact (i) budget utilization and (ii) the resultant quality of judgments. Experiments on NIST TREC Robust 2003 and Robust 2004 test collections show that not only can we reliably evaluate IR systems with fewer topics, but also that: 1) when topics are intelligently selected, deep judging is often more cost-effective than shallow judging in evaluation reliability; and 2) topic familiarity and topic generation costs greatly impact the evaluation cost vs. reliability trade-off. Our findings challenge conventional wisdom in showing that deep judging is often preferable to shallow judging when topics are selected intelligently.
Content
Vgl.: https://doi.org/10.1016/j.ipm.2017.09.002.
Theme
Retrievalstudien

Similar documents (content)

  1. Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011) 0.63
    0.6312302 = sum of:
      0.6312302 = product of:
        1.3150629 = sum of:
          0.01772269 = weight(abstract_txt:many in 1198) [ClassicSimilarity], result of:
            0.01772269 = score(doc=1198,freq=3.0), product of:
              0.045542125 = queryWeight, product of:
                1.0018048 = boost
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.011065283 = queryNorm
              0.3891494 = fieldWeight in 1198, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.030219408 = weight(abstract_txt:reliable in 1198) [ClassicSimilarity], result of:
            0.030219408 = score(doc=1198,freq=1.0), product of:
              0.08189565 = queryWeight, product of:
                1.0968854 = boost
                6.7474093 = idf(docFreq=134, maxDocs=42306)
                0.011065283 = queryNorm
              0.36899894 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.7474093 = idf(docFreq=134, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.032673065 = weight(abstract_txt:reliability in 1198) [ClassicSimilarity], result of:
            0.032673065 = score(doc=1198,freq=1.0), product of:
              0.08627073 = queryWeight, product of:
                1.1258035 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.011065283 = queryNorm
              0.37872714 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.018345997 = weight(abstract_txt:method in 1198) [ClassicSimilarity], result of:
            0.018345997 = score(doc=1198,freq=1.0), product of:
              0.07397888 = queryWeight, product of:
                1.474347 = boost
                4.534668 = idf(docFreq=1233, maxDocs=42306)
                0.011065283 = queryNorm
              0.24798965 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.534668 = idf(docFreq=1233, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.028993104 = weight(abstract_txt:collections in 1198) [ClassicSimilarity], result of:
            0.028993104 = score(doc=1198,freq=2.0), product of:
              0.079664834 = queryWeight, product of:
                1.5299568 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.011065283 = queryNorm
              0.36393854 = fieldWeight in 1198, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.026774358 = weight(abstract_txt:evaluation in 1198) [ClassicSimilarity], result of:
            0.026774358 = score(doc=1198,freq=1.0), product of:
              0.10895738 = queryWeight, product of:
                2.1913905 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.011065283 = queryNorm
              0.2457324 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.06534613 = weight(abstract_txt:judgments in 1198) [ClassicSimilarity], result of:
            0.06534613 = score(doc=1198,freq=1.0), product of:
              0.17254145 = queryWeight, product of:
                2.251607 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.011065283 = queryNorm
              0.37872714 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.075300895 = weight(abstract_txt:deep in 1198) [ClassicSimilarity], result of:
            0.075300895 = score(doc=1198,freq=1.0), product of:
              0.2042915 = queryWeight, product of:
                2.7392142 = boost
                6.7400293 = idf(docFreq=135, maxDocs=42306)
                0.011065283 = queryNorm
              0.36859536 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.7400293 = idf(docFreq=135, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.07515399 = weight(abstract_txt:topics in 1198) [ClassicSimilarity], result of:
            0.07515399 = score(doc=1198,freq=2.0), product of:
              0.1894007 = queryWeight, product of:
                3.3361964 = boost
                5.1305914 = idf(docFreq=679, maxDocs=42306)
                0.011065283 = queryNorm
              0.3967989 = fieldWeight in 1198, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.1305914 = idf(docFreq=679, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.07195951 = weight(abstract_txt:topic in 1198) [ClassicSimilarity], result of:
            0.07195951 = score(doc=1198,freq=1.0), product of:
              0.2577804 = queryWeight, product of:
                4.5639105 = boost
                5.104465 = idf(docFreq=697, maxDocs=42306)
                0.011065283 = queryNorm
              0.27915043 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.104465 = idf(docFreq=697, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.2009418 = weight(abstract_txt:shallow in 1198) [ClassicSimilarity], result of:
            0.2009418 = score(doc=1198,freq=1.0), product of:
              0.43968308 = queryWeight, product of:
                4.754827 = boost
                8.356848 = idf(docFreq=26, maxDocs=42306)
                0.011065283 = queryNorm
              0.4570151 = fieldWeight in 1198, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.356848 = idf(docFreq=26, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
          0.67163193 = weight(abstract_txt:judging in 1198) [ClassicSimilarity], result of:
            0.67163193 = score(doc=1198,freq=5.0), product of:
              0.6682751 = queryWeight, product of:
                7.34835 = boost
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.011065283 = queryNorm
              1.0050231 = fieldWeight in 1198, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.0546875 = fieldNorm(doc=1198)
        0.48 = coord(12/25)
    
  2. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.30
    0.29709157 = sum of:
      0.29709157 = product of:
        0.8252543 = sum of:
          0.04884194 = weight(abstract_txt:reliable in 4592) [ClassicSimilarity], result of:
            0.04884194 = score(doc=4592,freq=2.0), product of:
              0.08189565 = queryWeight, product of:
                1.0968854 = boost
                6.7474093 = idf(docFreq=134, maxDocs=42306)
                0.011065283 = queryNorm
              0.59639233 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.7474093 = idf(docFreq=134, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.044212744 = weight(abstract_txt:human in 4592) [ClassicSimilarity], result of:
            0.044212744 = score(doc=4592,freq=6.0), product of:
              0.060825713 = queryWeight, product of:
                1.1577641 = boost
                4.7479334 = idf(docFreq=996, maxDocs=42306)
                0.011065283 = queryNorm
              0.7268759 = fieldWeight in 4592, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                4.7479334 = idf(docFreq=996, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.021950293 = weight(abstract_txt:test in 4592) [ClassicSimilarity], result of:
            0.021950293 = score(doc=4592,freq=1.0), product of:
              0.069299944 = queryWeight, product of:
                1.235785 = boost
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.011065283 = queryNorm
              0.3167433 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.067893 = idf(docFreq=723, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.029651606 = weight(abstract_txt:method in 4592) [ClassicSimilarity], result of:
            0.029651606 = score(doc=4592,freq=2.0), product of:
              0.07397888 = queryWeight, product of:
                1.474347 = boost
                4.534668 = idf(docFreq=1233, maxDocs=42306)
                0.011065283 = queryNorm
              0.4008118 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.534668 = idf(docFreq=1233, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.023429966 = weight(abstract_txt:collections in 4592) [ClassicSimilarity], result of:
            0.023429966 = score(doc=4592,freq=1.0), product of:
              0.079664834 = queryWeight, product of:
                1.5299568 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.011065283 = queryNorm
              0.29410675 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.030599266 = weight(abstract_txt:evaluation in 4592) [ClassicSimilarity], result of:
            0.030599266 = score(doc=4592,freq=1.0), product of:
              0.10895738 = queryWeight, product of:
                2.1913905 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.011065283 = queryNorm
              0.28083703 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.16699246 = weight(abstract_txt:judgments in 4592) [ClassicSimilarity], result of:
            0.16699246 = score(doc=4592,freq=5.0), product of:
              0.17254145 = queryWeight, product of:
                2.251607 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.011065283 = queryNorm
              0.9678396 = fieldWeight in 4592, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.11630413 = weight(abstract_txt:topic in 4592) [ClassicSimilarity], result of:
            0.11630413 = score(doc=4592,freq=2.0), product of:
              0.2577804 = queryWeight, product of:
                4.5639105 = boost
                5.104465 = idf(docFreq=697, maxDocs=42306)
                0.011065283 = queryNorm
              0.4511752 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.104465 = idf(docFreq=697, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
          0.3432719 = weight(abstract_txt:judging in 4592) [ClassicSimilarity], result of:
            0.3432719 = score(doc=4592,freq=1.0), product of:
              0.6682751 = queryWeight, product of:
                7.34835 = boost
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.011065283 = queryNorm
              0.51366854 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.0625 = fieldNorm(doc=4592)
        0.36 = coord(9/25)
    
  3. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.18
    0.17711262 = sum of:
      0.17711262 = product of:
        0.7379693 = sum of:
          0.019242475 = weight(abstract_txt:effective in 2016) [ClassicSimilarity], result of:
            0.019242475 = score(doc=2016,freq=1.0), product of:
              0.06347654 = queryWeight, product of:
                1.1827232 = boost
                4.8502893 = idf(docFreq=899, maxDocs=42306)
                0.011065283 = queryNorm
              0.30314308 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.8502893 = idf(docFreq=899, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.023429966 = weight(abstract_txt:collections in 2016) [ClassicSimilarity], result of:
            0.023429966 = score(doc=2016,freq=1.0), product of:
              0.079664834 = queryWeight, product of:
                1.5299568 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.011065283 = queryNorm
              0.29410675 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.0432739 = weight(abstract_txt:evaluation in 2016) [ClassicSimilarity], result of:
            0.0432739 = score(doc=2016,freq=2.0), product of:
              0.10895738 = queryWeight, product of:
                2.1913905 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.011065283 = queryNorm
              0.39716354 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.105615295 = weight(abstract_txt:judgments in 2016) [ClassicSimilarity], result of:
            0.105615295 = score(doc=2016,freq=2.0), product of:
              0.17254145 = queryWeight, product of:
                2.251607 = boost
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.011065283 = queryNorm
              0.6121155 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.9252963 = idf(docFreq=112, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.060947925 = weight(abstract_txt:selection in 2016) [ClassicSimilarity], result of:
            0.060947925 = score(doc=2016,freq=1.0), product of:
              0.18158118 = queryWeight, product of:
                3.0556266 = boost
                5.3704176 = idf(docFreq=534, maxDocs=42306)
                0.011065283 = queryNorm
              0.3356511 = fieldWeight in 2016, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.3704176 = idf(docFreq=534, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
          0.48545974 = weight(abstract_txt:judging in 2016) [ClassicSimilarity], result of:
            0.48545974 = score(doc=2016,freq=2.0), product of:
              0.6682751 = queryWeight, product of:
                7.34835 = boost
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.011065283 = queryNorm
              0.726437 = fieldWeight in 2016, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.0625 = fieldNorm(doc=2016)
        0.24 = coord(6/25)
    
  4. Xu, J.; Croft, W.B.: Topic-based language models for distributed retrieval (2000) 0.15
    0.1472154 = sum of:
      0.1472154 = product of:
        0.5257693 = sum of:
          0.020672165 = weight(abstract_txt:many in 1039) [ClassicSimilarity], result of:
            0.020672165 = score(doc=1039,freq=2.0), product of:
              0.045542125 = queryWeight, product of:
                1.0018048 = boost
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.011065283 = queryNorm
              0.45391306 = fieldWeight in 1039, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.078125 = fieldNorm(doc=1039)
          0.016396584 = weight(abstract_txt:over in 1039) [ClassicSimilarity], result of:
            0.016396584 = score(doc=1039,freq=1.0), product of:
              0.049166378 = queryWeight, product of:
                1.0409038 = boost
                4.268695 = idf(docFreq=1609, maxDocs=42306)
                0.011065283 = queryNorm
              0.3334918 = fieldWeight in 1039, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.268695 = idf(docFreq=1609, maxDocs=42306)
                0.078125 = fieldNorm(doc=1039)
          0.024053091 = weight(abstract_txt:effective in 1039) [ClassicSimilarity], result of:
            0.024053091 = score(doc=1039,freq=1.0), product of:
              0.06347654 = queryWeight, product of:
                1.1827232 = boost
                4.8502893 = idf(docFreq=899, maxDocs=42306)
                0.011065283 = queryNorm
              0.37892884 = fieldWeight in 1039, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.8502893 = idf(docFreq=899, maxDocs=42306)
                0.078125 = fieldNorm(doc=1039)
          0.05857491 = weight(abstract_txt:collections in 1039) [ClassicSimilarity], result of:
            0.05857491 = score(doc=1039,freq=4.0), product of:
              0.079664834 = queryWeight, product of:
                1.5299568 = boost
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.011065283 = queryNorm
              0.73526686 = fieldWeight in 1039, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.705708 = idf(docFreq=1039, maxDocs=42306)
                0.078125 = fieldNorm(doc=1039)
          0.076184906 = weight(abstract_txt:selection in 1039) [ClassicSimilarity], result of:
            0.076184906 = score(doc=1039,freq=1.0), product of:
              0.18158118 = queryWeight, product of:
                3.0556266 = boost
                5.3704176 = idf(docFreq=534, maxDocs=42306)
                0.011065283 = queryNorm
              0.4195639 = fieldWeight in 1039, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.3704176 = idf(docFreq=534, maxDocs=42306)
                0.078125 = fieldNorm(doc=1039)
          0.15183401 = weight(abstract_txt:topics in 1039) [ClassicSimilarity], result of:
            0.15183401 = score(doc=1039,freq=4.0), product of:
              0.1894007 = queryWeight, product of:
                3.3361964 = boost
                5.1305914 = idf(docFreq=679, maxDocs=42306)
                0.011065283 = queryNorm
              0.80165493 = fieldWeight in 1039, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                5.1305914 = idf(docFreq=679, maxDocs=42306)
                0.078125 = fieldNorm(doc=1039)
          0.1780536 = weight(abstract_txt:topic in 1039) [ClassicSimilarity], result of:
            0.1780536 = score(doc=1039,freq=3.0), product of:
              0.2577804 = queryWeight, product of:
                4.5639105 = boost
                5.104465 = idf(docFreq=697, maxDocs=42306)
                0.011065283 = queryNorm
              0.6907182 = fieldWeight in 1039, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                5.104465 = idf(docFreq=697, maxDocs=42306)
                0.078125 = fieldNorm(doc=1039)
        0.28 = coord(7/25)
    
  5. Fitzgerald, M.A.; Galloway, C.: Relevance judging, evaluation, and decision making in virtual libraries : a descriptive study (2001) 0.14
    0.14427868 = sum of:
      0.14427868 = product of:
        0.90174174 = sum of:
          0.017540913 = weight(abstract_txt:many in 944) [ClassicSimilarity], result of:
            0.017540913 = score(doc=944,freq=1.0), product of:
              0.045542125 = queryWeight, product of:
                1.0018048 = boost
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.011065283 = queryNorm
              0.385158 = fieldWeight in 944, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.108352 = idf(docFreq=1889, maxDocs=42306)
                0.09375 = fieldNorm(doc=944)
          0.06491085 = weight(abstract_txt:evaluation in 944) [ClassicSimilarity], result of:
            0.06491085 = score(doc=944,freq=2.0), product of:
              0.10895738 = queryWeight, product of:
                2.1913905 = boost
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.011065283 = queryNorm
              0.5957453 = fieldWeight in 944, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.4933925 = idf(docFreq=1285, maxDocs=42306)
                0.09375 = fieldNorm(doc=944)
          0.0911004 = weight(abstract_txt:topics in 944) [ClassicSimilarity], result of:
            0.0911004 = score(doc=944,freq=1.0), product of:
              0.1894007 = queryWeight, product of:
                3.3361964 = boost
                5.1305914 = idf(docFreq=679, maxDocs=42306)
                0.011065283 = queryNorm
              0.48099294 = fieldWeight in 944, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.1305914 = idf(docFreq=679, maxDocs=42306)
                0.09375 = fieldNorm(doc=944)
          0.7281896 = weight(abstract_txt:judging in 944) [ClassicSimilarity], result of:
            0.7281896 = score(doc=944,freq=2.0), product of:
              0.6682751 = queryWeight, product of:
                7.34835 = boost
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.011065283 = queryNorm
              1.0896554 = fieldWeight in 944, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.218697 = idf(docFreq=30, maxDocs=42306)
                0.09375 = fieldNorm(doc=944)
        0.16 = coord(4/25)