Document (#32940)

Author
Hobson, S.P.
Dorr, B.J.
Monz, C.
Schwartz, R.
Title
Task-based evaluation of text summarization using Relevance Prediction
Source
Information processing and management. 43(2007) no.6, S.1482-1499
Year
2007
Abstract
This article introduces a new task-based evaluation measure called Relevance Prediction that is a more intuitive measure of an individual's performance on a real-world task than interannotator agreement. Relevance Prediction parallels what a user does in the real world task of browsing a set of documents using standard search tools, i.e., the user judges relevance based on a short summary and then that same user - not an independent user - decides whether to open (and judge) the corresponding document. This measure is shown to be a more reliable measure of task performance than LDC Agreement, a current gold-standard based measure used in the summarization evaluation community. Our goal is to provide a stable framework within which developers of new automatic measures may make stronger statistical statements about the effectiveness of their measures in predicting summary usefulness. We demonstrate - as a proof-of-concept methodology for automatic metric developers - that a current automatic evaluation measure has a better correlation with Relevance Prediction than with LDC Agreement and that the significance level for detected differences is higher for the former than for the latter.
Theme
Automatisches Abstracting

Similar documents (author)

  1. Zajic, D.; Dorr, B.J.; Lin, J.; Schwartz, R.: Multi-candidate reduction : sentence compression as a tool for document summarization tasks (2007) 3.83
    3.8310022 = sum of:
      3.8310022 = sum of:
        1.5363272 = weight(author_txt:schwartz in 2945) [ClassicSimilarity], result of:
          1.5363272 = score(doc=2945,freq=1.0), product of:
            0.60775656 = queryWeight, product of:
              8.089171 = idf(docFreq=34, maxDocs=41962)
              0.07513212 = queryNorm
            2.5278661 = fieldWeight in 2945, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              8.089171 = idf(docFreq=34, maxDocs=41962)
              0.3125 = fieldNorm(doc=2945)
        2.294675 = weight(author_txt:dorr in 2945) [ClassicSimilarity], result of:
          2.294675 = score(doc=2945,freq=1.0), product of:
            0.79412335 = queryWeight, product of:
              1.1430867 = boost
              9.246624 = idf(docFreq=10, maxDocs=41962)
              0.07513212 = queryNorm
            2.88957 = fieldWeight in 2945, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.246624 = idf(docFreq=10, maxDocs=41962)
              0.3125 = fieldNorm(doc=2945)
    
  2. Dorr, B.J.: Large-scale dictionary construction for foreign language tutoring and interlingual machine translation (1997) 2.29
    2.294675 = sum of:
      2.294675 = product of:
        4.58935 = sum of:
          4.58935 = weight(author_txt:dorr in 4658) [ClassicSimilarity], result of:
            4.58935 = score(doc=4658,freq=1.0), product of:
              0.79412335 = queryWeight, product of:
                1.1430867 = boost
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.07513212 = queryNorm
              5.77914 = fieldWeight in 4658, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.625 = fieldNorm(doc=4658)
        0.5 = coord(1/2)
    
  3. Dorr, B.J.; Olsen, M.B.: Multilingual generation : the role of telicity in lexical choice and syntactic realization (1996) 1.84
    1.83574 = sum of:
      1.83574 = product of:
        3.67148 = sum of:
          3.67148 = weight(author_txt:dorr in 950) [ClassicSimilarity], result of:
            3.67148 = score(doc=950,freq=1.0), product of:
              0.79412335 = queryWeight, product of:
                1.1430867 = boost
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.07513212 = queryNorm
              4.623312 = fieldWeight in 950, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.5 = fieldNorm(doc=950)
        0.5 = coord(1/2)
    
  4. Oard, D.W.; Dorr, B.J.: Evaluating cross-laguage text filtering effectiveness (1998) 1.84
    1.83574 = sum of:
      1.83574 = product of:
        3.67148 = sum of:
          3.67148 = weight(author_txt:dorr in 215) [ClassicSimilarity], result of:
            3.67148 = score(doc=215,freq=1.0), product of:
              0.79412335 = queryWeight, product of:
                1.1430867 = boost
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.07513212 = queryNorm
              4.623312 = fieldWeight in 215, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.5 = fieldNorm(doc=215)
        0.5 = coord(1/2)
    
  5. Dorr, B.J.; Gaasterland, T.: Exploiting aspectual features and connecting words for summarization-inspired temporal-relation extraction (2007) 1.84
    1.83574 = sum of:
      1.83574 = product of:
        3.67148 = sum of:
          3.67148 = weight(author_txt:dorr in 2951) [ClassicSimilarity], result of:
            3.67148 = score(doc=2951,freq=1.0), product of:
              0.79412335 = queryWeight, product of:
                1.1430867 = boost
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.07513212 = queryNorm
              4.623312 = fieldWeight in 2951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.246624 = idf(docFreq=10, maxDocs=41962)
                0.5 = fieldNorm(doc=2951)
        0.5 = coord(1/2)
    

Similar documents (content)

  1. Díaz, A.; Gervás, P.: User-model based personalized summarization (2007) 0.27
    0.2701014 = sum of:
      0.2701014 = product of:
        0.75028163 = sum of:
          0.066801846 = weight(abstract_txt:judge in 2953) [ClassicSimilarity], result of:
            0.066801846 = score(doc=2953,freq=1.0), product of:
              0.13476694 = queryWeight, product of:
                7.930948 = idf(docFreq=40, maxDocs=41962)
                0.01699254 = queryNorm
              0.49568424 = fieldWeight in 2953, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.930948 = idf(docFreq=40, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.018416231 = weight(abstract_txt:that in 2953) [ClassicSimilarity], result of:
            0.018416231 = score(doc=2953,freq=6.0), product of:
              0.04986867 = queryWeight, product of:
                1.2166123 = boost
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.01699254 = queryNorm
              0.3692946 = fieldWeight in 2953, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.103045665 = weight(abstract_txt:summary in 2953) [ClassicSimilarity], result of:
            0.103045665 = score(doc=2953,freq=2.0), product of:
              0.17991921 = queryWeight, product of:
                1.6340377 = boost
                6.479734 = idf(docFreq=174, maxDocs=41962)
                0.01699254 = queryNorm
              0.572733 = fieldWeight in 2953, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.479734 = idf(docFreq=174, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.13944523 = weight(abstract_txt:summarization in 2953) [ClassicSimilarity], result of:
            0.13944523 = score(doc=2953,freq=2.0), product of:
              0.22012033 = queryWeight, product of:
                1.8073963 = boost
                7.167183 = idf(docFreq=87, maxDocs=41962)
                0.01699254 = queryNorm
              0.63349545 = fieldWeight in 2953, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.167183 = idf(docFreq=87, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.0706872 = weight(abstract_txt:user in 2953) [ClassicSimilarity], result of:
            0.0706872 = score(doc=2953,freq=7.0), product of:
              0.11612852 = queryWeight, product of:
                1.8565547 = boost
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.01699254 = queryNorm
              0.608698 = fieldWeight in 2953, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.03245334 = weight(abstract_txt:than in 2953) [ClassicSimilarity], result of:
            0.03245334 = score(doc=2953,freq=1.0), product of:
              0.13220584 = queryWeight, product of:
                1.9809049 = boost
                3.9276135 = idf(docFreq=2245, maxDocs=41962)
                0.01699254 = queryNorm
              0.24547584 = fieldWeight in 2953, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.9276135 = idf(docFreq=2245, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.0487882 = weight(abstract_txt:evaluation in 2953) [ClassicSimilarity], result of:
            0.0487882 = score(doc=2953,freq=1.0), product of:
              0.17349525 = queryWeight, product of:
                2.2692487 = boost
                4.4993234 = idf(docFreq=1267, maxDocs=41962)
                0.01699254 = queryNorm
              0.2812077 = fieldWeight in 2953, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4993234 = idf(docFreq=1267, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.14086413 = weight(abstract_txt:relevance in 2953) [ClassicSimilarity], result of:
            0.14086413 = score(doc=2953,freq=3.0), product of:
              0.26274893 = queryWeight, product of:
                3.1222222 = boost
                4.952436 = idf(docFreq=805, maxDocs=41962)
                0.01699254 = queryNorm
              0.5361169 = fieldWeight in 2953, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.952436 = idf(docFreq=805, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
          0.12977983 = weight(abstract_txt:measure in 2953) [ClassicSimilarity], result of:
            0.12977983 = score(doc=2953,freq=1.0), product of:
              0.38128197 = queryWeight, product of:
                4.1200933 = boost
                5.446041 = idf(docFreq=491, maxDocs=41962)
                0.01699254 = queryNorm
              0.34037757 = fieldWeight in 2953, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.446041 = idf(docFreq=491, maxDocs=41962)
                0.0625 = fieldNorm(doc=2953)
        0.36 = coord(9/25)
    
  2. Shah, C.; Hendahewa, C.; González-Ibáñez, R.: Rain or shine? : forecasting search process performance in exploratory search tasks (2016) 0.26
    0.26197404 = sum of:
      0.26197404 = product of:
        0.72770566 = sum of:
          0.021731848 = weight(abstract_txt:current in 13) [ClassicSimilarity], result of:
            0.021731848 = score(doc=13,freq=1.0), product of:
              0.080315396 = queryWeight, product of:
                1.0917488 = boost
                4.3293014 = idf(docFreq=1502, maxDocs=41962)
                0.01699254 = queryNorm
              0.27058133 = fieldWeight in 13, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.3293014 = idf(docFreq=1502, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.046964053 = weight(abstract_txt:performance in 13) [ClassicSimilarity], result of:
            0.046964053 = score(doc=13,freq=3.0), product of:
              0.093083136 = queryWeight, product of:
                1.1753273 = boost
                4.66073 = idf(docFreq=1078, maxDocs=41962)
                0.01699254 = queryNorm
              0.5045388 = fieldWeight in 13, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.66073 = idf(docFreq=1078, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.010632615 = weight(abstract_txt:that in 13) [ClassicSimilarity], result of:
            0.010632615 = score(doc=13,freq=2.0), product of:
              0.04986867 = queryWeight, product of:
                1.2166123 = boost
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.01699254 = queryNorm
              0.21321233 = fieldWeight in 13, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.02534456 = weight(abstract_txt:based in 13) [ClassicSimilarity], result of:
            0.02534456 = score(doc=13,freq=2.0), product of:
              0.08898652 = queryWeight, product of:
                1.6251761 = boost
                3.2222967 = idf(docFreq=4546, maxDocs=41962)
                0.01699254 = queryNorm
              0.28481346 = fieldWeight in 13, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.2222967 = idf(docFreq=4546, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.0377839 = weight(abstract_txt:user in 13) [ClassicSimilarity], result of:
            0.0377839 = score(doc=13,freq=2.0), product of:
              0.11612852 = queryWeight, product of:
                1.8565547 = boost
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.01699254 = queryNorm
              0.32536277 = fieldWeight in 13, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.0487882 = weight(abstract_txt:evaluation in 13) [ClassicSimilarity], result of:
            0.0487882 = score(doc=13,freq=1.0), product of:
              0.17349525 = queryWeight, product of:
                2.2692487 = boost
                4.4993234 = idf(docFreq=1267, maxDocs=41962)
                0.01699254 = queryNorm
              0.2812077 = fieldWeight in 13, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4993234 = idf(docFreq=1267, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.08132795 = weight(abstract_txt:relevance in 13) [ClassicSimilarity], result of:
            0.08132795 = score(doc=13,freq=1.0), product of:
              0.26274893 = queryWeight, product of:
                3.1222222 = boost
                4.952436 = idf(docFreq=805, maxDocs=41962)
                0.01699254 = queryNorm
              0.30952725 = fieldWeight in 13, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.952436 = idf(docFreq=805, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.081944555 = weight(abstract_txt:task in 13) [ClassicSimilarity], result of:
            0.081944555 = score(doc=13,freq=1.0), product of:
              0.2640753 = queryWeight, product of:
                3.1300929 = boost
                4.9649205 = idf(docFreq=795, maxDocs=41962)
                0.01699254 = queryNorm
              0.31030753 = fieldWeight in 13, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.9649205 = idf(docFreq=795, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
          0.37318796 = weight(abstract_txt:prediction in 13) [ClassicSimilarity], result of:
            0.37318796 = score(doc=13,freq=3.0), product of:
              0.46700594 = queryWeight, product of:
                3.7230556 = boost
                7.3818398 = idf(docFreq=70, maxDocs=41962)
                0.01699254 = queryNorm
              0.79910755 = fieldWeight in 13, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                7.3818398 = idf(docFreq=70, maxDocs=41962)
                0.0625 = fieldNorm(doc=13)
        0.36 = coord(9/25)
    
  3. Dunlavy, D.M.; O'Leary, D.P.; Conroy, J.M.; Schlesinger, J.D.: QCS: A system for querying, clustering and summarizing documents (2007) 0.26
    0.25510255 = sum of:
      0.25510255 = product of:
        0.5797785 = sum of:
          0.019015366 = weight(abstract_txt:current in 2948) [ClassicSimilarity], result of:
            0.019015366 = score(doc=2948,freq=1.0), product of:
              0.080315396 = queryWeight, product of:
                1.0917488 = boost
                4.3293014 = idf(docFreq=1502, maxDocs=41962)
                0.01699254 = queryNorm
              0.23675866 = fieldWeight in 2948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.3293014 = idf(docFreq=1502, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.03355274 = weight(abstract_txt:performance in 2948) [ClassicSimilarity], result of:
            0.03355274 = score(doc=2948,freq=2.0), product of:
              0.093083136 = queryWeight, product of:
                1.1753273 = boost
                4.66073 = idf(docFreq=1078, maxDocs=41962)
                0.01699254 = queryNorm
              0.36045992 = fieldWeight in 2948, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.66073 = idf(docFreq=1078, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.024717893 = weight(abstract_txt:standard in 2948) [ClassicSimilarity], result of:
            0.024717893 = score(doc=2948,freq=1.0), product of:
              0.09566138 = queryWeight, product of:
                1.1914934 = boost
                4.724836 = idf(docFreq=1011, maxDocs=41962)
                0.01699254 = queryNorm
              0.25838947 = fieldWeight in 2948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.724836 = idf(docFreq=1011, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.006578595 = weight(abstract_txt:that in 2948) [ClassicSimilarity], result of:
            0.006578595 = score(doc=2948,freq=1.0), product of:
              0.04986867 = queryWeight, product of:
                1.2166123 = boost
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.01699254 = queryNorm
              0.1319184 = fieldWeight in 2948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.09016496 = weight(abstract_txt:summary in 2948) [ClassicSimilarity], result of:
            0.09016496 = score(doc=2948,freq=2.0), product of:
              0.17991921 = queryWeight, product of:
                1.6340377 = boost
                6.479734 = idf(docFreq=174, maxDocs=41962)
                0.01699254 = queryNorm
              0.50114137 = fieldWeight in 2948, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.479734 = idf(docFreq=174, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.17255466 = weight(abstract_txt:summarization in 2948) [ClassicSimilarity], result of:
            0.17255466 = score(doc=2948,freq=4.0), product of:
              0.22012033 = queryWeight, product of:
                1.8073963 = boost
                7.167183 = idf(docFreq=87, maxDocs=41962)
                0.01699254 = queryNorm
              0.78391063 = fieldWeight in 2948, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                7.167183 = idf(docFreq=87, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.023377594 = weight(abstract_txt:user in 2948) [ClassicSimilarity], result of:
            0.023377594 = score(doc=2948,freq=1.0), product of:
              0.11612852 = queryWeight, product of:
                1.8565547 = boost
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.01699254 = queryNorm
              0.20130795 = fieldWeight in 2948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.04934626 = weight(abstract_txt:automatic in 2948) [ClassicSimilarity], result of:
            0.04934626 = score(doc=2948,freq=1.0), product of:
              0.17361833 = queryWeight, product of:
                1.965924 = boost
                5.1972136 = idf(docFreq=630, maxDocs=41962)
                0.01699254 = queryNorm
              0.28422263 = fieldWeight in 2948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.1972136 = idf(docFreq=630, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.028396674 = weight(abstract_txt:than in 2948) [ClassicSimilarity], result of:
            0.028396674 = score(doc=2948,freq=1.0), product of:
              0.13220584 = queryWeight, product of:
                1.9809049 = boost
                3.9276135 = idf(docFreq=2245, maxDocs=41962)
                0.01699254 = queryNorm
              0.21479136 = fieldWeight in 2948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.9276135 = idf(docFreq=2245, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.060372323 = weight(abstract_txt:evaluation in 2948) [ClassicSimilarity], result of:
            0.060372323 = score(doc=2948,freq=2.0), product of:
              0.17349525 = queryWeight, product of:
                2.2692487 = boost
                4.4993234 = idf(docFreq=1267, maxDocs=41962)
                0.01699254 = queryNorm
              0.3479768 = fieldWeight in 2948, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.4993234 = idf(docFreq=1267, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
          0.07170149 = weight(abstract_txt:task in 2948) [ClassicSimilarity], result of:
            0.07170149 = score(doc=2948,freq=1.0), product of:
              0.2640753 = queryWeight, product of:
                3.1300929 = boost
                4.9649205 = idf(docFreq=795, maxDocs=41962)
                0.01699254 = queryNorm
              0.2715191 = fieldWeight in 2948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.9649205 = idf(docFreq=795, maxDocs=41962)
                0.0546875 = fieldNorm(doc=2948)
        0.44 = coord(11/25)
    
  4. Kar, M.; Nunes, S.; Ribeiro, C.: Summarization of changes in dynamic text collections using Latent Dirichlet Allocation model (2015) 0.22
    0.22412764 = sum of:
      0.22412764 = product of:
        0.56031907 = sum of:
          0.02512265 = weight(abstract_txt:world in 4677) [ClassicSimilarity], result of:
            0.02512265 = score(doc=4677,freq=2.0), product of:
              0.08506035 = queryWeight, product of:
                1.1235358 = boost
                4.455352 = idf(docFreq=1324, maxDocs=41962)
                0.01699254 = queryNorm
              0.29535088 = fieldWeight in 4677, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.455352 = idf(docFreq=1324, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.021186763 = weight(abstract_txt:standard in 4677) [ClassicSimilarity], result of:
            0.021186763 = score(doc=4677,freq=1.0), product of:
              0.09566138 = queryWeight, product of:
                1.1914934 = boost
                4.724836 = idf(docFreq=1011, maxDocs=41962)
                0.01699254 = queryNorm
              0.22147667 = fieldWeight in 4677, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.724836 = idf(docFreq=1011, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.00976668 = weight(abstract_txt:that in 4677) [ClassicSimilarity], result of:
            0.00976668 = score(doc=4677,freq=3.0), product of:
              0.04986867 = queryWeight, product of:
                1.2166123 = boost
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.01699254 = queryNorm
              0.19584802 = fieldWeight in 4677, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.030905759 = weight(abstract_txt:real in 4677) [ClassicSimilarity], result of:
            0.030905759 = score(doc=4677,freq=1.0), product of:
              0.12304193 = queryWeight, product of:
                1.351294 = boost
                5.3585215 = idf(docFreq=536, maxDocs=41962)
                0.01699254 = queryNorm
              0.2511807 = fieldWeight in 4677, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.3585215 = idf(docFreq=536, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.01900842 = weight(abstract_txt:based in 4677) [ClassicSimilarity], result of:
            0.01900842 = score(doc=4677,freq=2.0), product of:
              0.08898652 = queryWeight, product of:
                1.6251761 = boost
                3.2222967 = idf(docFreq=4546, maxDocs=41962)
                0.01699254 = queryNorm
              0.2136101 = fieldWeight in 4677, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.2222967 = idf(docFreq=4546, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.12219713 = weight(abstract_txt:summary in 4677) [ClassicSimilarity], result of:
            0.12219713 = score(doc=4677,freq=5.0), product of:
              0.17991921 = queryWeight, product of:
                1.6340377 = boost
                6.479734 = idf(docFreq=174, maxDocs=41962)
                0.01699254 = queryNorm
              0.67917776 = fieldWeight in 4677, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                6.479734 = idf(docFreq=174, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.1653617 = weight(abstract_txt:summarization in 4677) [ClassicSimilarity], result of:
            0.1653617 = score(doc=4677,freq=5.0), product of:
              0.22012033 = queryWeight, product of:
                1.8073963 = boost
                7.167183 = idf(docFreq=87, maxDocs=41962)
                0.01699254 = queryNorm
              0.7512332 = fieldWeight in 4677, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                7.167183 = idf(docFreq=87, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.020037938 = weight(abstract_txt:user in 4677) [ClassicSimilarity], result of:
            0.020037938 = score(doc=4677,freq=1.0), product of:
              0.11612852 = queryWeight, product of:
                1.8565547 = boost
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.01699254 = queryNorm
              0.17254966 = fieldWeight in 4677, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.059816696 = weight(abstract_txt:automatic in 4677) [ClassicSimilarity], result of:
            0.059816696 = score(doc=4677,freq=2.0), product of:
              0.17361833 = queryWeight, product of:
                1.965924 = boost
                5.1972136 = idf(docFreq=630, maxDocs=41962)
                0.01699254 = queryNorm
              0.34452984 = fieldWeight in 4677, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.1972136 = idf(docFreq=630, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
          0.08691532 = weight(abstract_txt:task in 4677) [ClassicSimilarity], result of:
            0.08691532 = score(doc=4677,freq=2.0), product of:
              0.2640753 = queryWeight, product of:
                3.1300929 = boost
                4.9649205 = idf(docFreq=795, maxDocs=41962)
                0.01699254 = queryNorm
              0.32913083 = fieldWeight in 4677, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.9649205 = idf(docFreq=795, maxDocs=41962)
                0.046875 = fieldNorm(doc=4677)
        0.4 = coord(10/25)
    
  5. Nunes, S.; Ribeiro, C.; David, G.: Term weighting based on document revision history (2011) 0.21
    0.21315232 = sum of:
      0.21315232 = product of:
        0.4844371 = sum of:
          0.066801846 = weight(abstract_txt:judge in 1947) [ClassicSimilarity], result of:
            0.066801846 = score(doc=1947,freq=1.0), product of:
              0.13476694 = queryWeight, product of:
                7.930948 = idf(docFreq=40, maxDocs=41962)
                0.01699254 = queryNorm
              0.49568424 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.930948 = idf(docFreq=40, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.07014232 = weight(abstract_txt:gold in 1947) [ClassicSimilarity], result of:
            0.07014232 = score(doc=1947,freq=1.0), product of:
              0.13922305 = queryWeight, product of:
                1.0163982 = boost
                8.061001 = idf(docFreq=35, maxDocs=41962)
                0.01699254 = queryNorm
              0.50381255 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.061001 = idf(docFreq=35, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.021731848 = weight(abstract_txt:current in 1947) [ClassicSimilarity], result of:
            0.021731848 = score(doc=1947,freq=1.0), product of:
              0.080315396 = queryWeight, product of:
                1.0917488 = boost
                4.3293014 = idf(docFreq=1502, maxDocs=41962)
                0.01699254 = queryNorm
              0.27058133 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.3293014 = idf(docFreq=1502, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.023685861 = weight(abstract_txt:world in 1947) [ClassicSimilarity], result of:
            0.023685861 = score(doc=1947,freq=1.0), product of:
              0.08506035 = queryWeight, product of:
                1.1235358 = boost
                4.455352 = idf(docFreq=1324, maxDocs=41962)
                0.01699254 = queryNorm
              0.2784595 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.455352 = idf(docFreq=1324, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.02824902 = weight(abstract_txt:standard in 1947) [ClassicSimilarity], result of:
            0.02824902 = score(doc=1947,freq=1.0), product of:
              0.09566138 = queryWeight, product of:
                1.1914934 = boost
                4.724836 = idf(docFreq=1011, maxDocs=41962)
                0.01699254 = queryNorm
              0.29530224 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.724836 = idf(docFreq=1011, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.015036789 = weight(abstract_txt:that in 1947) [ClassicSimilarity], result of:
            0.015036789 = score(doc=1947,freq=4.0), product of:
              0.04986867 = queryWeight, product of:
                1.2166123 = boost
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.01699254 = queryNorm
              0.30152777 = fieldWeight in 1947, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                2.4122221 = idf(docFreq=10221, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.041207675 = weight(abstract_txt:real in 1947) [ClassicSimilarity], result of:
            0.041207675 = score(doc=1947,freq=1.0), product of:
              0.12304193 = queryWeight, product of:
                1.351294 = boost
                5.3585215 = idf(docFreq=536, maxDocs=41962)
                0.01699254 = queryNorm
              0.3349076 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.3585215 = idf(docFreq=536, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.04316334 = weight(abstract_txt:measures in 1947) [ClassicSimilarity], result of:
            0.04316334 = score(doc=1947,freq=1.0), product of:
              0.12690471 = queryWeight, product of:
                1.3723414 = boost
                5.441984 = idf(docFreq=493, maxDocs=41962)
                0.01699254 = queryNorm
              0.340124 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.441984 = idf(docFreq=493, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.017921312 = weight(abstract_txt:based in 1947) [ClassicSimilarity], result of:
            0.017921312 = score(doc=1947,freq=1.0), product of:
              0.08898652 = queryWeight, product of:
                1.6251761 = boost
                3.2222967 = idf(docFreq=4546, maxDocs=41962)
                0.01699254 = queryNorm
              0.20139354 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.2222967 = idf(docFreq=4546, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.02671725 = weight(abstract_txt:user in 1947) [ClassicSimilarity], result of:
            0.02671725 = score(doc=1947,freq=1.0), product of:
              0.11612852 = queryWeight, product of:
                1.8565547 = boost
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.01699254 = queryNorm
              0.23006622 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6810596 = idf(docFreq=2873, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
          0.12977983 = weight(abstract_txt:measure in 1947) [ClassicSimilarity], result of:
            0.12977983 = score(doc=1947,freq=1.0), product of:
              0.38128197 = queryWeight, product of:
                4.1200933 = boost
                5.446041 = idf(docFreq=491, maxDocs=41962)
                0.01699254 = queryNorm
              0.34037757 = fieldWeight in 1947, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.446041 = idf(docFreq=491, maxDocs=41962)
                0.0625 = fieldNorm(doc=1947)
        0.44 = coord(11/25)