Document (#32943)

Author
Hirao, T.
Okumura, M.
Yasuda, N.
Isozaki, H.
Title
Supervised automatic evaluation for summarization with voted regression model
Source
Information processing and management. 43(2007) no.6, S.1521-1535
Year
2007
Abstract
The high quality evaluation of generated summaries is needed if we are to improve automatic summarization systems. Although human evaluation provides better results than automatic evaluation methods, its cost is huge and it is difficult to reproduce the results. Therefore, we need an automatic method that simulates human evaluation if we are to improve our summarization system efficiently. Although automatic evaluation methods have been proposed, they are unreliable when used for individual summaries. To solve this problem, we propose a supervised automatic evaluation method based on a new regression model called the voted regression model (VRM). VRM has two characteristics: (1) model selection based on 'corrected AIC' to avoid multicollinearity, (2) voting by the selected models to alleviate the problem of overfitting. Evaluation results obtained for TSC3 and DUC2004 show that our method achieved error reductions of about 17-51% compared with conventional automatic evaluation methods. Moreover, our method obtained the highest correlation coefficients in several different experiments.
Theme
Automatisches Abstracting

Similar documents (content)

  1. Sjöbergh, J.: Older versions of the ROUGEeval summarization evaluation system were easier to fool (2007) 0.38
    0.3839339 = sum of:
      0.3839339 = product of:
        1.3711925 = sum of:
          0.04870383 = weight(abstract_txt:human in 940) [ClassicSimilarity], result of:
            0.04870383 = score(doc=940,freq=1.0), product of:
              0.09490445 = queryWeight, product of:
                1.3678108 = boost
                4.692005 = idf(docFreq=1101, maxDocs=44218)
                0.014787748 = queryNorm
              0.51318806 = fieldWeight in 940, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.692005 = idf(docFreq=1101, maxDocs=44218)
                0.109375 = fieldNorm(doc=940)
          0.23201044 = weight(abstract_txt:summaries in 940) [ClassicSimilarity], result of:
            0.23201044 = score(doc=940,freq=2.0), product of:
              0.21325843 = queryWeight, product of:
                2.0503871 = boost
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.014787748 = queryNorm
              1.0879309 = fieldWeight in 940, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.109375 = fieldNorm(doc=940)
          0.059731856 = weight(abstract_txt:model in 940) [ClassicSimilarity], result of:
            0.059731856 = score(doc=940,freq=1.0), product of:
              0.13700145 = queryWeight, product of:
                2.3241296 = boost
                3.986234 = idf(docFreq=2231, maxDocs=44218)
                0.014787748 = queryNorm
              0.43599433 = fieldWeight in 940, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.986234 = idf(docFreq=2231, maxDocs=44218)
                0.109375 = fieldNorm(doc=940)
          0.14893295 = weight(abstract_txt:method in 940) [ClassicSimilarity], result of:
            0.14893295 = score(doc=940,freq=3.0), product of:
              0.17466582 = queryWeight, product of:
                2.624229 = boost
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.014787748 = queryNorm
              0.85267365 = fieldWeight in 940, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.109375 = fieldNorm(doc=940)
          0.36293298 = weight(abstract_txt:summarization in 940) [ClassicSimilarity], result of:
            0.36293298 = score(doc=940,freq=2.0), product of:
              0.32896462 = queryWeight, product of:
                3.118911 = boost
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.014787748 = queryNorm
              1.1032584 = fieldWeight in 940, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.109375 = fieldNorm(doc=940)
          0.32732353 = weight(abstract_txt:automatic in 940) [ClassicSimilarity], result of:
            0.32732353 = score(doc=940,freq=2.0), product of:
              0.4072946 = queryWeight, product of:
                5.3011622 = boost
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.014787748 = queryNorm
              0.803653 = fieldWeight in 940, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.109375 = fieldNorm(doc=940)
          0.19155698 = weight(abstract_txt:evaluation in 940) [ClassicSimilarity], result of:
            0.19155698 = score(doc=940,freq=1.0), product of:
              0.39040375 = queryWeight, product of:
                5.8849936 = boost
                4.4860687 = idf(docFreq=1353, maxDocs=44218)
                0.014787748 = queryNorm
              0.49066377 = fieldWeight in 940, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4860687 = idf(docFreq=1353, maxDocs=44218)
                0.109375 = fieldNorm(doc=940)
        0.28 = coord(7/25)
    
  2. Ercan, G.; Cicekli, I.: Using lexical chains for keyword extraction (2007) 0.30
    0.2958834 = sum of:
      0.2958834 = product of:
        0.9246356 = sum of:
          0.050724827 = weight(abstract_txt:problem in 951) [ClassicSimilarity], result of:
            0.050724827 = score(doc=951,freq=2.0), product of:
              0.085772105 = queryWeight, product of:
                1.3003366 = boost
                4.460548 = idf(docFreq=1388, maxDocs=44218)
                0.014787748 = queryNorm
              0.5913907 = fieldWeight in 951, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.460548 = idf(docFreq=1388, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
          0.04425412 = weight(abstract_txt:although in 951) [ClassicSimilarity], result of:
            0.04425412 = score(doc=951,freq=1.0), product of:
              0.09866842 = queryWeight, product of:
                1.3946712 = boost
                4.7841444 = idf(docFreq=1004, maxDocs=44218)
                0.014787748 = queryNorm
              0.44851354 = fieldWeight in 951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.7841444 = idf(docFreq=1004, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
          0.025602149 = weight(abstract_txt:results in 951) [ClassicSimilarity], result of:
            0.025602149 = score(doc=951,freq=1.0), product of:
              0.07841944 = queryWeight, product of:
                1.5227908 = boost
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.014787748 = queryNorm
              0.32647708 = fieldWeight in 951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
          0.07710194 = weight(abstract_txt:obtained in 951) [ClassicSimilarity], result of:
            0.07710194 = score(doc=951,freq=1.0), product of:
              0.14286283 = queryWeight, product of:
                1.6781948 = boost
                5.756716 = idf(docFreq=379, maxDocs=44218)
                0.014787748 = queryNorm
              0.5396921 = fieldWeight in 951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.756716 = idf(docFreq=379, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
          0.14061958 = weight(abstract_txt:summaries in 951) [ClassicSimilarity], result of:
            0.14061958 = score(doc=951,freq=1.0), product of:
              0.21325843 = queryWeight, product of:
                2.0503871 = boost
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.014787748 = queryNorm
              0.6593858 = fieldWeight in 951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
          0.16797435 = weight(abstract_txt:supervised in 951) [ClassicSimilarity], result of:
            0.16797435 = score(doc=951,freq=1.0), product of:
              0.2400883 = queryWeight, product of:
                2.175546 = boost
                7.462781 = idf(docFreq=68, maxDocs=44218)
                0.014787748 = queryNorm
              0.69963574 = fieldWeight in 951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.462781 = idf(docFreq=68, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
          0.2199706 = weight(abstract_txt:summarization in 951) [ClassicSimilarity], result of:
            0.2199706 = score(doc=951,freq=1.0), product of:
              0.32896462 = queryWeight, product of:
                3.118911 = boost
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.014787748 = queryNorm
              0.66867554 = fieldWeight in 951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
          0.19838803 = weight(abstract_txt:automatic in 951) [ClassicSimilarity], result of:
            0.19838803 = score(doc=951,freq=1.0), product of:
              0.4072946 = queryWeight, product of:
                5.3011622 = boost
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.014787748 = queryNorm
              0.48708728 = fieldWeight in 951, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.09375 = fieldNorm(doc=951)
        0.32 = coord(8/25)
    
  3. Vanderwende, L.; Suzuki, H.; Brockett, J.M.; Nenkova, A.: Beyond SumBasic : task-focused summarization with sentence simplification and lexical expansion (2007) 0.26
    0.2581439 = sum of:
      0.2581439 = product of:
        0.9219425 = sum of:
          0.05566152 = weight(abstract_txt:human in 948) [ClassicSimilarity], result of:
            0.05566152 = score(doc=948,freq=4.0), product of:
              0.09490445 = queryWeight, product of:
                1.3678108 = boost
                4.692005 = idf(docFreq=1101, maxDocs=44218)
                0.014787748 = queryNorm
              0.58650064 = fieldWeight in 948, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.692005 = idf(docFreq=1101, maxDocs=44218)
                0.0625 = fieldNorm(doc=948)
          0.024137938 = weight(abstract_txt:results in 948) [ClassicSimilarity], result of:
            0.024137938 = score(doc=948,freq=2.0), product of:
              0.07841944 = queryWeight, product of:
                1.5227908 = boost
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.014787748 = queryNorm
              0.30780554 = fieldWeight in 948, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.0625 = fieldNorm(doc=948)
          0.1623735 = weight(abstract_txt:summaries in 948) [ClassicSimilarity], result of:
            0.1623735 = score(doc=948,freq=3.0), product of:
              0.21325843 = queryWeight, product of:
                2.0503871 = boost
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.014787748 = queryNorm
              0.7613931 = fieldWeight in 948, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.0625 = fieldNorm(doc=948)
          0.049135134 = weight(abstract_txt:method in 948) [ClassicSimilarity], result of:
            0.049135134 = score(doc=948,freq=1.0), product of:
              0.17466582 = queryWeight, product of:
                2.624229 = boost
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.014787748 = queryNorm
              0.28130937 = fieldWeight in 948, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.0625 = fieldNorm(doc=948)
          0.25400016 = weight(abstract_txt:summarization in 948) [ClassicSimilarity], result of:
            0.25400016 = score(doc=948,freq=3.0), product of:
              0.32896462 = queryWeight, product of:
                3.118911 = boost
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.014787748 = queryNorm
              0.77211994 = fieldWeight in 948, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.0625 = fieldNorm(doc=948)
          0.18704201 = weight(abstract_txt:automatic in 948) [ClassicSimilarity], result of:
            0.18704201 = score(doc=948,freq=2.0), product of:
              0.4072946 = queryWeight, product of:
                5.3011622 = boost
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.014787748 = queryNorm
              0.45923027 = fieldWeight in 948, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.0625 = fieldNorm(doc=948)
          0.18959223 = weight(abstract_txt:evaluation in 948) [ClassicSimilarity], result of:
            0.18959223 = score(doc=948,freq=3.0), product of:
              0.39040375 = queryWeight, product of:
                5.8849936 = boost
                4.4860687 = idf(docFreq=1353, maxDocs=44218)
                0.014787748 = queryNorm
              0.48563117 = fieldWeight in 948, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.4860687 = idf(docFreq=1353, maxDocs=44218)
                0.0625 = fieldNorm(doc=948)
        0.28 = coord(7/25)
    
  4. Saggion, H.; Lapalme, G.: Selective analysis for the automatic generation of summaries (2000) 0.25
    0.2516569 = sum of:
      0.2516569 = product of:
        0.89877456 = sum of:
          0.041746143 = weight(abstract_txt:human in 132) [ClassicSimilarity], result of:
            0.041746143 = score(doc=132,freq=1.0), product of:
              0.09490445 = queryWeight, product of:
                1.3678108 = boost
                4.692005 = idf(docFreq=1101, maxDocs=44218)
                0.014787748 = queryNorm
              0.43987548 = fieldWeight in 132, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.692005 = idf(docFreq=1101, maxDocs=44218)
                0.09375 = fieldNorm(doc=132)
          0.025602149 = weight(abstract_txt:results in 132) [ClassicSimilarity], result of:
            0.025602149 = score(doc=132,freq=1.0), product of:
              0.07841944 = queryWeight, product of:
                1.5227908 = boost
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.014787748 = queryNorm
              0.32647708 = fieldWeight in 132, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.09375 = fieldNorm(doc=132)
          0.07710194 = weight(abstract_txt:obtained in 132) [ClassicSimilarity], result of:
            0.07710194 = score(doc=132,freq=1.0), product of:
              0.14286283 = queryWeight, product of:
                1.6781948 = boost
                5.756716 = idf(docFreq=379, maxDocs=44218)
                0.014787748 = queryNorm
              0.5396921 = fieldWeight in 132, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.756716 = idf(docFreq=379, maxDocs=44218)
                0.09375 = fieldNorm(doc=132)
          0.14061958 = weight(abstract_txt:summaries in 132) [ClassicSimilarity], result of:
            0.14061958 = score(doc=132,freq=1.0), product of:
              0.21325843 = queryWeight, product of:
                2.0503871 = boost
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.014787748 = queryNorm
              0.6593858 = fieldWeight in 132, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.033448 = idf(docFreq=105, maxDocs=44218)
                0.09375 = fieldNorm(doc=132)
          0.10423136 = weight(abstract_txt:method in 132) [ClassicSimilarity], result of:
            0.10423136 = score(doc=132,freq=2.0), product of:
              0.17466582 = queryWeight, product of:
                2.624229 = boost
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.014787748 = queryNorm
              0.5967473 = fieldWeight in 132, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.09375 = fieldNorm(doc=132)
          0.31108537 = weight(abstract_txt:summarization in 132) [ClassicSimilarity], result of:
            0.31108537 = score(doc=132,freq=2.0), product of:
              0.32896462 = queryWeight, product of:
                3.118911 = boost
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.014787748 = queryNorm
              0.94565 = fieldWeight in 132, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.09375 = fieldNorm(doc=132)
          0.19838803 = weight(abstract_txt:automatic in 132) [ClassicSimilarity], result of:
            0.19838803 = score(doc=132,freq=1.0), product of:
              0.4072946 = queryWeight, product of:
                5.3011622 = boost
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.014787748 = queryNorm
              0.48708728 = fieldWeight in 132, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.09375 = fieldNorm(doc=132)
        0.28 = coord(7/25)
    
  5. Oh, H.; Nam, S.; Zhu, Y.: Structured abstract summarization of scientific articles : summarization using full-text section information (2023) 0.24
    0.24284686 = sum of:
      0.24284686 = product of:
        0.75889647 = sum of:
          0.032875605 = weight(abstract_txt:improve in 889) [ClassicSimilarity], result of:
            0.032875605 = score(doc=889,freq=1.0), product of:
              0.10605204 = queryWeight, product of:
                1.4459133 = boost
                4.9599204 = idf(docFreq=842, maxDocs=44218)
                0.014787748 = queryNorm
              0.30999503 = fieldWeight in 889, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.9599204 = idf(docFreq=842, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
          0.024137938 = weight(abstract_txt:results in 889) [ClassicSimilarity], result of:
            0.024137938 = score(doc=889,freq=2.0), product of:
              0.07841944 = queryWeight, product of:
                1.5227908 = boost
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.014787748 = queryNorm
              0.30780554 = fieldWeight in 889, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.482422 = idf(docFreq=3693, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
          0.02881808 = weight(abstract_txt:methods in 889) [ClassicSimilarity], result of:
            0.02881808 = score(doc=889,freq=1.0), product of:
              0.11119288 = queryWeight, product of:
                1.8132885 = boost
                4.146752 = idf(docFreq=1900, maxDocs=44218)
                0.014787748 = queryNorm
              0.259172 = fieldWeight in 889, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.146752 = idf(docFreq=1900, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
          0.03413249 = weight(abstract_txt:model in 889) [ClassicSimilarity], result of:
            0.03413249 = score(doc=889,freq=1.0), product of:
              0.13700145 = queryWeight, product of:
                2.3241296 = boost
                3.986234 = idf(docFreq=2231, maxDocs=44218)
                0.014787748 = queryNorm
              0.24913962 = fieldWeight in 889, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.986234 = idf(docFreq=2231, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
          0.049135134 = weight(abstract_txt:method in 889) [ClassicSimilarity], result of:
            0.049135134 = score(doc=889,freq=1.0), product of:
              0.17466582 = queryWeight, product of:
                2.624229 = boost
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.014787748 = queryNorm
              0.28130937 = fieldWeight in 889, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.50095 = idf(docFreq=1333, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
          0.2932941 = weight(abstract_txt:summarization in 889) [ClassicSimilarity], result of:
            0.2932941 = score(doc=889,freq=4.0), product of:
              0.32896462 = queryWeight, product of:
                3.118911 = boost
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.014787748 = queryNorm
              0.89156735 = fieldWeight in 889, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                7.132539 = idf(docFreq=95, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
          0.18704201 = weight(abstract_txt:automatic in 889) [ClassicSimilarity], result of:
            0.18704201 = score(doc=889,freq=2.0), product of:
              0.4072946 = queryWeight, product of:
                5.3011622 = boost
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.014787748 = queryNorm
              0.45923027 = fieldWeight in 889, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.1955976 = idf(docFreq=665, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
          0.10946113 = weight(abstract_txt:evaluation in 889) [ClassicSimilarity], result of:
            0.10946113 = score(doc=889,freq=1.0), product of:
              0.39040375 = queryWeight, product of:
                5.8849936 = boost
                4.4860687 = idf(docFreq=1353, maxDocs=44218)
                0.014787748 = queryNorm
              0.2803793 = fieldWeight in 889, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.4860687 = idf(docFreq=1353, maxDocs=44218)
                0.0625 = fieldNorm(doc=889)
        0.32 = coord(8/25)