Document (#32939)

Author

Hobson, S.P.
Dorr, B.J.
Monz, C.
Schwartz, R.

Title

Task-based evaluation of text summarization using Relevance Prediction

Source

Information processing and management. 43(2007) no.6, S.1482-1499

Year

2007

Abstract

This article introduces a new task-based evaluation measure called Relevance Prediction that is a more intuitive measure of an individual's performance on a real-world task than interannotator agreement. Relevance Prediction parallels what a user does in the real world task of browsing a set of documents using standard search tools, i.e., the user judges relevance based on a short summary and then that same user - not an independent user - decides whether to open (and judge) the corresponding document. This measure is shown to be a more reliable measure of task performance than LDC Agreement, a current gold-standard based measure used in the summarization evaluation community. Our goal is to provide a stable framework within which developers of new automatic measures may make stronger statistical statements about the effectiveness of their measures in predicting summary usefulness. We demonstrate - as a proof-of-concept methodology for automatic metric developers - that a current automatic evaluation measure has a better correlation with Relevance Prediction than with LDC Agreement and that the significance level for detected differences is higher for the former than for the latter.

Theme

Automatisches Abstracting

Similar documents (author)

1. Zajic, D.; Dorr, B.J.; Lin, J.; Schwartz, R.: Multi-candidate reduction : sentence compression as a tool for document summarization tasks (2007) 3.85

   3.8479462 = sum of:
     3.8479462 = sum of:
       1.53575 = weight(author_txt:schwartz in 944) [ClassicSimilarity], result of:
         1.53575 = score(doc=944,freq=1.0), product of:
           0.6057164 = queryWeight, product of:
             8.113368 = idf(docFreq=35, maxDocs=44218)
             0.07465659 = queryNorm
           2.5354276 = fieldWeight in 944, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.113368 = idf(docFreq=35, maxDocs=44218)
             0.3125 = fieldNorm(doc=944)
       2.3121963 = weight(author_txt:dorr in 944) [ClassicSimilarity], result of:
         2.3121963 = score(doc=944,freq=1.0), product of:
           0.79568064 = queryWeight, product of:
             1.1461322 = boost
             9.298992 = idf(docFreq=10, maxDocs=44218)
             0.07465659 = queryNorm
           2.905935 = fieldWeight in 944, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.298992 = idf(docFreq=10, maxDocs=44218)
             0.3125 = fieldNorm(doc=944)
   

2. Dorr, B.J.: Large-scale dictionary construction for foreign language tutoring and interlingual machine translation (1997) 2.31

   2.3121963 = sum of:
     2.3121963 = product of:
       4.6243925 = sum of:
         4.6243925 = weight(author_txt:dorr in 3244) [ClassicSimilarity], result of:
           4.6243925 = score(doc=3244,freq=1.0), product of:
             0.79568064 = queryWeight, product of:
               1.1461322 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.07465659 = queryNorm
             5.81187 = fieldWeight in 3244, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.625 = fieldNorm(doc=3244)
       0.5 = coord(1/2)
   

3. Dorr, B.J.; Olsen, M.B.: Multilingual generation : the role of telicity in lexical choice and syntactic realization (1996) 1.85

   1.849757 = sum of:
     1.849757 = product of:
       3.699514 = sum of:
         3.699514 = weight(author_txt:dorr in 536) [ClassicSimilarity], result of:
           3.699514 = score(doc=536,freq=1.0), product of:
             0.79568064 = queryWeight, product of:
               1.1461322 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.07465659 = queryNorm
             4.649496 = fieldWeight in 536, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.5 = fieldNorm(doc=536)
       0.5 = coord(1/2)
   

4. Oard, D.W.; Dorr, B.J.: Evaluating cross-laguage text filtering effectiveness (1998) 1.85

   1.849757 = sum of:
     1.849757 = product of:
       3.699514 = sum of:
         3.699514 = weight(author_txt:dorr in 6214) [ClassicSimilarity], result of:
           3.699514 = score(doc=6214,freq=1.0), product of:
             0.79568064 = queryWeight, product of:
               1.1461322 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.07465659 = queryNorm
             4.649496 = fieldWeight in 6214, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.5 = fieldNorm(doc=6214)
       0.5 = coord(1/2)
   

5. Dorr, B.J.; Gaasterland, T.: Exploiting aspectual features and connecting words for summarization-inspired temporal-relation extraction (2007) 1.85

   1.849757 = sum of:
     1.849757 = product of:
       3.699514 = sum of:
         3.699514 = weight(author_txt:dorr in 950) [ClassicSimilarity], result of:
           3.699514 = score(doc=950,freq=1.0), product of:
             0.79568064 = queryWeight, product of:
               1.1461322 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.07465659 = queryNorm
             4.649496 = fieldWeight in 950, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.5 = fieldNorm(doc=950)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Díaz, A.; Gervás, P.: User-model based personalized summarization (2007) 0.27

   0.2721908 = sum of:
     0.2721908 = product of:
       0.7560855 = sum of:
         0.06751208 = weight(abstract_txt:judge in 952) [ClassicSimilarity], result of:
           0.06751208 = score(doc=952,freq=1.0), product of:
             0.1365139 = queryWeight, product of:
               1.0028481 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.017203512 = queryNorm
             0.4945436 = fieldWeight in 952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.017762324 = weight(abstract_txt:that in 952) [ClassicSimilarity], result of:
           0.017762324 = score(doc=952,freq=6.0), product of:
             0.048965704 = queryWeight, product of:
               1.2012205 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017203512 = queryNorm
             0.36275032 = fieldWeight in 952, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.104446806 = weight(abstract_txt:summary in 952) [ClassicSimilarity], result of:
           0.104446806 = score(doc=952,freq=2.0), product of:
             0.18260781 = queryWeight, product of:
               1.640293 = boost
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.017203512 = queryNorm
             0.5719734 = fieldWeight in 952, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.13985732 = weight(abstract_txt:summarization in 952) [ClassicSimilarity], result of:
           0.13985732 = score(doc=952,freq=2.0), product of:
             0.22184315 = queryWeight, product of:
               1.8079431 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.017203512 = queryNorm
             0.6304333 = fieldWeight in 952, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.072079934 = weight(abstract_txt:user in 952) [ClassicSimilarity], result of:
           0.072079934 = score(doc=952,freq=7.0), product of:
             0.1183367 = queryWeight, product of:
               1.8673962 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.017203512 = queryNorm
             0.60910887 = fieldWeight in 952, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.03221211 = weight(abstract_txt:than in 952) [ClassicSimilarity], result of:
           0.03221211 = score(doc=952,freq=1.0), product of:
             0.13231893 = queryWeight, product of:
               1.9746392 = boost
               3.8950868 = idf(docFreq=2444, maxDocs=44218)
               0.017203512 = queryNorm
             0.24344292 = fieldWeight in 952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.8950868 = idf(docFreq=2444, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.04921137 = weight(abstract_txt:evaluation in 952) [ClassicSimilarity], result of:
           0.04921137 = score(doc=952,freq=1.0), product of:
             0.17551714 = queryWeight, product of:
               2.2742412 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017203512 = queryNorm
             0.2803793 = fieldWeight in 952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.14156878 = weight(abstract_txt:relevance in 952) [ClassicSimilarity], result of:
           0.14156878 = score(doc=952,freq=3.0), product of:
             0.26516557 = queryWeight, product of:
               3.12529 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.017203512 = queryNorm
             0.5338882 = fieldWeight in 952, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
         0.13143478 = weight(abstract_txt:measure in 952) [ClassicSimilarity], result of:
           0.13143478 = score(doc=952,freq=1.0), product of:
             0.38676444 = queryWeight, product of:
               4.1347156 = boost
               5.437306 = idf(docFreq=522, maxDocs=44218)
               0.017203512 = queryNorm
             0.33983162 = fieldWeight in 952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.437306 = idf(docFreq=522, maxDocs=44218)
               0.0625 = fieldNorm(doc=952)
       0.36 = coord(9/25)
   

2. Dunlavy, D.M.; O'Leary, D.P.; Conroy, J.M.; Schlesinger, J.D.: QCS: A system for querying, clustering and summarizing documents (2007) 0.26

   0.25612235 = sum of:
     0.25612235 = product of:
       0.5820963 = sum of:
         0.01893545 = weight(abstract_txt:current in 947) [ClassicSimilarity], result of:
           0.01893545 = score(doc=947,freq=1.0), product of:
             0.08055842 = queryWeight, product of:
               1.0894749 = boost
               4.298101 = idf(docFreq=1633, maxDocs=44218)
               0.017203512 = queryNorm
             0.23505239 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.298101 = idf(docFreq=1633, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.03348282 = weight(abstract_txt:performance in 947) [ClassicSimilarity], result of:
           0.03348282 = score(doc=947,freq=2.0), product of:
             0.09349719 = queryWeight, product of:
               1.1737106 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.017203512 = queryNorm
             0.3581158 = fieldWeight in 947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.025160052 = weight(abstract_txt:standard in 947) [ClassicSimilarity], result of:
           0.025160052 = score(doc=947,freq=1.0), product of:
             0.09736471 = queryWeight, product of:
               1.19774 = boost
               4.725219 = idf(docFreq=1065, maxDocs=44218)
               0.017203512 = queryNorm
             0.2584104 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.725219 = idf(docFreq=1065, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.006345009 = weight(abstract_txt:that in 947) [ClassicSimilarity], result of:
           0.006345009 = score(doc=947,freq=1.0), product of:
             0.048965704 = queryWeight, product of:
               1.2012205 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017203512 = queryNorm
             0.12958068 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.09139096 = weight(abstract_txt:summary in 947) [ClassicSimilarity], result of:
           0.09139096 = score(doc=947,freq=2.0), product of:
             0.18260781 = queryWeight, product of:
               1.640293 = boost
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.017203512 = queryNorm
             0.5004767 = fieldWeight in 947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.1730646 = weight(abstract_txt:summarization in 947) [ClassicSimilarity], result of:
           0.1730646 = score(doc=947,freq=4.0), product of:
             0.22184315 = queryWeight, product of:
               1.8079431 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.017203512 = queryNorm
             0.78012145 = fieldWeight in 947, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.023838198 = weight(abstract_txt:user in 947) [ClassicSimilarity], result of:
           0.023838198 = score(doc=947,freq=1.0), product of:
             0.1183367 = queryWeight, product of:
               1.8673962 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.017203512 = queryNorm
             0.20144382 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.028185595 = weight(abstract_txt:than in 947) [ClassicSimilarity], result of:
           0.028185595 = score(doc=947,freq=1.0), product of:
             0.13231893 = queryWeight, product of:
               1.9746392 = boost
               3.8950868 = idf(docFreq=2444, maxDocs=44218)
               0.017203512 = queryNorm
             0.21301256 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.8950868 = idf(docFreq=2444, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.050169937 = weight(abstract_txt:automatic in 947) [ClassicSimilarity], result of:
           0.050169937 = score(doc=947,freq=1.0), product of:
             0.17657125 = queryWeight, product of:
               1.9754561 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.017203512 = queryNorm
             0.28413424 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.060895965 = weight(abstract_txt:evaluation in 947) [ClassicSimilarity], result of:
           0.060895965 = score(doc=947,freq=2.0), product of:
             0.17551714 = queryWeight, product of:
               2.2742412 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017203512 = queryNorm
             0.34695166 = fieldWeight in 947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.070627704 = weight(abstract_txt:task in 947) [ClassicSimilarity], result of:
           0.070627704 = score(doc=947,freq=1.0), product of:
             0.26296055 = queryWeight, product of:
               3.1122687 = boost
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.017203512 = queryNorm
             0.2685867 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
       0.44 = coord(11/25)
   

3. Shah, C.; Hendahewa, C.; González-Ibáñez, R.: Rain or shine? : forecasting search process performance in exploratory search tasks (2016) 0.25

   0.2535383 = sum of:
     0.2535383 = product of:
       0.70427305 = sum of:
         0.021640513 = weight(abstract_txt:current in 3012) [ClassicSimilarity], result of:
           0.021640513 = score(doc=3012,freq=1.0), product of:
             0.08055842 = queryWeight, product of:
               1.0894749 = boost
               4.298101 = idf(docFreq=1633, maxDocs=44218)
               0.017203512 = queryNorm
             0.2686313 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.298101 = idf(docFreq=1633, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.046866182 = weight(abstract_txt:performance in 3012) [ClassicSimilarity], result of:
           0.046866182 = score(doc=3012,freq=3.0), product of:
             0.09349719 = queryWeight, product of:
               1.1737106 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.017203512 = queryNorm
             0.50125766 = fieldWeight in 3012, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.0102550825 = weight(abstract_txt:that in 3012) [ClassicSimilarity], result of:
           0.0102550825 = score(doc=3012,freq=2.0), product of:
             0.048965704 = queryWeight, product of:
               1.2012205 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017203512 = queryNorm
             0.20943399 = fieldWeight in 3012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.02497515 = weight(abstract_txt:based in 3012) [ClassicSimilarity], result of:
           0.02497515 = score(doc=3012,freq=2.0), product of:
             0.08863488 = queryWeight, product of:
               1.6161404 = boost
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.017203512 = queryNorm
             0.28177565 = fieldWeight in 3012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.038528346 = weight(abstract_txt:user in 3012) [ClassicSimilarity], result of:
           0.038528346 = score(doc=3012,freq=2.0), product of:
             0.1183367 = queryWeight, product of:
               1.8673962 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.017203512 = queryNorm
             0.32558239 = fieldWeight in 3012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.04921137 = weight(abstract_txt:evaluation in 3012) [ClassicSimilarity], result of:
           0.04921137 = score(doc=3012,freq=1.0), product of:
             0.17551714 = queryWeight, product of:
               2.2742412 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017203512 = queryNorm
             0.2803793 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.08071738 = weight(abstract_txt:task in 3012) [ClassicSimilarity], result of:
           0.08071738 = score(doc=3012,freq=1.0), product of:
             0.26296055 = queryWeight, product of:
               3.1122687 = boost
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.017203512 = queryNorm
             0.30695623 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.08173477 = weight(abstract_txt:relevance in 3012) [ClassicSimilarity], result of:
           0.08173477 = score(doc=3012,freq=1.0), product of:
             0.26516557 = queryWeight, product of:
               3.12529 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.017203512 = queryNorm
             0.3082405 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.35034424 = weight(abstract_txt:prediction in 3012) [ClassicSimilarity], result of:
           0.35034424 = score(doc=3012,freq=3.0), product of:
             0.45036587 = queryWeight, product of:
               3.6430025 = boost
               7.1860275 = idf(docFreq=90, maxDocs=44218)
               0.017203512 = queryNorm
             0.7779103 = fieldWeight in 3012, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.1860275 = idf(docFreq=90, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
       0.36 = coord(9/25)
   

4. Kannan, R.; Ghinea, G.; Swaminathan, S.: What do you wish to see? : A summarization system for movies based on user preferences (2015) 0.23

   0.23266463 = sum of:
     0.23266463 = product of:
       0.6462906 = sum of:
         0.05857118 = weight(abstract_txt:detected in 2683) [ClassicSimilarity], result of:
           0.05857118 = score(doc=2683,freq=1.0), product of:
             0.13573958 = queryWeight, product of:
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.017203512 = queryNorm
             0.43149668 = fieldWeight in 2683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.023675928 = weight(abstract_txt:performance in 2683) [ClassicSimilarity], result of:
           0.023675928 = score(doc=2683,freq=1.0), product of:
             0.09349719 = queryWeight, product of:
               1.1737106 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.017203512 = queryNorm
             0.2532261 = fieldWeight in 2683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.006345009 = weight(abstract_txt:that in 2683) [ClassicSimilarity], result of:
           0.006345009 = score(doc=2683,freq=1.0), product of:
             0.048965704 = queryWeight, product of:
               1.2012205 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017203512 = queryNorm
             0.12958068 = fieldWeight in 2683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.03829476 = weight(abstract_txt:measures in 2683) [ClassicSimilarity], result of:
           0.03829476 = score(doc=2683,freq=1.0), product of:
             0.12883091 = queryWeight, product of:
               1.3777543 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.017203512 = queryNorm
             0.2972482 = fieldWeight in 2683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.026764665 = weight(abstract_txt:based in 2683) [ClassicSimilarity], result of:
           0.026764665 = score(doc=2683,freq=3.0), product of:
             0.08863488 = queryWeight, product of:
               1.6161404 = boost
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.017203512 = queryNorm
             0.3019654 = fieldWeight in 2683, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.14450179 = weight(abstract_txt:summary in 2683) [ClassicSimilarity], result of:
           0.14450179 = score(doc=2683,freq=5.0), product of:
             0.18260781 = queryWeight, product of:
               1.640293 = boost
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.017203512 = queryNorm
             0.7913232 = fieldWeight in 2683, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.22894295 = weight(abstract_txt:summarization in 2683) [ClassicSimilarity], result of:
           0.22894295 = score(doc=2683,freq=7.0), product of:
             0.22184315 = queryWeight, product of:
               1.8079431 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.017203512 = queryNorm
             1.0320036 = fieldWeight in 2683, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.047676396 = weight(abstract_txt:user in 2683) [ClassicSimilarity], result of:
           0.047676396 = score(doc=2683,freq=4.0), product of:
             0.1183367 = queryWeight, product of:
               1.8673962 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.017203512 = queryNorm
             0.40288764 = fieldWeight in 2683, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
         0.07151792 = weight(abstract_txt:relevance in 2683) [ClassicSimilarity], result of:
           0.07151792 = score(doc=2683,freq=1.0), product of:
             0.26516557 = queryWeight, product of:
               3.12529 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.017203512 = queryNorm
             0.26971045 = fieldWeight in 2683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2683)
       0.36 = coord(9/25)
   

5. Kar, M.; Nunes, S.; Ribeiro, C.: Summarization of changes in dynamic text collections using Latent Dirichlet Allocation model (2015) 0.22

   0.2248079 = sum of:
     0.2248079 = product of:
       0.56201977 = sum of:
         0.025156708 = weight(abstract_txt:world in 2676) [ClassicSimilarity], result of:
           0.025156708 = score(doc=2676,freq=2.0), product of:
             0.085634924 = queryWeight, product of:
               1.1232779 = boost
               4.4314575 = idf(docFreq=1429, maxDocs=44218)
               0.017203512 = queryNorm
             0.29376692 = fieldWeight in 2676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4314575 = idf(docFreq=1429, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.02156576 = weight(abstract_txt:standard in 2676) [ClassicSimilarity], result of:
           0.02156576 = score(doc=2676,freq=1.0), product of:
             0.09736471 = queryWeight, product of:
               1.19774 = boost
               4.725219 = idf(docFreq=1065, maxDocs=44218)
               0.017203512 = queryNorm
             0.22149463 = fieldWeight in 2676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.725219 = idf(docFreq=1065, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.009419894 = weight(abstract_txt:that in 2676) [ClassicSimilarity], result of:
           0.009419894 = score(doc=2676,freq=3.0), product of:
             0.048965704 = queryWeight, product of:
               1.2012205 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.017203512 = queryNorm
             0.19237739 = fieldWeight in 2676, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.030575367 = weight(abstract_txt:real in 2676) [ClassicSimilarity], result of:
           0.030575367 = score(doc=2676,freq=1.0), product of:
             0.12287763 = queryWeight, product of:
               1.3455448 = boost
               5.308326 = idf(docFreq=594, maxDocs=44218)
               0.017203512 = queryNorm
             0.24882777 = fieldWeight in 2676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.308326 = idf(docFreq=594, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.018731363 = weight(abstract_txt:based in 2676) [ClassicSimilarity], result of:
           0.018731363 = score(doc=2676,freq=2.0), product of:
             0.08863488 = queryWeight, product of:
               1.6161404 = boost
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.017203512 = queryNorm
             0.21133174 = fieldWeight in 2676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.12385868 = weight(abstract_txt:summary in 2676) [ClassicSimilarity], result of:
           0.12385868 = score(doc=2676,freq=5.0), product of:
             0.18260781 = queryWeight, product of:
               1.640293 = boost
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.017203512 = queryNorm
             0.678277 = fieldWeight in 2676, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.16585039 = weight(abstract_txt:summarization in 2676) [ClassicSimilarity], result of:
           0.16585039 = score(doc=2676,freq=5.0), product of:
             0.22184315 = queryWeight, product of:
               1.8079431 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.017203512 = queryNorm
             0.747602 = fieldWeight in 2676, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.02043274 = weight(abstract_txt:user in 2676) [ClassicSimilarity], result of:
           0.02043274 = score(doc=2676,freq=1.0), product of:
             0.1183367 = queryWeight, product of:
               1.8673962 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.017203512 = queryNorm
             0.17266613 = fieldWeight in 2676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.060815148 = weight(abstract_txt:automatic in 2676) [ClassicSimilarity], result of:
           0.060815148 = score(doc=2676,freq=2.0), product of:
             0.17657125 = queryWeight, product of:
               1.9754561 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.017203512 = queryNorm
             0.3444227 = fieldWeight in 2676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.08561371 = weight(abstract_txt:task in 2676) [ClassicSimilarity], result of:
           0.08561371 = score(doc=2676,freq=2.0), product of:
             0.26296055 = queryWeight, product of:
               3.1122687 = boost
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.017203512 = queryNorm
             0.32557625 = fieldWeight in 2676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
       0.4 = coord(10/25)