Document (#32437)

Author
Wacholder, N.
Kelly, D.
Kantor, P.
Rittman, R.
Sun, Y.
Bai, B.
Small, S.
Yamrom, B.
Strzalkowski, T.
Title
¬A model for quantitative evaluation of an end-to-end question-answering system
Source
Journal of the American Society for Information Science and Technology. 58(2007) no.8, S.1082-1099
Year
2007
Abstract
We describe a procedure for quantitative evaluation of interactive question-answering systems and illustrate it with application to the High-Quality Interactive QuestionAnswering (HITIQA) system. Our objectives were (a) to design a method to realistically and reliably assess interactive question-answering systems by comparing the quality of reports produced using different systems, (b) to conduct a pilot test of this method, and (c) to perform a formative evaluation of the HITIQA system. Far more important than the specific information gathered from this pilot evaluation is the development of (a) a protocol for evaluating an emerging technology, (b) reusable assessment instruments, and (c) the knowledge gained in conducting the evaluation. We conclude that this method, which uses a surprisingly small number of subjects and does not rely on predetermined relevance judgments, measures the impact of system change on work produced by users. Therefore this method can be used to compare the product of interactive systems that use different underlying technologies.
Object
HITIQA

Similar documents (author)

  1. Kelly, D.; Wacholder, N.; Rittman, R.; Sun, Y.; Kantor, P.; Small, S.; Strzalkowski, T.: Using interview data to identify evaluation criteria for interactive, analytical question-answering systems (2007) 4.94
    4.941671 = sum of:
      4.941671 = sum of:
        0.76793236 = weight(author_txt:kantor in 2333) [ClassicSimilarity], result of:
          0.76793236 = score(doc=2333,freq=1.0), product of:
            0.37696943 = queryWeight, product of:
              8.148484 = idf(docFreq=33, maxDocs=43254)
              0.04626252 = queryNorm
            2.037121 = fieldWeight in 2333, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              8.148484 = idf(docFreq=33, maxDocs=43254)
              0.25 = fieldNorm(doc=2333)
        0.8706775 = weight(author_txt:kelly in 2333) [ClassicSimilarity], result of:
          0.8706775 = score(doc=2333,freq=1.0), product of:
            0.40988532 = queryWeight, product of:
              1.042745 = boost
              8.496791 = idf(docFreq=23, maxDocs=43254)
              0.04626252 = queryNorm
            2.1241977 = fieldWeight in 2333, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              8.496791 = idf(docFreq=23, maxDocs=43254)
              0.25 = fieldNorm(doc=2333)
        1.0013657 = weight(author_txt:small in 2333) [ClassicSimilarity], result of:
          1.0013657 = score(doc=2333,freq=1.0), product of:
            0.44993794 = queryWeight, product of:
              1.0925045 = boost
              8.902256 = idf(docFreq=15, maxDocs=43254)
              0.04626252 = queryNorm
            2.225564 = fieldWeight in 2333, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              8.902256 = idf(docFreq=15, maxDocs=43254)
              0.25 = fieldNorm(doc=2333)
        1.133204 = weight(author_txt:strzalkowski in 2333) [ClassicSimilarity], result of:
          1.133204 = score(doc=2333,freq=1.0), product of:
            0.4886106 = queryWeight, product of:
              1.1384878 = boost
              9.27695 = idf(docFreq=10, maxDocs=43254)
              0.04626252 = queryNorm
            2.3192375 = fieldWeight in 2333, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.27695 = idf(docFreq=10, maxDocs=43254)
              0.25 = fieldNorm(doc=2333)
        1.1684912 = weight(author_txt:wacholder in 2333) [ClassicSimilarity], result of:
          1.1684912 = score(doc=2333,freq=1.0), product of:
            0.49870202 = queryWeight, product of:
              1.1501845 = boost
              9.37226 = idf(docFreq=9, maxDocs=43254)
              0.04626252 = queryNorm
            2.343065 = fieldWeight in 2333, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.37226 = idf(docFreq=9, maxDocs=43254)
              0.25 = fieldNorm(doc=2333)
    
  2. Ng, K.B.; Kantor, P.B.; Strzalkowski, T.; Wacholder, N.; Tang, R.; Bai, B.; Rittman,; Song, P.; Sun, Y.: Automated judgment of document qualities (2006) 1.61
    1.6115545 = sum of:
      1.6115545 = product of:
        2.685924 = sum of:
          0.6719408 = weight(author_txt:kantor in 1308) [ClassicSimilarity], result of:
            0.6719408 = score(doc=1308,freq=1.0), product of:
              0.37696943 = queryWeight, product of:
                8.148484 = idf(docFreq=33, maxDocs=43254)
                0.04626252 = queryNorm
              1.782481 = fieldWeight in 1308, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.148484 = idf(docFreq=33, maxDocs=43254)
                0.21875 = fieldNorm(doc=1308)
          0.99155354 = weight(author_txt:strzalkowski in 1308) [ClassicSimilarity], result of:
            0.99155354 = score(doc=1308,freq=1.0), product of:
              0.4886106 = queryWeight, product of:
                1.1384878 = boost
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.04626252 = queryNorm
              2.0293329 = fieldWeight in 1308, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.21875 = fieldNorm(doc=1308)
          1.0224298 = weight(author_txt:wacholder in 1308) [ClassicSimilarity], result of:
            1.0224298 = score(doc=1308,freq=1.0), product of:
              0.49870202 = queryWeight, product of:
                1.1501845 = boost
                9.37226 = idf(docFreq=9, maxDocs=43254)
                0.04626252 = queryNorm
              2.0501819 = fieldWeight in 1308, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.37226 = idf(docFreq=9, maxDocs=43254)
                0.21875 = fieldNorm(doc=1308)
        0.6 = coord(3/5)
    
  3. Wacholder, N.: Interactive query formulation (2011) 0.58
    0.5842456 = sum of:
      0.5842456 = product of:
        2.9212282 = sum of:
          2.9212282 = weight(author_txt:wacholder in 661) [ClassicSimilarity], result of:
            2.9212282 = score(doc=661,freq=1.0), product of:
              0.49870202 = queryWeight, product of:
                1.1501845 = boost
                9.37226 = idf(docFreq=9, maxDocs=43254)
                0.04626252 = queryNorm
              5.8576627 = fieldWeight in 661, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.37226 = idf(docFreq=9, maxDocs=43254)
                0.625 = fieldNorm(doc=661)
        0.2 = coord(1/5)
    
  4. Strzalkowski, T.: Natural language information retrieval (1995) 0.57
    0.566602 = sum of:
      0.566602 = product of:
        2.83301 = sum of:
          2.83301 = weight(author_txt:strzalkowski in 2983) [ClassicSimilarity], result of:
            2.83301 = score(doc=2983,freq=1.0), product of:
              0.4886106 = queryWeight, product of:
                1.1384878 = boost
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.04626252 = queryNorm
              5.798094 = fieldWeight in 2983, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.625 = fieldNorm(doc=2983)
        0.2 = coord(1/5)
    
  5. Strzalkowski, T.: Reversible grammar in natural langugae processing (19??) 0.57
    0.566602 = sum of:
      0.566602 = product of:
        2.83301 = sum of:
          2.83301 = weight(author_txt:strzalkowski in 6362) [ClassicSimilarity], result of:
            2.83301 = score(doc=6362,freq=1.0), product of:
              0.4886106 = queryWeight, product of:
                1.1384878 = boost
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.04626252 = queryNorm
              5.798094 = fieldWeight in 6362, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.625 = fieldNorm(doc=6362)
        0.2 = coord(1/5)
    

Similar documents (content)

  1. Sun, Y.; Kantor, P.B.; Morse, E.L.: Using cross-evaluation to evaluate interactive QA systems (2011) 0.63
    0.6265346 = sum of:
      0.6265346 = product of:
        1.3052804 = sum of:
          0.07379669 = weight(abstract_txt:judgments in 1209) [ClassicSimilarity], result of:
            0.07379669 = score(doc=1209,freq=1.0), product of:
              0.13664727 = queryWeight, product of:
                1.0393487 = boost
                6.912671 = idf(docFreq=116, maxDocs=43254)
                0.019019267 = queryNorm
              0.5400524 = fieldWeight in 1209, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.912671 = idf(docFreq=116, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.09862944 = weight(abstract_txt:surprisingly in 1209) [ClassicSimilarity], result of:
            0.09862944 = score(doc=1209,freq=1.0), product of:
              0.16579854 = queryWeight, product of:
                1.1448568 = boost
                7.614402 = idf(docFreq=57, maxDocs=43254)
                0.019019267 = queryNorm
              0.59487516 = fieldWeight in 1209, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.614402 = idf(docFreq=57, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.15144202 = weight(abstract_txt:predetermined in 1209) [ClassicSimilarity], result of:
            0.15144202 = score(doc=1209,freq=1.0), product of:
              0.22066864 = queryWeight, product of:
                1.320782 = boost
                8.784473 = idf(docFreq=17, maxDocs=43254)
                0.019019267 = queryNorm
              0.686287 = fieldWeight in 1209, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.784473 = idf(docFreq=17, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.045585822 = weight(abstract_txt:quality in 1209) [ClassicSimilarity], result of:
            0.045585822 = score(doc=1209,freq=1.0), product of:
              0.12487425 = queryWeight, product of:
                1.4051162 = boost
                4.672689 = idf(docFreq=1098, maxDocs=43254)
                0.019019267 = queryNorm
              0.36505383 = fieldWeight in 1209, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.672689 = idf(docFreq=1098, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.022249462 = weight(abstract_txt:this in 1209) [ClassicSimilarity], result of:
            0.022249462 = score(doc=1209,freq=3.0), product of:
              0.067624226 = queryWeight, product of:
                1.4623184 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.019019267 = queryNorm
              0.32901615 = fieldWeight in 1209, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.034039948 = weight(abstract_txt:system in 1209) [ClassicSimilarity], result of:
            0.034039948 = score(doc=1209,freq=1.0), product of:
              0.12949578 = queryWeight, product of:
                2.0235717 = boost
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.019019267 = queryNorm
              0.2628653 = fieldWeight in 1209, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.07948281 = weight(abstract_txt:systems in 1209) [ClassicSimilarity], result of:
            0.07948281 = score(doc=1209,freq=5.0), product of:
              0.13328722 = queryWeight, product of:
                2.0529816 = boost
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.019019267 = queryNorm
              0.5963273 = fieldWeight in 1209, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.09583133 = weight(abstract_txt:question in 1209) [ClassicSimilarity], result of:
            0.09583133 = score(doc=1209,freq=1.0), product of:
              0.23457862 = queryWeight, product of:
                2.3586617 = boost
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.019019267 = queryNorm
              0.4085254 = fieldWeight in 1209, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.14221855 = weight(abstract_txt:method in 1209) [ClassicSimilarity], result of:
            0.14221855 = score(doc=1209,freq=3.0), product of:
              0.23291285 = queryWeight, product of:
                2.7138605 = boost
                4.5124474 = idf(docFreq=1289, maxDocs=43254)
                0.019019267 = queryNorm
              0.6106084 = fieldWeight in 1209, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.5124474 = idf(docFreq=1289, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.20266387 = weight(abstract_txt:answering in 1209) [ClassicSimilarity], result of:
            0.20266387 = score(doc=1209,freq=1.0), product of:
              0.3864865 = queryWeight, product of:
                3.0275311 = boost
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.019019267 = queryNorm
              0.524375 = fieldWeight in 1209, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.21632423 = weight(abstract_txt:interactive in 1209) [ClassicSimilarity], result of:
            0.21632423 = score(doc=1209,freq=2.0), product of:
              0.35263315 = queryWeight, product of:
                3.3392763 = boost
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.019019267 = queryNorm
              0.61345404 = fieldWeight in 1209, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
          0.14301638 = weight(abstract_txt:evaluation in 1209) [ClassicSimilarity], result of:
            0.14301638 = score(doc=1209,freq=2.0), product of:
              0.28827935 = queryWeight, product of:
                3.3756125 = boost
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.019019267 = queryNorm
              0.49610344 = fieldWeight in 1209, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.078125 = fieldNorm(doc=1209)
        0.48 = coord(12/25)
    
  2. Kelly, D.; Wacholder, N.; Rittman, R.; Sun, Y.; Kantor, P.; Small, S.; Strzalkowski, T.: Using interview data to identify evaluation criteria for interactive, analytical question-answering systems (2007) 0.35
    0.3475862 = sum of:
      0.3475862 = product of:
        1.0862069 = sum of:
          0.045585822 = weight(abstract_txt:quality in 2333) [ClassicSimilarity], result of:
            0.045585822 = score(doc=2333,freq=1.0), product of:
              0.12487425 = queryWeight, product of:
                1.4051162 = boost
                4.672689 = idf(docFreq=1098, maxDocs=43254)
                0.019019267 = queryNorm
              0.36505383 = fieldWeight in 2333, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.672689 = idf(docFreq=1098, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
          0.01816661 = weight(abstract_txt:this in 2333) [ClassicSimilarity], result of:
            0.01816661 = score(doc=2333,freq=2.0), product of:
              0.067624226 = queryWeight, product of:
                1.4623184 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.019019267 = queryNorm
              0.26864055 = fieldWeight in 2333, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
          0.04813976 = weight(abstract_txt:system in 2333) [ClassicSimilarity], result of:
            0.04813976 = score(doc=2333,freq=2.0), product of:
              0.12949578 = queryWeight, product of:
                2.0235717 = boost
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.019019267 = queryNorm
              0.37174767 = fieldWeight in 2333, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
          0.07109159 = weight(abstract_txt:systems in 2333) [ClassicSimilarity], result of:
            0.07109159 = score(doc=2333,freq=4.0), product of:
              0.13328722 = queryWeight, product of:
                2.0529816 = boost
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.019019267 = queryNorm
              0.5333714 = fieldWeight in 2333, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
          0.13552594 = weight(abstract_txt:question in 2333) [ClassicSimilarity], result of:
            0.13552594 = score(doc=2333,freq=2.0), product of:
              0.23457862 = queryWeight, product of:
                2.3586617 = boost
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.019019267 = queryNorm
              0.5777421 = fieldWeight in 2333, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
          0.28661 = weight(abstract_txt:answering in 2333) [ClassicSimilarity], result of:
            0.28661 = score(doc=2333,freq=2.0), product of:
              0.3864865 = queryWeight, product of:
                3.0275311 = boost
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.019019267 = queryNorm
              0.7415783 = fieldWeight in 2333, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
          0.30592865 = weight(abstract_txt:interactive in 2333) [ClassicSimilarity], result of:
            0.30592865 = score(doc=2333,freq=4.0), product of:
              0.35263315 = queryWeight, product of:
                3.3392763 = boost
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.019019267 = queryNorm
              0.867555 = fieldWeight in 2333, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
          0.17515858 = weight(abstract_txt:evaluation in 2333) [ClassicSimilarity], result of:
            0.17515858 = score(doc=2333,freq=3.0), product of:
              0.28827935 = queryWeight, product of:
                3.3756125 = boost
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.019019267 = queryNorm
              0.60760015 = fieldWeight in 2333, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.078125 = fieldNorm(doc=2333)
        0.32 = coord(8/25)
    
  3. López-Ostenero, F.; Peinado, V.; Gonzalo, J.; Verdejo, F.: Interactive question answering : Is Cross-Language harder than monolingual searching? (2008) 0.32
    0.31608763 = sum of:
      0.31608763 = product of:
        0.8780212 = sum of:
          0.022308217 = weight(abstract_txt:different in 4024) [ClassicSimilarity], result of:
            0.022308217 = score(doc=4024,freq=1.0), product of:
              0.077546634 = queryWeight, product of:
                1.1072795 = boost
                3.6822383 = idf(docFreq=2958, maxDocs=43254)
                0.019019267 = queryNorm
              0.28767487 = fieldWeight in 4024, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6822383 = idf(docFreq=2958, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.01816661 = weight(abstract_txt:this in 4024) [ClassicSimilarity], result of:
            0.01816661 = score(doc=4024,freq=2.0), product of:
              0.067624226 = queryWeight, product of:
                1.4623184 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.019019267 = queryNorm
              0.26864055 = fieldWeight in 4024, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.0975952 = weight(abstract_txt:quantitative in 4024) [ClassicSimilarity], result of:
            0.0975952 = score(doc=4024,freq=1.0), product of:
              0.20743021 = queryWeight, product of:
                1.8109723 = boost
                6.022356 = idf(docFreq=284, maxDocs=43254)
                0.019019267 = queryNorm
              0.47049657 = fieldWeight in 4024, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.022356 = idf(docFreq=284, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.034039948 = weight(abstract_txt:system in 4024) [ClassicSimilarity], result of:
            0.034039948 = score(doc=4024,freq=1.0), product of:
              0.12949578 = queryWeight, product of:
                2.0235717 = boost
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.019019267 = queryNorm
              0.2628653 = fieldWeight in 4024, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.050269343 = weight(abstract_txt:systems in 4024) [ClassicSimilarity], result of:
            0.050269343 = score(doc=4024,freq=2.0), product of:
              0.13328722 = queryWeight, product of:
                2.0529816 = boost
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.019019267 = queryNorm
              0.3771505 = fieldWeight in 4024, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.13552594 = weight(abstract_txt:question in 4024) [ClassicSimilarity], result of:
            0.13552594 = score(doc=4024,freq=2.0), product of:
              0.23457862 = queryWeight, product of:
                2.3586617 = boost
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.019019267 = queryNorm
              0.5777421 = fieldWeight in 4024, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.20266387 = weight(abstract_txt:answering in 4024) [ClassicSimilarity], result of:
            0.20266387 = score(doc=4024,freq=1.0), product of:
              0.3864865 = queryWeight, product of:
                3.0275311 = boost
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.019019267 = queryNorm
              0.524375 = fieldWeight in 4024, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.21632423 = weight(abstract_txt:interactive in 4024) [ClassicSimilarity], result of:
            0.21632423 = score(doc=4024,freq=2.0), product of:
              0.35263315 = queryWeight, product of:
                3.3392763 = boost
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.019019267 = queryNorm
              0.61345404 = fieldWeight in 4024, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
          0.10112785 = weight(abstract_txt:evaluation in 4024) [ClassicSimilarity], result of:
            0.10112785 = score(doc=4024,freq=1.0), product of:
              0.28827935 = queryWeight, product of:
                3.3756125 = boost
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.019019267 = queryNorm
              0.3507981 = fieldWeight in 4024, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.078125 = fieldNorm(doc=4024)
        0.36 = coord(9/25)
    
  4. Lin, J.; Katz, B.: Building a reusable test collection for question answering (2006) 0.31
    0.3143628 = sum of:
      0.3143628 = product of:
        0.9823837 = sum of:
          0.022308217 = weight(abstract_txt:different in 46) [ClassicSimilarity], result of:
            0.022308217 = score(doc=46,freq=1.0), product of:
              0.077546634 = queryWeight, product of:
                1.1072795 = boost
                3.6822383 = idf(docFreq=2958, maxDocs=43254)
                0.019019267 = queryNorm
              0.28767487 = fieldWeight in 46, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6822383 = idf(docFreq=2958, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
          0.14129873 = weight(abstract_txt:reusable in 46) [ClassicSimilarity], result of:
            0.14129873 = score(doc=46,freq=1.0), product of:
              0.21070194 = queryWeight, product of:
                1.2906102 = boost
                8.583802 = idf(docFreq=21, maxDocs=43254)
                0.019019267 = queryNorm
              0.67060953 = fieldWeight in 46, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.583802 = idf(docFreq=21, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
          0.022249462 = weight(abstract_txt:this in 46) [ClassicSimilarity], result of:
            0.022249462 = score(doc=46,freq=3.0), product of:
              0.067624226 = queryWeight, product of:
                1.4623184 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.019019267 = queryNorm
              0.32901615 = fieldWeight in 46, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
          0.04813976 = weight(abstract_txt:system in 46) [ClassicSimilarity], result of:
            0.04813976 = score(doc=46,freq=2.0), product of:
              0.12949578 = queryWeight, product of:
                2.0235717 = boost
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.019019267 = queryNorm
              0.37174767 = fieldWeight in 46, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
          0.050269343 = weight(abstract_txt:systems in 46) [ClassicSimilarity], result of:
            0.050269343 = score(doc=46,freq=2.0), product of:
              0.13328722 = queryWeight, product of:
                2.0529816 = boost
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.019019267 = queryNorm
              0.3771505 = fieldWeight in 46, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
          0.19166265 = weight(abstract_txt:question in 46) [ClassicSimilarity], result of:
            0.19166265 = score(doc=46,freq=4.0), product of:
              0.23457862 = queryWeight, product of:
                2.3586617 = boost
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.019019267 = queryNorm
              0.8170508 = fieldWeight in 46, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                5.229125 = idf(docFreq=629, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
          0.40532774 = weight(abstract_txt:answering in 46) [ClassicSimilarity], result of:
            0.40532774 = score(doc=46,freq=4.0), product of:
              0.3864865 = queryWeight, product of:
                3.0275311 = boost
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.019019267 = queryNorm
              1.04875 = fieldWeight in 46, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.7120004 = idf(docFreq=142, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
          0.10112785 = weight(abstract_txt:evaluation in 46) [ClassicSimilarity], result of:
            0.10112785 = score(doc=46,freq=1.0), product of:
              0.28827935 = queryWeight, product of:
                3.3756125 = boost
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.019019267 = queryNorm
              0.3507981 = fieldWeight in 46, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.078125 = fieldNorm(doc=46)
        0.32 = coord(8/25)
    
  5. Blandford, A.; Adams, A.; Attfield, S.; Buchanan, G.; Gow, J.; Makri, S.; Rimmer, J.; Warwick, C.: ¬The PRET A Rapporter framework : evaluating digital libraries from the perspective of information work (2008) 0.24
    0.24419901 = sum of:
      0.24419901 = product of:
        0.7631219 = sum of:
          0.066929676 = weight(abstract_txt:conduct in 4022) [ClassicSimilarity], result of:
            0.066929676 = score(doc=4022,freq=1.0), product of:
              0.12803307 = queryWeight, product of:
                1.0060554 = boost
                6.6912384 = idf(docFreq=145, maxDocs=43254)
                0.019019267 = queryNorm
              0.522753 = fieldWeight in 4022, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.6912384 = idf(docFreq=145, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
          0.078278586 = weight(abstract_txt:conducting in 4022) [ClassicSimilarity], result of:
            0.078278586 = score(doc=4022,freq=1.0), product of:
              0.1421254 = queryWeight, product of:
                1.0599775 = boost
                7.0498724 = idf(docFreq=101, maxDocs=43254)
                0.019019267 = queryNorm
              0.5507713 = fieldWeight in 4022, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.0498724 = idf(docFreq=101, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
          0.022308217 = weight(abstract_txt:different in 4022) [ClassicSimilarity], result of:
            0.022308217 = score(doc=4022,freq=1.0), product of:
              0.077546634 = queryWeight, product of:
                1.1072795 = boost
                3.6822383 = idf(docFreq=2958, maxDocs=43254)
                0.019019267 = queryNorm
              0.28767487 = fieldWeight in 4022, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6822383 = idf(docFreq=2958, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
          0.01816661 = weight(abstract_txt:this in 4022) [ClassicSimilarity], result of:
            0.01816661 = score(doc=4022,freq=2.0), product of:
              0.067624226 = queryWeight, product of:
                1.4623184 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.019019267 = queryNorm
              0.26864055 = fieldWeight in 4022, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
          0.05895892 = weight(abstract_txt:system in 4022) [ClassicSimilarity], result of:
            0.05895892 = score(doc=4022,freq=3.0), product of:
              0.12949578 = queryWeight, product of:
                2.0235717 = boost
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.019019267 = queryNorm
              0.45529607 = fieldWeight in 4022, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                3.364676 = idf(docFreq=4064, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
          0.07948281 = weight(abstract_txt:systems in 4022) [ClassicSimilarity], result of:
            0.07948281 = score(doc=4022,freq=5.0), product of:
              0.13328722 = queryWeight, product of:
                2.0529816 = boost
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.019019267 = queryNorm
              0.5963273 = fieldWeight in 4022, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                3.4135768 = idf(docFreq=3870, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
          0.15296432 = weight(abstract_txt:interactive in 4022) [ClassicSimilarity], result of:
            0.15296432 = score(doc=4022,freq=1.0), product of:
              0.35263315 = queryWeight, product of:
                3.3392763 = boost
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.019019267 = queryNorm
              0.4337775 = fieldWeight in 4022, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.552352 = idf(docFreq=455, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
          0.28603277 = weight(abstract_txt:evaluation in 4022) [ClassicSimilarity], result of:
            0.28603277 = score(doc=4022,freq=8.0), product of:
              0.28827935 = queryWeight, product of:
                3.3756125 = boost
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.019019267 = queryNorm
              0.9922069 = fieldWeight in 4022, product of:
                2.828427 = tf(freq=8.0), with freq of:
                  8.0 = termFreq=8.0
                4.490216 = idf(docFreq=1318, maxDocs=43254)
                0.078125 = fieldNorm(doc=4022)
        0.32 = coord(8/25)