Document (#32333)

Author

Kelly, D.
Wacholder, N.
Rittman, R.
Sun, Y.
Kantor, P.
Small, S.
Strzalkowski, T.

Title

Using interview data to identify evaluation criteria for interactive, analytical question-answering systems

Source

Journal of the American Society for Information Science and Technology. 58(2007) no.7, S.1032-1043

Year

2007

Abstract

The purpose of this work is to identify potential evaluation criteria for interactive, analytical question-answering (QA) systems by analyzing evaluative comments made by users of such a system. Qualitative data collected from intelligence analysts during interviews and focus groups were analyzed to identify common themes related to performance, use, and usability. These data were collected as part of an intensive, three-day evaluation workshop of the High-Quality Interactive Question Answering (HITIQA) system. Inductive coding and memoing were used to identify and categorize these data. Results suggest potential evaluation criteria for interactive, analytical QA systems, which can be used to guide the development and design of future systems and evaluations. This work contributes to studies of QA systems, information seeking and use behaviors, and interactive searching.

Similar documents (author)

1. Wacholder, N.; Kelly, D.; Kantor, P.; Rittman, R.; Sun, Y.; Bai, B.; Small, S.; Yamrom, B.; Strzalkowski, T.: ¬A model for quantitative evaluation of an end-to-end question-answering system (2007) 4.33

   4.334742 = sum of:
     4.334742 = sum of:
       0.6740695 = weight(author_txt:kantor in 435) [ClassicSimilarity], result of:
         0.6740695 = score(doc=435,freq=1.0), product of:
           0.37714344 = queryWeight, product of:
             8.1705265 = idf(docFreq=33, maxDocs=44218)
             0.046159014 = queryNorm
           1.7873027 = fieldWeight in 435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.1705265 = idf(docFreq=33, maxDocs=44218)
             0.21875 = fieldNorm(doc=435)
       0.7640026 = weight(author_txt:kelly in 435) [ClassicSimilarity], result of:
         0.7640026 = score(doc=435,freq=1.0), product of:
           0.40998378 = queryWeight, product of:
             1.0426296 = boost
             8.518833 = idf(docFreq=23, maxDocs=44218)
             0.046159014 = queryNorm
           1.8634948 = fieldWeight in 435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.518833 = idf(docFreq=23, maxDocs=44218)
             0.21875 = fieldNorm(doc=435)
       0.87836856 = weight(author_txt:small in 435) [ClassicSimilarity], result of:
         0.87836856 = score(doc=435,freq=1.0), product of:
           0.44994003 = queryWeight, product of:
             1.092255 = boost
             8.924298 = idf(docFreq=15, maxDocs=44218)
             0.046159014 = queryNorm
           1.9521903 = fieldWeight in 435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.924298 = idf(docFreq=15, maxDocs=44218)
             0.21875 = fieldNorm(doc=435)
       0.9937157 = weight(author_txt:strzalkowski in 435) [ClassicSimilarity], result of:
         0.9937157 = score(doc=435,freq=1.0), product of:
           0.48851538 = queryWeight, product of:
             1.1381142 = boost
             9.298992 = idf(docFreq=10, maxDocs=44218)
             0.046159014 = queryNorm
           2.0341544 = fieldWeight in 435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.298992 = idf(docFreq=10, maxDocs=44218)
             0.21875 = fieldNorm(doc=435)
       1.0245854 = weight(author_txt:wacholder in 435) [ClassicSimilarity], result of:
         1.0245854 = score(doc=435,freq=1.0), product of:
           0.49858078 = queryWeight, product of:
             1.1497793 = boost
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.046159014 = queryNorm
           2.0550036 = fieldWeight in 435, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.394302 = idf(docFreq=9, maxDocs=44218)
             0.21875 = fieldNorm(doc=435)
   

2. Ng, K.B.; Kantor, P.B.; Strzalkowski, T.; Wacholder, N.; Tang, R.; Bai, B.; Rittman,; Song, P.; Sun, Y.: Automated judgment of document qualities (2006) 1.62

   1.6154224 = sum of:
     1.6154224 = product of:
       2.6923704 = sum of:
         0.6740695 = weight(author_txt:kantor in 182) [ClassicSimilarity], result of:
           0.6740695 = score(doc=182,freq=1.0), product of:
             0.37714344 = queryWeight, product of:
               8.1705265 = idf(docFreq=33, maxDocs=44218)
               0.046159014 = queryNorm
             1.7873027 = fieldWeight in 182, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.1705265 = idf(docFreq=33, maxDocs=44218)
               0.21875 = fieldNorm(doc=182)
         0.9937157 = weight(author_txt:strzalkowski in 182) [ClassicSimilarity], result of:
           0.9937157 = score(doc=182,freq=1.0), product of:
             0.48851538 = queryWeight, product of:
               1.1381142 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046159014 = queryNorm
             2.0341544 = fieldWeight in 182, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.21875 = fieldNorm(doc=182)
         1.0245854 = weight(author_txt:wacholder in 182) [ClassicSimilarity], result of:
           1.0245854 = score(doc=182,freq=1.0), product of:
             0.49858078 = queryWeight, product of:
               1.1497793 = boost
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.046159014 = queryNorm
             2.0550036 = fieldWeight in 182, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.21875 = fieldNorm(doc=182)
       0.6 = coord(3/5)
   

3. Wacholder, N.: Interactive query formulation (2011) 0.59

   0.58547735 = sum of:
     0.58547735 = product of:
       2.9273868 = sum of:
         2.9273868 = weight(author_txt:wacholder in 4196) [ClassicSimilarity], result of:
           2.9273868 = score(doc=4196,freq=1.0), product of:
             0.49858078 = queryWeight, product of:
               1.1497793 = boost
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.046159014 = queryNorm
             5.871439 = fieldWeight in 4196, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.625 = fieldNorm(doc=4196)
       0.2 = coord(1/5)
   

4. Strzalkowski, T.: Natural language information retrieval (1995) 0.57

   0.5678376 = sum of:
     0.5678376 = product of:
       2.8391879 = sum of:
         2.8391879 = weight(author_txt:strzalkowski in 1914) [ClassicSimilarity], result of:
           2.8391879 = score(doc=1914,freq=1.0), product of:
             0.48851538 = queryWeight, product of:
               1.1381142 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046159014 = queryNorm
             5.81187 = fieldWeight in 1914, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.625 = fieldNorm(doc=1914)
       0.2 = coord(1/5)
   

5. Strzalkowski, T.: Reversible grammar in natural langugae processing (19??) 0.57

   0.5678376 = sum of:
     0.5678376 = product of:
       2.8391879 = sum of:
         2.8391879 = weight(author_txt:strzalkowski in 5293) [ClassicSimilarity], result of:
           2.8391879 = score(doc=5293,freq=1.0), product of:
             0.48851538 = queryWeight, product of:
               1.1381142 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046159014 = queryNorm
             5.81187 = fieldWeight in 5293, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.625 = fieldNorm(doc=5293)
       0.2 = coord(1/5)
   

Similar documents (content)

1. Wacholder, N.; Kelly, D.; Kantor, P.; Rittman, R.; Sun, Y.; Bai, B.; Small, S.; Yamrom, B.; Strzalkowski, T.: ¬A model for quantitative evaluation of an end-to-end question-answering system (2007) 0.40

   0.40196666 = sum of:
     0.40196666 = product of:
       1.116574 = sum of:
         0.016407864 = weight(abstract_txt:used in 435) [ClassicSimilarity], result of:
           0.016407864 = score(doc=435,freq=1.0), product of:
             0.06251919 = queryWeight, product of:
               1.0321218 = boost
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.018031579 = queryNorm
             0.26244524 = fieldWeight in 435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.028750712 = weight(abstract_txt:system in 435) [ClassicSimilarity], result of:
           0.028750712 = score(doc=435,freq=3.0), product of:
             0.06300437 = queryWeight, product of:
               1.0361189 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.018031579 = queryNorm
             0.45632887 = fieldWeight in 435, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.02364478 = weight(abstract_txt:work in 435) [ClassicSimilarity], result of:
           0.02364478 = score(doc=435,freq=1.0), product of:
             0.07976313 = queryWeight, product of:
               1.1658034 = boost
               3.7943997 = idf(docFreq=2703, maxDocs=44218)
               0.018031579 = queryNorm
             0.29643747 = fieldWeight in 435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7943997 = idf(docFreq=2703, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.032093577 = weight(abstract_txt:were in 435) [ClassicSimilarity], result of:
           0.032093577 = score(doc=435,freq=1.0), product of:
             0.111932054 = queryWeight, product of:
               1.6914032 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.018031579 = queryNorm
             0.28672373 = fieldWeight in 435, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.12967186 = weight(abstract_txt:question in 435) [ClassicSimilarity], result of:
           0.12967186 = score(doc=435,freq=2.0), product of:
             0.22537005 = queryWeight, product of:
               2.4000373 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.018031579 = queryNorm
             0.57537305 = fieldWeight in 435, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.085952125 = weight(abstract_txt:systems in 435) [ClassicSimilarity], result of:
           0.085952125 = score(doc=435,freq=4.0), product of:
             0.16122897 = queryWeight, product of:
               2.6206908 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.018031579 = queryNorm
             0.53310597 = fieldWeight in 435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.15630172 = weight(abstract_txt:evaluation in 435) [ClassicSimilarity], result of:
           0.15630172 = score(doc=435,freq=4.0), product of:
             0.22298612 = queryWeight, product of:
               2.756628 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.018031579 = queryNorm
             0.70094824 = fieldWeight in 435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.2728205 = weight(abstract_txt:answering in 435) [ClassicSimilarity], result of:
           0.2728205 = score(doc=435,freq=2.0), product of:
             0.37004146 = queryWeight, product of:
               3.0753539 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.018031579 = queryNorm
             0.73727006 = fieldWeight in 435, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
         0.37093085 = weight(abstract_txt:interactive in 435) [ClassicSimilarity], result of:
           0.37093085 = score(doc=435,freq=4.0), product of:
             0.42736664 = queryWeight, product of:
               4.266724 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.018031579 = queryNorm
             0.86794525 = fieldWeight in 435, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.078125 = fieldNorm(doc=435)
       0.36 = coord(9/25)
   

2. Pomerantz, J.: ¬A linguistic analysis of question taxonomies (2005) 0.24

   0.2432254 = sum of:
     0.2432254 = product of:
       0.8686621 = sum of:
         0.028373739 = weight(abstract_txt:work in 3465) [ClassicSimilarity], result of:
           0.028373739 = score(doc=3465,freq=1.0), product of:
             0.07976313 = queryWeight, product of:
               1.1658034 = boost
               3.7943997 = idf(docFreq=2703, maxDocs=44218)
               0.018031579 = queryNorm
             0.355725 = fieldWeight in 3465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7943997 = idf(docFreq=2703, maxDocs=44218)
               0.09375 = fieldNorm(doc=3465)
         0.0516499 = weight(abstract_txt:potential in 3465) [ClassicSimilarity], result of:
           0.0516499 = score(doc=3465,freq=1.0), product of:
             0.11891522 = queryWeight, product of:
               1.4234525 = boost
               4.632983 = idf(docFreq=1168, maxDocs=44218)
               0.018031579 = queryNorm
             0.43434218 = fieldWeight in 3465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.632983 = idf(docFreq=1168, maxDocs=44218)
               0.09375 = fieldNorm(doc=3465)
         0.03851229 = weight(abstract_txt:were in 3465) [ClassicSimilarity], result of:
           0.03851229 = score(doc=3465,freq=1.0), product of:
             0.111932054 = queryWeight, product of:
               1.6914032 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.018031579 = queryNorm
             0.34406847 = fieldWeight in 3465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.09375 = fieldNorm(doc=3465)
         0.24603505 = weight(abstract_txt:question in 3465) [ClassicSimilarity], result of:
           0.24603505 = score(doc=3465,freq=5.0), product of:
             0.22537005 = queryWeight, product of:
               2.4000373 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.018031579 = queryNorm
             1.0916936 = fieldWeight in 3465, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.09375 = fieldNorm(doc=3465)
         0.051571276 = weight(abstract_txt:systems in 3465) [ClassicSimilarity], result of:
           0.051571276 = score(doc=3465,freq=1.0), product of:
             0.16122897 = queryWeight, product of:
               2.6206908 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.018031579 = queryNorm
             0.3198636 = fieldWeight in 3465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.09375 = fieldNorm(doc=3465)
         0.12513524 = weight(abstract_txt:identify in 3465) [ClassicSimilarity], result of:
           0.12513524 = score(doc=3465,freq=1.0), product of:
             0.2702636 = queryWeight, product of:
               3.0348208 = boost
               4.9387927 = idf(docFreq=860, maxDocs=44218)
               0.018031579 = queryNorm
             0.4630118 = fieldWeight in 3465, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9387927 = idf(docFreq=860, maxDocs=44218)
               0.09375 = fieldNorm(doc=3465)
         0.32738456 = weight(abstract_txt:answering in 3465) [ClassicSimilarity], result of:
           0.32738456 = score(doc=3465,freq=2.0), product of:
             0.37004146 = queryWeight, product of:
               3.0753539 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.018031579 = queryNorm
             0.884724 = fieldWeight in 3465, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.09375 = fieldNorm(doc=3465)
       0.28 = coord(7/25)
   

3. Sun, Y.; Kantor, P.B.; Morse, E.L.: Using cross-evaluation to evaluate interactive QA systems (2011) 0.22

   0.22022541 = sum of:
     0.22022541 = product of:
       0.7865193 = sum of:
         0.016407864 = weight(abstract_txt:used in 4744) [ClassicSimilarity], result of:
           0.016407864 = score(doc=4744,freq=1.0), product of:
             0.06251919 = queryWeight, product of:
               1.0321218 = boost
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.018031579 = queryNorm
             0.26244524 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3592992 = idf(docFreq=4177, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.01659923 = weight(abstract_txt:system in 4744) [ClassicSimilarity], result of:
           0.01659923 = score(doc=4744,freq=1.0), product of:
             0.06300437 = queryWeight, product of:
               1.0361189 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.018031579 = queryNorm
             0.2634616 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.091691844 = weight(abstract_txt:question in 4744) [ClassicSimilarity], result of:
           0.091691844 = score(doc=4744,freq=1.0), product of:
             0.22537005 = queryWeight, product of:
               2.4000373 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.018031579 = queryNorm
             0.40685016 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.096097395 = weight(abstract_txt:systems in 4744) [ClassicSimilarity], result of:
           0.096097395 = score(doc=4744,freq=5.0), product of:
             0.16122897 = queryWeight, product of:
               2.6206908 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.018031579 = queryNorm
             0.5960306 = fieldWeight in 4744, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.11052201 = weight(abstract_txt:evaluation in 4744) [ClassicSimilarity], result of:
           0.11052201 = score(doc=4744,freq=2.0), product of:
             0.22298612 = queryWeight, product of:
               2.756628 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.018031579 = queryNorm
             0.49564525 = fieldWeight in 4744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.1929132 = weight(abstract_txt:answering in 4744) [ClassicSimilarity], result of:
           0.1929132 = score(doc=4744,freq=1.0), product of:
             0.37004146 = queryWeight, product of:
               3.0753539 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.018031579 = queryNorm
             0.5213286 = fieldWeight in 4744, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
         0.2622877 = weight(abstract_txt:interactive in 4744) [ClassicSimilarity], result of:
           0.2622877 = score(doc=4744,freq=2.0), product of:
             0.42736664 = queryWeight, product of:
               4.266724 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.018031579 = queryNorm
             0.61372995 = fieldWeight in 4744, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.078125 = fieldNorm(doc=4744)
       0.28 = coord(7/25)
   

4. Efthimiadis, E.N.: Interactive query expansion : a user-based evaluation in a relevance feedback environment (2000) 0.19

   0.1857711 = sum of:
     0.1857711 = product of:
       0.5805347 = sum of:
         0.04541769 = weight(abstract_txt:evaluations in 5701) [ClassicSimilarity], result of:
           0.04541769 = score(doc=5701,freq=1.0), product of:
             0.124085195 = queryWeight, product of:
               1.0281802 = boost
               6.6929407 = idf(docFreq=148, maxDocs=44218)
               0.018031579 = queryNorm
             0.3660202 = fieldWeight in 5701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6929407 = idf(docFreq=148, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
         0.011619462 = weight(abstract_txt:system in 5701) [ClassicSimilarity], result of:
           0.011619462 = score(doc=5701,freq=1.0), product of:
             0.06300437 = queryWeight, product of:
               1.0361189 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.018031579 = queryNorm
             0.18442312 = fieldWeight in 5701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
         0.050234396 = weight(abstract_txt:were in 5701) [ClassicSimilarity], result of:
           0.050234396 = score(doc=5701,freq=5.0), product of:
             0.111932054 = queryWeight, product of:
               1.6914032 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.018031579 = queryNorm
             0.44879365 = fieldWeight in 5701, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
         0.05388943 = weight(abstract_txt:collected in 5701) [ClassicSimilarity], result of:
           0.05388943 = score(doc=5701,freq=1.0), product of:
             0.17521948 = queryWeight, product of:
               1.7278874 = boost
               5.6238427 = idf(docFreq=433, maxDocs=44218)
               0.018031579 = queryNorm
             0.3075539 = fieldWeight in 5701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6238427 = idf(docFreq=433, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
         0.03182461 = weight(abstract_txt:data in 5701) [ClassicSimilarity], result of:
           0.03182461 = score(doc=5701,freq=2.0), product of:
             0.12333567 = queryWeight, product of:
               2.0501404 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.018031579 = queryNorm
             0.2580325 = fieldWeight in 5701, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
         0.042544134 = weight(abstract_txt:systems in 5701) [ClassicSimilarity], result of:
           0.042544134 = score(doc=5701,freq=2.0), product of:
             0.16122897 = queryWeight, product of:
               2.6206908 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.018031579 = queryNorm
             0.263874 = fieldWeight in 5701, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
         0.054705605 = weight(abstract_txt:evaluation in 5701) [ClassicSimilarity], result of:
           0.054705605 = score(doc=5701,freq=1.0), product of:
             0.22298612 = queryWeight, product of:
               2.756628 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.018031579 = queryNorm
             0.24533188 = fieldWeight in 5701, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
         0.29029933 = weight(abstract_txt:interactive in 5701) [ClassicSimilarity], result of:
           0.29029933 = score(doc=5701,freq=5.0), product of:
             0.42736664 = queryWeight, product of:
               4.266724 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.018031579 = queryNorm
             0.6792746 = fieldWeight in 5701, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.0546875 = fieldNorm(doc=5701)
       0.32 = coord(8/25)
   

5. López-Ostenero, F.; Peinado, V.; Gonzalo, J.; Verdejo, F.: Interactive question answering : Is Cross-Language harder than monolingual searching? (2008) 0.18

   0.17769605 = sum of:
     0.17769605 = product of:
       0.7404002 = sum of:
         0.01659923 = weight(abstract_txt:system in 2023) [ClassicSimilarity], result of:
           0.01659923 = score(doc=2023,freq=1.0), product of:
             0.06300437 = queryWeight, product of:
               1.0361189 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.018031579 = queryNorm
             0.2634616 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.12967186 = weight(abstract_txt:question in 2023) [ClassicSimilarity], result of:
           0.12967186 = score(doc=2023,freq=2.0), product of:
             0.22537005 = queryWeight, product of:
               2.4000373 = boost
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.018031579 = queryNorm
             0.57537305 = fieldWeight in 2023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.207682 = idf(docFreq=657, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.060777336 = weight(abstract_txt:systems in 2023) [ClassicSimilarity], result of:
           0.060777336 = score(doc=2023,freq=2.0), product of:
             0.16122897 = queryWeight, product of:
               2.6206908 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.018031579 = queryNorm
             0.37696287 = fieldWeight in 2023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.07815086 = weight(abstract_txt:evaluation in 2023) [ClassicSimilarity], result of:
           0.07815086 = score(doc=2023,freq=1.0), product of:
             0.22298612 = queryWeight, product of:
               2.756628 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.018031579 = queryNorm
             0.35047412 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.1929132 = weight(abstract_txt:answering in 2023) [ClassicSimilarity], result of:
           0.1929132 = score(doc=2023,freq=1.0), product of:
             0.37004146 = queryWeight, product of:
               3.0753539 = boost
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.018031579 = queryNorm
             0.5213286 = fieldWeight in 2023, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6730065 = idf(docFreq=151, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
         0.2622877 = weight(abstract_txt:interactive in 2023) [ClassicSimilarity], result of:
           0.2622877 = score(doc=2023,freq=2.0), product of:
             0.42736664 = queryWeight, product of:
               4.266724 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.018031579 = queryNorm
             0.61372995 = fieldWeight in 2023, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.078125 = fieldNorm(doc=2023)
       0.24 = coord(6/25)