Document (#42099)

Author

Losada, D.E.
Parapar, J.
Barreiro, A.

Title

Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems

Source

Information processing and management. 53(2017) no.5, S.1005-1025

Year

2017

Abstract

Evaluating Information Retrieval systems is crucial to making progress in search technologies. Evaluation is often based on assembling reference collections consisting of documents, queries and relevance judgments done by humans. In large-scale environments, exhaustively judging relevance becomes infeasible. Instead, only a pool of documents is judged for relevance. By selectively choosing documents from the pool we can optimize the number of judgments required to identify a given number of relevant documents. We argue that this iterative selection process can be naturally modeled as a reinforcement learning problem and propose innovative and formal adjudication methods based on multi-armed bandits. Casting document judging as a multi-armed bandit problem is not only theoretically appealing, but also leads to highly effective adjudication methods. Under this bandit allocation framework, we consider stationary and non-stationary models and propose seven new document adjudication methods (five stationary methods and two non-stationary variants). Our paper also reports a series of experiments performed to thoroughly compare our new methods against current adjudication methods. This comparative study includes existing methods designed for pooling-based evaluation and existing methods designed for metasearch. Our experiments show that our theoretically grounded adjudication methods can substantially minimize the assessment effort.

Content

Vgl.: https://doi.org/10.1016/j.ipm.2017.04.005.

Theme

Retrievalstudien

Similar documents (author)

1. Losada, D.E.; Barreiro, A.: Emebedding term similarity and inverse document frequency into a logical model of information retrieval (2003) 4.88

   4.8754888 = sum of:
     4.8754888 = weight(author_txt:losada in 1422) [ClassicSimilarity], result of:
       4.8754888 = fieldWeight in 1422, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.5 = fieldNorm(doc=1422)
   

2. Fernández, R.T.; Losada, D.E.: Effective sentence retrieval based on query-independent evidence (2012) 4.88

   4.8754888 = sum of:
     4.8754888 = weight(author_txt:losada in 2728) [ClassicSimilarity], result of:
       4.8754888 = fieldWeight in 2728, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.5 = fieldNorm(doc=2728)
   

3. Sweeney, S.; Crestani, F.; Losada, D.E.: 'Show me more' : incremental length summarisation using novelty detection (2008) 3.66

   3.6566167 = sum of:
     3.6566167 = weight(author_txt:losada in 2054) [ClassicSimilarity], result of:
       3.6566167 = fieldWeight in 2054, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.375 = fieldNorm(doc=2054)
   

4. Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019) 3.66

   3.6566167 = sum of:
     3.6566167 = weight(author_txt:losada in 4674) [ClassicSimilarity], result of:
       3.6566167 = fieldWeight in 4674, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.375 = fieldNorm(doc=4674)
   

5. Parapar, J.; Losada, D.E.; Presedo-Quindimil, M.A.; Barreiro, A.: Using score distributions to compare statistical significance tests for information retrieval evaluation (2020) 3.05

   3.0471804 = sum of:
     3.0471804 = weight(author_txt:losada in 5506) [ClassicSimilarity], result of:
       3.0471804 = fieldWeight in 5506, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.7509775 = idf(docFreq=6, maxDocs=44218)
         0.3125 = fieldNorm(doc=5506)
   

Similar documents (content)

1. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.31

   0.31227094 = sum of:
     0.31227094 = product of:
       0.7806773 = sum of:
         0.07088871 = weight(abstract_txt:infeasible in 5092) [ClassicSimilarity], result of:
           0.07088871 = score(doc=5092,freq=1.0), product of:
             0.15724471 = queryWeight, product of:
               1.1115396 = boost
               9.617446 = idf(docFreq=7, maxDocs=44218)
               0.014709278 = queryNorm
             0.45081776 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.617446 = idf(docFreq=7, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.02175267 = weight(abstract_txt:propose in 5092) [ClassicSimilarity], result of:
           0.02175267 = score(doc=5092,freq=1.0), product of:
             0.090131454 = queryWeight, product of:
               1.1901182 = boost
               5.1486683 = idf(docFreq=697, maxDocs=44218)
               0.014709278 = queryNorm
             0.24134383 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1486683 = idf(docFreq=697, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.024138557 = weight(abstract_txt:experiments in 5092) [ClassicSimilarity], result of:
           0.024138557 = score(doc=5092,freq=1.0), product of:
             0.09660706 = queryWeight, product of:
               1.2321295 = boost
               5.3304167 = idf(docFreq=581, maxDocs=44218)
               0.014709278 = queryNorm
             0.24986328 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3304167 = idf(docFreq=581, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.010327214 = weight(abstract_txt:based in 5092) [ClassicSimilarity], result of:
           0.010327214 = score(doc=5092,freq=1.0), product of:
             0.069108814 = queryWeight, product of:
               1.4737834 = boost
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.014709278 = queryNorm
             0.14943412 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.048261613 = weight(abstract_txt:evaluation in 5092) [ClassicSimilarity], result of:
           0.048261613 = score(doc=5092,freq=5.0), product of:
             0.10263834 = queryWeight, product of:
               1.5554367 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.014709278 = queryNorm
             0.47021037 = fieldWeight in 5092, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.10438963 = weight(abstract_txt:judgments in 5092) [ClassicSimilarity], result of:
           0.10438963 = score(doc=5092,freq=4.0), product of:
             0.1615423 = queryWeight, product of:
               1.5932907 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014709278 = queryNorm
             0.6462062 = fieldWeight in 5092, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.040556747 = weight(abstract_txt:relevance in 5092) [ClassicSimilarity], result of:
           0.040556747 = score(doc=5092,freq=2.0), product of:
             0.12405011 = queryWeight, product of:
               1.71 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014709278 = queryNorm
             0.32693842 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.2810668 = weight(abstract_txt:judging in 5092) [ClassicSimilarity], result of:
           0.2810668 = score(doc=5092,freq=10.0), product of:
             0.23036034 = queryWeight, product of:
               1.9026363 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.014709278 = queryNorm
             1.220118 = fieldWeight in 5092, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.12815496 = weight(abstract_txt:pooling in 5092) [ClassicSimilarity], result of:
           0.12815496 = score(doc=5092,freq=1.0), product of:
             0.29400742 = queryWeight, product of:
               2.1494684 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.014709278 = queryNorm
             0.43589026 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.051140442 = weight(abstract_txt:methods in 5092) [ClassicSimilarity], result of:
           0.051140442 = score(doc=5092,freq=1.0), product of:
             0.26309654 = queryWeight, product of:
               4.31336 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.014709278 = queryNorm
             0.194379 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
       0.4 = coord(10/25)
   

2. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.30

   0.29565528 = sum of:
     0.29565528 = product of:
       0.8212646 = sum of:
         0.09451828 = weight(abstract_txt:infeasible in 2591) [ClassicSimilarity], result of:
           0.09451828 = score(doc=2591,freq=1.0), product of:
             0.15724471 = queryWeight, product of:
               1.1115396 = boost
               9.617446 = idf(docFreq=7, maxDocs=44218)
               0.014709278 = queryNorm
             0.6010904 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.617446 = idf(docFreq=7, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.013769619 = weight(abstract_txt:based in 2591) [ClassicSimilarity], result of:
           0.013769619 = score(doc=2591,freq=1.0), product of:
             0.069108814 = queryWeight, product of:
               1.4737834 = boost
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.014709278 = queryNorm
             0.19924548 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.028777666 = weight(abstract_txt:evaluation in 2591) [ClassicSimilarity], result of:
           0.028777666 = score(doc=2591,freq=1.0), product of:
             0.10263834 = queryWeight, product of:
               1.5554367 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.014709278 = queryNorm
             0.2803793 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.15561488 = weight(abstract_txt:judgments in 2591) [ClassicSimilarity], result of:
           0.15561488 = score(doc=2591,freq=5.0), product of:
             0.1615423 = queryWeight, product of:
               1.5932907 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014709278 = queryNorm
             0.9633073 = fieldWeight in 2591, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.1011663 = weight(abstract_txt:relevance in 2591) [ClassicSimilarity], result of:
           0.1011663 = score(doc=2591,freq=7.0), product of:
             0.12405011 = queryWeight, product of:
               1.71 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014709278 = queryNorm
             0.8155277 = fieldWeight in 2591, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.108122565 = weight(abstract_txt:pool in 2591) [ClassicSimilarity], result of:
           0.108122565 = score(doc=2591,freq=1.0), product of:
             0.21669708 = queryWeight, product of:
               1.8453487 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.014709278 = queryNorm
             0.4989572 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.11850818 = weight(abstract_txt:judging in 2591) [ClassicSimilarity], result of:
           0.11850818 = score(doc=2591,freq=1.0), product of:
             0.23036034 = queryWeight, product of:
               1.9026363 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.014709278 = queryNorm
             0.514447 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.06441258 = weight(abstract_txt:documents in 2591) [ClassicSimilarity], result of:
           0.06441258 = score(doc=2591,freq=3.0), product of:
             0.14437619 = queryWeight, product of:
               2.381605 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.014709278 = queryNorm
             0.44614407 = fieldWeight in 2591, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.1363745 = weight(abstract_txt:methods in 2591) [ClassicSimilarity], result of:
           0.1363745 = score(doc=2591,freq=4.0), product of:
             0.26309654 = queryWeight, product of:
               4.31336 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.014709278 = queryNorm
             0.518344 = fieldWeight in 2591, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
       0.36 = coord(9/25)
   

3. Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019) 0.26

   0.25592548 = sum of:
     0.25592548 = product of:
       0.79976714 = sum of:
         0.02900356 = weight(abstract_txt:propose in 4674) [ClassicSimilarity], result of:
           0.02900356 = score(doc=4674,freq=1.0), product of:
             0.090131454 = queryWeight, product of:
               1.1901182 = boost
               5.1486683 = idf(docFreq=697, maxDocs=44218)
               0.014709278 = queryNorm
             0.32179177 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1486683 = idf(docFreq=697, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.028777666 = weight(abstract_txt:evaluation in 4674) [ClassicSimilarity], result of:
           0.028777666 = score(doc=4674,freq=1.0), product of:
             0.10263834 = queryWeight, product of:
               1.5554367 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.014709278 = queryNorm
             0.2803793 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.12053876 = weight(abstract_txt:judgments in 4674) [ClassicSimilarity], result of:
           0.12053876 = score(doc=4674,freq=3.0), product of:
             0.1615423 = queryWeight, product of:
               1.5932907 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014709278 = queryNorm
             0.74617463 = fieldWeight in 4674, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.0662289 = weight(abstract_txt:relevance in 4674) [ClassicSimilarity], result of:
           0.0662289 = score(doc=4674,freq=3.0), product of:
             0.12405011 = queryWeight, product of:
               1.71 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014709278 = queryNorm
             0.5338882 = fieldWeight in 4674, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.1529084 = weight(abstract_txt:pool in 4674) [ClassicSimilarity], result of:
           0.1529084 = score(doc=4674,freq=2.0), product of:
             0.21669708 = queryWeight, product of:
               1.8453487 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.014709278 = queryNorm
             0.70563203 = fieldWeight in 4674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.1708733 = weight(abstract_txt:pooling in 4674) [ClassicSimilarity], result of:
           0.1708733 = score(doc=4674,freq=1.0), product of:
             0.29400742 = queryWeight, product of:
               2.1494684 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.014709278 = queryNorm
             0.581187 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.06441258 = weight(abstract_txt:documents in 4674) [ClassicSimilarity], result of:
           0.06441258 = score(doc=4674,freq=3.0), product of:
             0.14437619 = queryWeight, product of:
               2.381605 = boost
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.014709278 = queryNorm
             0.44614407 = fieldWeight in 4674, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.1213026 = idf(docFreq=1949, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.16702399 = weight(abstract_txt:methods in 4674) [ClassicSimilarity], result of:
           0.16702399 = score(doc=4674,freq=6.0), product of:
             0.26309654 = queryWeight, product of:
               4.31336 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.014709278 = queryNorm
             0.6348392 = fieldWeight in 4674, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
       0.32 = coord(8/25)
   

4. Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011) 0.22

   0.22047383 = sum of:
     0.22047383 = product of:
       0.7874065 = sum of:
         0.017039033 = weight(abstract_txt:based in 4197) [ClassicSimilarity], result of:
           0.017039033 = score(doc=4197,freq=2.0), product of:
             0.069108814 = queryWeight, product of:
               1.4737834 = boost
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.014709278 = queryNorm
             0.24655369 = fieldWeight in 4197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.025180457 = weight(abstract_txt:evaluation in 4197) [ClassicSimilarity], result of:
           0.025180457 = score(doc=4197,freq=1.0), product of:
             0.10263834 = queryWeight, product of:
               1.5554367 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.014709278 = queryNorm
             0.24533188 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.06089395 = weight(abstract_txt:judgments in 4197) [ClassicSimilarity], result of:
           0.06089395 = score(doc=4197,freq=1.0), product of:
             0.1615423 = queryWeight, product of:
               1.5932907 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014709278 = queryNorm
             0.3769536 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.13379484 = weight(abstract_txt:pool in 4197) [ClassicSimilarity], result of:
           0.13379484 = score(doc=4197,freq=2.0), product of:
             0.21669708 = queryWeight, product of:
               1.8453487 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.014709278 = queryNorm
             0.617428 = fieldWeight in 4197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.23186831 = weight(abstract_txt:judging in 4197) [ClassicSimilarity], result of:
           0.23186831 = score(doc=4197,freq=5.0), product of:
             0.23036034 = queryWeight, product of:
               1.9026363 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.014709278 = queryNorm
             1.0065461 = fieldWeight in 4197, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.2589661 = weight(abstract_txt:pooling in 4197) [ClassicSimilarity], result of:
           0.2589661 = score(doc=4197,freq=3.0), product of:
             0.29400742 = queryWeight, product of:
               2.1494684 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.014709278 = queryNorm
             0.8808148 = fieldWeight in 4197, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
         0.059663847 = weight(abstract_txt:methods in 4197) [ClassicSimilarity], result of:
           0.059663847 = score(doc=4197,freq=1.0), product of:
             0.26309654 = queryWeight, product of:
               4.31336 = boost
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.014709278 = queryNorm
             0.2267755 = fieldWeight in 4197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.146752 = idf(docFreq=1900, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4197)
       0.28 = coord(7/25)
   

5. Baillie, M.; Azzopardi, L.; Ruthven, I.: Evaluating epistemic uncertainty under incomplete assessments (2008) 0.12

   0.12435301 = sum of:
     0.12435301 = product of:
       0.44411787 = sum of:
         0.026726775 = weight(abstract_txt:existing in 2065) [ClassicSimilarity], result of:
           0.026726775 = score(doc=2065,freq=1.0), product of:
             0.07355291 = queryWeight, product of:
               1.0751073 = boost
               4.6511106 = idf(docFreq=1147, maxDocs=44218)
               0.014709278 = queryNorm
             0.36336803 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6511106 = idf(docFreq=1147, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.03625445 = weight(abstract_txt:propose in 2065) [ClassicSimilarity], result of:
           0.03625445 = score(doc=2065,freq=1.0), product of:
             0.090131454 = queryWeight, product of:
               1.1901182 = boost
               5.1486683 = idf(docFreq=697, maxDocs=44218)
               0.014709278 = queryNorm
             0.4022397 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1486683 = idf(docFreq=697, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.04023093 = weight(abstract_txt:experiments in 2065) [ClassicSimilarity], result of:
           0.04023093 = score(doc=2065,freq=1.0), product of:
             0.09660706 = queryWeight, product of:
               1.2321295 = boost
               5.3304167 = idf(docFreq=581, maxDocs=44218)
               0.014709278 = queryNorm
             0.41643882 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3304167 = idf(docFreq=581, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.017212024 = weight(abstract_txt:based in 2065) [ClassicSimilarity], result of:
           0.017212024 = score(doc=2065,freq=1.0), product of:
             0.069108814 = queryWeight, product of:
               1.4737834 = boost
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.014709278 = queryNorm
             0.24905685 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1879277 = idf(docFreq=4958, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.062305473 = weight(abstract_txt:evaluation in 2065) [ClassicSimilarity], result of:
           0.062305473 = score(doc=2065,freq=3.0), product of:
             0.10263834 = queryWeight, product of:
               1.5554367 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.014709278 = queryNorm
             0.607039 = fieldWeight in 2065, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.04779659 = weight(abstract_txt:relevance in 2065) [ClassicSimilarity], result of:
           0.04779659 = score(doc=2065,freq=1.0), product of:
             0.12405011 = queryWeight, product of:
               1.71 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014709278 = queryNorm
             0.38530064 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
         0.21359162 = weight(abstract_txt:pooling in 2065) [ClassicSimilarity], result of:
           0.21359162 = score(doc=2065,freq=1.0), product of:
             0.29400742 = queryWeight, product of:
               2.1494684 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.014709278 = queryNorm
             0.72648376 = fieldWeight in 2065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.078125 = fieldNorm(doc=2065)
       0.28 = coord(7/25)