Document (#43647)

Author

Purpura, A.
Silvello, G.
Susto, G.A.

Title

Learning to rank from relevance judgments distributions

Source

Journal of the Association for Information Science and Technology. 73(2022) no.9, S.1236-1252

Year

2022

Abstract

LEarning TO Rank (LETOR) algorithms are usually trained on annotated corpora where a single relevance label is assigned to each available document-topic pair. Within the Cranfield framework, relevance labels result from merging either multiple expertly curated or crowdsourced human assessments. In this paper, we explore how to train LETOR models with relevance judgments distributions (either real or synthetically generated) assigned to document-topic pairs instead of single-valued relevance labels. We propose five new probabilistic loss functions to deal with the higher expressive power provided by relevance judgments distributions and show how they can be applied both to neural and gradient boosting machine (GBM) architectures. Moreover, we show how training a LETOR model on a sampled version of the relevance judgments from certain probability distributions can improve its performance when relying either on traditional or probabilistic loss functions. Finally, we validate our hypothesis on real-world crowdsourced relevance judgments distributions. Overall, we observe that relying on relevance judgments distributions to train different LETOR models can boost their performance and even outperform strong baselines such as LambdaMART on several test collections.

Content

https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24629. Vgl.: https://doi.org/10.1002/asi.24629.

Theme

Retrievalalgorithmen

Similar documents (author)

1. Silvello, G.: Learning to cite framework : how to automatically construct citations for hierarchical data (2017) 6.01

   6.010904 = sum of:
     6.010904 = weight(author_txt:silvello in 3647) [ClassicSimilarity], result of:
       6.010904 = fieldWeight in 3647, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.625 = fieldNorm(doc=3647)
   

2. Silvello, G.: Theory and practice of data citation (2018) 6.01

   6.010904 = sum of:
     6.010904 = weight(author_txt:silvello in 4006) [ClassicSimilarity], result of:
       6.010904 = fieldWeight in 4006, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.625 = fieldNorm(doc=4006)
   

3. Ferro, N.; Silvello, G.: NESTOR: a formal model for digital archives (2013) 4.81

   4.808723 = sum of:
     4.808723 = weight(author_txt:silvello in 2707) [ClassicSimilarity], result of:
       4.808723 = fieldWeight in 2707, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.5 = fieldNorm(doc=2707)
   

4. Ferro, N.; Silvello, G.: Toward an anatomy of IR system component performances (2018) 4.81

   4.808723 = sum of:
     4.808723 = weight(author_txt:silvello in 4035) [ClassicSimilarity], result of:
       4.808723 = fieldWeight in 4035, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.5 = fieldNorm(doc=4035)
   

5. Ferro, N.; Silvello, G.; Keskustalo, H.; Pirkola, A.; Järvelin, K.: ¬The twist measure for IR evaluation : taking user's effort into account (2016) 3.01

   3.005452 = sum of:
     3.005452 = weight(author_txt:silvello in 2771) [ClassicSimilarity], result of:
       3.005452 = fieldWeight in 2771, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.617446 = idf(docFreq=7, maxDocs=44218)
         0.3125 = fieldNorm(doc=2771)
   

Similar documents (content)

1. Bodoff, D.; Wu, B.; Wong, K.Y.M.: Relevance data for language models using maximum likelihood (2003) 0.25

   0.25476918 = sum of:
     0.25476918 = product of:
       1.0615382 = sum of:
         0.034758024 = weight(abstract_txt:document in 1822) [ClassicSimilarity], result of:
           0.034758024 = score(doc=1822,freq=2.0), product of:
             0.06107276 = queryWeight, product of:
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.014227425 = queryNorm
             0.5691248 = fieldWeight in 1822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.030848745 = weight(abstract_txt:performance in 1822) [ClassicSimilarity], result of:
           0.030848745 = score(doc=1822,freq=1.0), product of:
             0.07106338 = queryWeight, product of:
               1.0786963 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.014227425 = queryNorm
             0.43410188 = fieldWeight in 1822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.031072216 = weight(abstract_txt:models in 1822) [ClassicSimilarity], result of:
           0.031072216 = score(doc=1822,freq=1.0), product of:
             0.071406156 = queryWeight, product of:
               1.0812947 = boost
               4.6415744 = idf(docFreq=1158, maxDocs=44218)
               0.014227425 = queryNorm
             0.43514758 = fieldWeight in 1822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6415744 = idf(docFreq=1158, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.29591945 = weight(abstract_txt:distributions in 1822) [ClassicSimilarity], result of:
           0.29591945 = score(doc=1822,freq=1.0), product of:
             0.46271145 = queryWeight, product of:
               4.767514 = boost
               6.82169 = idf(docFreq=130, maxDocs=44218)
               0.014227425 = queryNorm
             0.63953346 = fieldWeight in 1822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.82169 = idf(docFreq=130, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.43172988 = weight(abstract_txt:judgments in 1822) [ClassicSimilarity], result of:
           0.43172988 = score(doc=1822,freq=2.0), product of:
             0.47241753 = queryWeight, product of:
               4.8172574 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014227425 = queryNorm
             0.91387355 = fieldWeight in 1822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.23720993 = weight(abstract_txt:relevance in 1822) [ClassicSimilarity], result of:
           0.23720993 = score(doc=1822,freq=2.0), product of:
             0.36277464 = queryWeight, product of:
               5.1701236 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014227425 = queryNorm
             0.65387684 = fieldWeight in 1822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
       0.24 = coord(6/25)
   

2. Spina, D.; Trippas, J.R.; Cavedon, L.; Sanderson, M.: Extracting audio summaries to support effective spoken document search (2017) 0.20

   0.19567847 = sum of:
     0.19567847 = product of:
       0.815327 = sum of:
         0.02896502 = weight(abstract_txt:document in 3788) [ClassicSimilarity], result of:
           0.02896502 = score(doc=3788,freq=2.0), product of:
             0.06107276 = queryWeight, product of:
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.014227425 = queryNorm
             0.4742707 = fieldWeight in 3788, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.078125 = fieldNorm(doc=3788)
         0.022146374 = weight(abstract_txt:show in 3788) [ClassicSimilarity], result of:
           0.022146374 = score(doc=3788,freq=1.0), product of:
             0.06433936 = queryWeight, product of:
               1.0263952 = boost
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.014227425 = queryNorm
             0.3442119 = fieldWeight in 3788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.078125 = fieldNorm(doc=3788)
         0.09015845 = weight(abstract_txt:curated in 3788) [ClassicSimilarity], result of:
           0.09015845 = score(doc=3788,freq=1.0), product of:
             0.1301975 = queryWeight, product of:
               1.0324346 = boost
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.014227425 = queryNorm
             0.69247454 = fieldWeight in 3788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.863674 = idf(docFreq=16, maxDocs=44218)
               0.078125 = fieldNorm(doc=3788)
         0.22198293 = weight(abstract_txt:crowdsourced in 3788) [ClassicSimilarity], result of:
           0.22198293 = score(doc=3788,freq=1.0), product of:
             0.2991034 = queryWeight, product of:
               2.213028 = boost
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.014227425 = queryNorm
             0.74216115 = fieldWeight in 3788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.078125 = fieldNorm(doc=3788)
         0.2543993 = weight(abstract_txt:judgments in 3788) [ClassicSimilarity], result of:
           0.2543993 = score(doc=3788,freq=1.0), product of:
             0.47241753 = queryWeight, product of:
               4.8172574 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014227425 = queryNorm
             0.5385052 = fieldWeight in 3788, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.078125 = fieldNorm(doc=3788)
         0.19767494 = weight(abstract_txt:relevance in 3788) [ClassicSimilarity], result of:
           0.19767494 = score(doc=3788,freq=2.0), product of:
             0.36277464 = queryWeight, product of:
               5.1701236 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014227425 = queryNorm
             0.5448974 = fieldWeight in 3788, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=3788)
       0.24 = coord(6/25)
   

3. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.17

   0.16788553 = sum of:
     0.16788553 = product of:
       0.8394276 = sum of:
         0.03277018 = weight(abstract_txt:document in 2591) [ClassicSimilarity], result of:
           0.03277018 = score(doc=2591,freq=4.0), product of:
             0.06107276 = queryWeight, product of:
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.014227425 = queryNorm
             0.53657603 = fieldWeight in 2591, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.017717099 = weight(abstract_txt:show in 2591) [ClassicSimilarity], result of:
           0.017717099 = score(doc=2591,freq=1.0), product of:
             0.06433936 = queryWeight, product of:
               1.0263952 = boost
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.014227425 = queryNorm
             0.27536952 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.03800422 = weight(abstract_txt:topic in 2591) [ClassicSimilarity], result of:
           0.03800422 = score(doc=2591,freq=2.0), product of:
             0.08493621 = queryWeight, product of:
               1.1792955 = boost
               5.062254 = idf(docFreq=760, maxDocs=44218)
               0.014227425 = queryNorm
             0.44744426 = fieldWeight in 2591, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.062254 = idf(docFreq=760, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.45508328 = weight(abstract_txt:judgments in 2591) [ClassicSimilarity], result of:
           0.45508328 = score(doc=2591,freq=5.0), product of:
             0.47241753 = queryWeight, product of:
               4.8172574 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014227425 = queryNorm
             0.9633073 = fieldWeight in 2591, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.29585275 = weight(abstract_txt:relevance in 2591) [ClassicSimilarity], result of:
           0.29585275 = score(doc=2591,freq=7.0), product of:
             0.36277464 = queryWeight, product of:
               5.1701236 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014227425 = queryNorm
             0.8155277 = fieldWeight in 2591, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
       0.2 = coord(5/25)
   

4. Keikha, M.; Crestani, F.; Carman, M.J.: Employing document dependency in blog search (2012) 0.17

   0.16584231 = sum of:
     0.16584231 = product of:
       0.592294 = sum of:
         0.01638509 = weight(abstract_txt:document in 4987) [ClassicSimilarity], result of:
           0.01638509 = score(doc=4987,freq=1.0), product of:
             0.06107276 = queryWeight, product of:
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.014227425 = queryNorm
             0.26828802 = fieldWeight in 4987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=4987)
         0.017717099 = weight(abstract_txt:show in 4987) [ClassicSimilarity], result of:
           0.017717099 = score(doc=4987,freq=1.0), product of:
             0.06433936 = queryWeight, product of:
               1.0263952 = boost
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.014227425 = queryNorm
             0.27536952 = fieldWeight in 4987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.0625 = fieldNorm(doc=4987)
         0.029084476 = weight(abstract_txt:performance in 4987) [ClassicSimilarity], result of:
           0.029084476 = score(doc=4987,freq=2.0), product of:
             0.07106338 = queryWeight, product of:
               1.0786963 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.014227425 = queryNorm
             0.40927517 = fieldWeight in 4987, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=4987)
         0.026873041 = weight(abstract_txt:topic in 4987) [ClassicSimilarity], result of:
           0.026873041 = score(doc=4987,freq=1.0), product of:
             0.08493621 = queryWeight, product of:
               1.1792955 = boost
               5.062254 = idf(docFreq=760, maxDocs=44218)
               0.014227425 = queryNorm
             0.31639087 = fieldWeight in 4987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.062254 = idf(docFreq=760, maxDocs=44218)
               0.0625 = fieldNorm(doc=4987)
         0.05491347 = weight(abstract_txt:rank in 4987) [ClassicSimilarity], result of:
           0.05491347 = score(doc=4987,freq=1.0), product of:
             0.13677317 = queryWeight, product of:
               1.4964999 = boost
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.014227425 = queryNorm
             0.40149298 = fieldWeight in 4987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.0625 = fieldNorm(doc=4987)
         0.19727963 = weight(abstract_txt:distributions in 4987) [ClassicSimilarity], result of:
           0.19727963 = score(doc=4987,freq=1.0), product of:
             0.46271145 = queryWeight, product of:
               4.767514 = boost
               6.82169 = idf(docFreq=130, maxDocs=44218)
               0.014227425 = queryNorm
             0.42635563 = fieldWeight in 4987, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.82169 = idf(docFreq=130, maxDocs=44218)
               0.0625 = fieldNorm(doc=4987)
         0.25004122 = weight(abstract_txt:relevance in 4987) [ClassicSimilarity], result of:
           0.25004122 = score(doc=4987,freq=5.0), product of:
             0.36277464 = queryWeight, product of:
               5.1701236 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014227425 = queryNorm
             0.6892467 = fieldWeight in 4987, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=4987)
       0.28 = coord(7/25)
   

5. Efron, M.; Winget, M.: Query polyrepresentation for ranking retrieval systems without relevance judgments (2010) 0.15

   0.146973 = sum of:
     0.146973 = product of:
       0.734865 = sum of:
         0.0335913 = weight(abstract_txt:topic in 3469) [ClassicSimilarity], result of:
           0.0335913 = score(doc=3469,freq=1.0), product of:
             0.08493621 = queryWeight, product of:
               1.1792955 = boost
               5.062254 = idf(docFreq=760, maxDocs=44218)
               0.014227425 = queryNorm
             0.3954886 = fieldWeight in 3469, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.062254 = idf(docFreq=760, maxDocs=44218)
               0.078125 = fieldNorm(doc=3469)
         0.052222114 = weight(abstract_txt:single in 3469) [ClassicSimilarity], result of:
           0.052222114 = score(doc=3469,freq=2.0), product of:
             0.090469316 = queryWeight, product of:
               1.2171017 = boost
               5.2245407 = idf(docFreq=646, maxDocs=44218)
               0.014227425 = queryNorm
             0.57723564 = fieldWeight in 3469, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.2245407 = idf(docFreq=646, maxDocs=44218)
               0.078125 = fieldNorm(doc=3469)
         0.068641834 = weight(abstract_txt:rank in 3469) [ClassicSimilarity], result of:
           0.068641834 = score(doc=3469,freq=1.0), product of:
             0.13677317 = queryWeight, product of:
               1.4964999 = boost
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.014227425 = queryNorm
             0.5018662 = fieldWeight in 3469, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.4238877 = idf(docFreq=194, maxDocs=44218)
               0.078125 = fieldNorm(doc=3469)
         0.44063246 = weight(abstract_txt:judgments in 3469) [ClassicSimilarity], result of:
           0.44063246 = score(doc=3469,freq=3.0), product of:
             0.47241753 = queryWeight, product of:
               4.8172574 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.014227425 = queryNorm
             0.9327183 = fieldWeight in 3469, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.078125 = fieldNorm(doc=3469)
         0.1397773 = weight(abstract_txt:relevance in 3469) [ClassicSimilarity], result of:
           0.1397773 = score(doc=3469,freq=1.0), product of:
             0.36277464 = queryWeight, product of:
               5.1701236 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.014227425 = queryNorm
             0.38530064 = fieldWeight in 3469, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=3469)
       0.2 = coord(5/25)