Document (#43564)

Author

Sun, M.
Danfa, J.B.
Teplitskiy, M.

Title

Does double-blind peer review reduce bias? : evidence from a top computer science conference

Source

Journal of the Association for Information Science and Technology. 73(2022) no.6, S.811-819

Year

2022

Abstract

Peer review is essential for advancing scientific research, but there are long-standing concerns that authors' prestige or other characteristics can bias reviewers. Double-blind peer review has been proposed as a way to reduce reviewer bias, but the evidence for its effectiveness is limited and mixed. Here, we examine the effects of double-blind peer review by analyzing the review files of 5,027 papers submitted to a top computer science conference that changed its reviewing format from single- to double-blind in 2018. First, we find that the scores given to the most prestigious authors significantly decreased after switching to double-blind review. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance significantly. Second, the inter-reviewer disagreement increased significantly in the double-blind format. Third, papers rejected in the single-blind format are cited more than those rejected under double-blind, suggesting that double-blind review better excludes poorer quality papers. Lastly, an apparently unrelated change in the rating scale from 10 to 4 points likely reduced prestige bias significantly such that papers' acceptance was affected. These results support the effectiveness of double-blind review in reducing biases, while opening new research directions on the impact of peer-review formats.

Content

Vgl.: https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24582. https://doi.org/10.1002/asi.24582.

Similar documents (content)

1. Mulligan, A.; Hall, L.; Raphael, E.: Peer review in a changing world : an international study measuring the attitudes of researchers (2013) 0.32

   0.32493588 = sum of:
     0.32493588 = product of:
       1.1604853 = sum of:
         0.017293312 = weight(abstract_txt:authors in 528) [ClassicSimilarity], result of:
           0.017293312 = score(doc=528,freq=2.0), product of:
             0.04208916 = queryWeight, product of:
               1.0765704 = boost
               4.648501 = idf(docFreq=1150, maxDocs=44218)
               0.008410366 = queryNorm
             0.4108733 = fieldWeight in 528, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.648501 = idf(docFreq=1150, maxDocs=44218)
               0.0625 = fieldNorm(doc=528)
         0.00905328 = weight(abstract_txt:that in 528) [ClassicSimilarity], result of:
           0.00905328 = score(doc=528,freq=5.0), product of:
             0.027339386 = queryWeight, product of:
               1.3718976 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008410366 = queryNorm
             0.3311442 = fieldWeight in 528, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=528)
         0.044678356 = weight(abstract_txt:papers in 528) [ClassicSimilarity], result of:
           0.044678356 = score(doc=528,freq=1.0), product of:
             0.13551049 = queryWeight, product of:
               3.0543146 = boost
               5.2752647 = idf(docFreq=614, maxDocs=44218)
               0.008410366 = queryNorm
             0.32970405 = fieldWeight in 528, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2752647 = idf(docFreq=614, maxDocs=44218)
               0.0625 = fieldNorm(doc=528)
         0.28931597 = weight(abstract_txt:peer in 528) [ClassicSimilarity], result of:
           0.28931597 = score(doc=528,freq=12.0), product of:
             0.2056345 = queryWeight, product of:
               3.7624896 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.008410366 = queryNorm
             1.4069427 = fieldWeight in 528, product of:
               3.4641016 = tf(freq=12.0), with freq of:
                 12.0 = termFreq=12.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.0625 = fieldNorm(doc=528)
         0.20836996 = weight(abstract_txt:review in 528) [ClassicSimilarity], result of:
           0.20836996 = score(doc=528,freq=11.0), product of:
             0.20689888 = queryWeight, product of:
               5.0634046 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.008410366 = queryNorm
             1.0071101 = fieldWeight in 528, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.0625 = fieldNorm(doc=528)
         0.27621552 = weight(abstract_txt:double in 528) [ClassicSimilarity], result of:
           0.27621552 = score(doc=528,freq=1.0), product of:
             0.5552617 = queryWeight, product of:
               8.294924 = boost
               7.9592175 = idf(docFreq=41, maxDocs=44218)
               0.008410366 = queryNorm
             0.4974511 = fieldWeight in 528, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.9592175 = idf(docFreq=41, maxDocs=44218)
               0.0625 = fieldNorm(doc=528)
         0.31555894 = weight(abstract_txt:blind in 528) [ClassicSimilarity], result of:
           0.31555894 = score(doc=528,freq=1.0), product of:
             0.62849975 = queryWeight, product of:
               9.302397 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.008410366 = queryNorm
             0.5020828 = fieldWeight in 528, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.0625 = fieldNorm(doc=528)
       0.28 = coord(7/25)
   

2. Zhao, Y.W.; Chi. C.-H.; Heuvel, W.J. van den: Imperfect referees : reducing the impact of multiple biases in peer review (2015) 0.20

   0.19511338 = sum of:
     0.19511338 = product of:
       0.6097293 = sum of:
         0.009800222 = weight(abstract_txt:computer in 2271) [ClassicSimilarity], result of:
           0.009800222 = score(doc=2271,freq=1.0), product of:
             0.036314946 = queryWeight, product of:
               4.317879 = idf(docFreq=1601, maxDocs=44218)
               0.008410366 = queryNorm
             0.26986745 = fieldWeight in 2271, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.317879 = idf(docFreq=1601, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
         0.017360764 = weight(abstract_txt:single in 2271) [ClassicSimilarity], result of:
           0.017360764 = score(doc=2271,freq=1.0), product of:
             0.053166818 = queryWeight, product of:
               1.2099785 = boost
               5.2245407 = idf(docFreq=646, maxDocs=44218)
               0.008410366 = queryNorm
             0.3265338 = fieldWeight in 2271, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2245407 = idf(docFreq=646, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
         0.029647022 = weight(abstract_txt:conference in 2271) [ClassicSimilarity], result of:
           0.029647022 = score(doc=2271,freq=2.0), product of:
             0.060289107 = queryWeight, product of:
               1.2884772 = boost
               5.563489 = idf(docFreq=460, maxDocs=44218)
               0.008410366 = queryNorm
             0.4917476 = fieldWeight in 2271, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.563489 = idf(docFreq=460, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
         0.0070126397 = weight(abstract_txt:that in 2271) [ClassicSimilarity], result of:
           0.0070126397 = score(doc=2271,freq=3.0), product of:
             0.027339386 = queryWeight, product of:
               1.3718976 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008410366 = queryNorm
             0.2565032 = fieldWeight in 2271, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
         0.08841684 = weight(abstract_txt:reviewer in 2271) [ClassicSimilarity], result of:
           0.08841684 = score(doc=2271,freq=1.0), product of:
             0.15738069 = queryWeight, product of:
               2.0817714 = boost
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.008410366 = queryNorm
             0.5618023 = fieldWeight in 2271, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
         0.18718205 = weight(abstract_txt:bias in 2271) [ClassicSimilarity], result of:
           0.18718205 = score(doc=2271,freq=5.0), product of:
             0.19118667 = queryWeight, product of:
               3.2448983 = boost
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.008410366 = queryNorm
             0.9790539 = fieldWeight in 2271, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
         0.14465798 = weight(abstract_txt:peer in 2271) [ClassicSimilarity], result of:
           0.14465798 = score(doc=2271,freq=3.0), product of:
             0.2056345 = queryWeight, product of:
               3.7624896 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.008410366 = queryNorm
             0.70347136 = fieldWeight in 2271, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
         0.1256518 = weight(abstract_txt:review in 2271) [ClassicSimilarity], result of:
           0.1256518 = score(doc=2271,freq=4.0), product of:
             0.20689888 = queryWeight, product of:
               5.0634046 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.008410366 = queryNorm
             0.60731024 = fieldWeight in 2271, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.0625 = fieldNorm(doc=2271)
       0.32 = coord(8/25)
   

3. Bornmann, L.: Interrater reliability and convergent validity of F1000Prime peer review (2015) 0.13

   0.12721846 = sum of:
     0.12721846 = product of:
       0.63609225 = sum of:
         0.0050609373 = weight(abstract_txt:that in 2328) [ClassicSimilarity], result of:
           0.0050609373 = score(doc=2328,freq=1.0), product of:
             0.027339386 = queryWeight, product of:
               1.3718976 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008410366 = queryNorm
             0.18511525 = fieldWeight in 2328, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=2328)
         0.05076027 = weight(abstract_txt:significantly in 2328) [ClassicSimilarity], result of:
           0.05076027 = score(doc=2328,freq=1.0), product of:
             0.11803589 = queryWeight, product of:
               2.5496428 = boost
               5.5045247 = idf(docFreq=488, maxDocs=44218)
               0.008410366 = queryNorm
             0.430041 = fieldWeight in 2328, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5045247 = idf(docFreq=488, maxDocs=44218)
               0.078125 = fieldNorm(doc=2328)
         0.11169589 = weight(abstract_txt:papers in 2328) [ClassicSimilarity], result of:
           0.11169589 = score(doc=2328,freq=4.0), product of:
             0.13551049 = queryWeight, product of:
               3.0543146 = boost
               5.2752647 = idf(docFreq=614, maxDocs=44218)
               0.008410366 = queryNorm
             0.8242601 = fieldWeight in 2328, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.2752647 = idf(docFreq=614, maxDocs=44218)
               0.078125 = fieldNorm(doc=2328)
         0.2762109 = weight(abstract_txt:peer in 2328) [ClassicSimilarity], result of:
           0.2762109 = score(doc=2328,freq=7.0), product of:
             0.2056345 = queryWeight, product of:
               3.7624896 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.008410366 = queryNorm
             1.3432128 = fieldWeight in 2328, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.078125 = fieldNorm(doc=2328)
         0.19236426 = weight(abstract_txt:review in 2328) [ClassicSimilarity], result of:
           0.19236426 = score(doc=2328,freq=6.0), product of:
             0.20689888 = queryWeight, product of:
               5.0634046 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.008410366 = queryNorm
             0.92975014 = fieldWeight in 2328, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.078125 = fieldNorm(doc=2328)
       0.2 = coord(5/25)
   

4. García, J.A.; Rodriguez-Sánchez, R.; Fdez-Valdivia, J.: Bias and effort in peer review (2015) 0.12

   0.12315222 = sum of:
     0.12315222 = product of:
       0.6157611 = sum of:
         0.0071572457 = weight(abstract_txt:that in 2121) [ClassicSimilarity], result of:
           0.0071572457 = score(doc=2121,freq=2.0), product of:
             0.027339386 = queryWeight, product of:
               1.3718976 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008410366 = queryNorm
             0.26179248 = fieldWeight in 2121, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
         0.11052105 = weight(abstract_txt:reviewer in 2121) [ClassicSimilarity], result of:
           0.11052105 = score(doc=2121,freq=1.0), product of:
             0.15738069 = queryWeight, product of:
               2.0817714 = boost
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.008410366 = queryNorm
             0.7022529 = fieldWeight in 2121, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
         0.18123823 = weight(abstract_txt:bias in 2121) [ClassicSimilarity], result of:
           0.18123823 = score(doc=2121,freq=3.0), product of:
             0.19118667 = queryWeight, product of:
               3.2448983 = boost
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.008410366 = queryNorm
             0.94796485 = fieldWeight in 2121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
         0.18082249 = weight(abstract_txt:peer in 2121) [ClassicSimilarity], result of:
           0.18082249 = score(doc=2121,freq=3.0), product of:
             0.2056345 = queryWeight, product of:
               3.7624896 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.008410366 = queryNorm
             0.8793392 = fieldWeight in 2121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
         0.13602208 = weight(abstract_txt:review in 2121) [ClassicSimilarity], result of:
           0.13602208 = score(doc=2121,freq=3.0), product of:
             0.20689888 = queryWeight, product of:
               5.0634046 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.008410366 = queryNorm
             0.6574326 = fieldWeight in 2121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
       0.2 = coord(5/25)
   

5. Horstmann, M.; Lorenz, M.; Watkowski, A.; Ioannidis, G.; Herzog, O.; King, A.; Evans, D.G.; Hagen, C.; Schlieder, C.; Burn, A.-M.; King, N.; Petrie, H.; Dijkstra, S.; Crombie, D: Automated interpretation and accessible presentation of technical diagrams for blind people (2004) 0.11

   0.10871845 = sum of:
     0.10871845 = product of:
       0.6794903 = sum of:
         0.016133202 = weight(abstract_txt:effectiveness in 5925) [ClassicSimilarity], result of:
           0.016133202 = score(doc=5925,freq=1.0), product of:
             0.050630067 = queryWeight, product of:
               1.1807598 = boost
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.008410366 = queryNorm
             0.31864864 = fieldWeight in 5925, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.0625 = fieldNorm(doc=5925)
         0.0080975 = weight(abstract_txt:that in 5925) [ClassicSimilarity], result of:
           0.0080975 = score(doc=5925,freq=4.0), product of:
             0.027339386 = queryWeight, product of:
               1.3718976 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.008410366 = queryNorm
             0.2961844 = fieldWeight in 5925, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=5925)
         0.024141768 = weight(abstract_txt:format in 5925) [ClassicSimilarity], result of:
           0.024141768 = score(doc=5925,freq=1.0), product of:
             0.075823635 = queryWeight, product of:
               1.7697227 = boost
               5.0942993 = idf(docFreq=736, maxDocs=44218)
               0.008410366 = queryNorm
             0.3183937 = fieldWeight in 5925, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0942993 = idf(docFreq=736, maxDocs=44218)
               0.0625 = fieldNorm(doc=5925)
         0.6311179 = weight(abstract_txt:blind in 5925) [ClassicSimilarity], result of:
           0.6311179 = score(doc=5925,freq=4.0), product of:
             0.62849975 = queryWeight, product of:
               9.302397 = boost
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.008410366 = queryNorm
             1.0041656 = fieldWeight in 5925, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.033325 = idf(docFreq=38, maxDocs=44218)
               0.0625 = fieldNorm(doc=5925)
       0.16 = coord(4/25)