Document (#39272)

Author

Zhao, Y.W.
Chi. C.-H.
Heuvel, W.J. van den

Title

Imperfect referees : reducing the impact of multiple biases in peer review

Source

Journal of the Association for Information Science and Technology. 66(2015) no.11, S.2340-2356

Year

2015

Abstract

Bias in peer review entails systematic prejudice that prevents accurate and objective assessment of scientific studies. The disparity between referees' opinions on the same paper typically makes it difficult to judge the paper's quality. This article presents a comprehensive study of peer review biases with regard to 2 aspects of referees: the static profiles (factual authority and self-reported confidence) and the dynamic behavioral context (the temporal ordering of reviews by a single reviewer), exploiting anonymized, real-world review reports of 2 different international conferences in information systems / computer science. Our work extends conventional bias research by considering multiple biases occurring simultaneously. Our findings show that the referees' static profiles are more dominant in peer review bias when compared to their dynamic behavioral context. Of the static profiles, self-reported confidence improved both conference fitness and impact-based bias reductions, while factual authority could only contribute to conference fitness-based bias reduction. Our results also clearly show that the reliability of referees' judgments varies along their static profiles and is contingent on the temporal interval between 2 consecutive reviews.

Content

Vgl.: http://onlinelibrary.wiley.com/doi/10.1002/asi.23317/abstract.

Similar documents (author)

1. Heuvel, C. van den: Web 2.0 and the Semantic Web in research from a historical perspective : the designs of Paul Otlet (1868-1944) for telecommunication and machine readable documentation to organize research and society (2009) 1.90

   1.8962314 = sum of:
     1.8962314 = product of:
       3.7924628 = sum of:
         3.7924628 = weight(author_txt:heuvel in 3251) [ClassicSimilarity], result of:
           3.7924628 = score(doc=3251,freq=1.0), product of:
             0.8305932 = queryWeight, product of:
               1.2212752 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.074475236 = queryNorm
             4.565969 = fieldWeight in 3251, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.5 = fieldNorm(doc=3251)
       0.5 = coord(1/2)
   

2. Heuvel, C. van den: Multidimensional classifications : past and future conceptualizations and visualizations (2012) 1.90

   1.8962314 = sum of:
     1.8962314 = product of:
       3.7924628 = sum of:
         3.7924628 = weight(author_txt:heuvel in 632) [ClassicSimilarity], result of:
           3.7924628 = score(doc=632,freq=1.0), product of:
             0.8305932 = queryWeight, product of:
               1.2212752 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.074475236 = queryNorm
             4.565969 = fieldWeight in 632, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.5 = fieldNorm(doc=632)
       0.5 = coord(1/2)
   

3. Heuvel, C. van den: ¬The Decimal Office : administration as a science in the Netherlands in the first decades of the twentieth century (2013) 1.90

   1.8962314 = sum of:
     1.8962314 = product of:
       3.7924628 = sum of:
         3.7924628 = weight(author_txt:heuvel in 5521) [ClassicSimilarity], result of:
           3.7924628 = score(doc=5521,freq=1.0), product of:
             0.8305932 = queryWeight, product of:
               1.2212752 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.074475236 = queryNorm
             4.565969 = fieldWeight in 5521, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.5 = fieldNorm(doc=5521)
       0.5 = coord(1/2)
   

4. Van den Heuvel, C. -> Heuvel, C. van den: 1.68

   1.6760477 = sum of:
     1.6760477 = product of:
       3.3520954 = sum of:
         3.3520954 = weight(author_txt:heuvel in 2293) [ClassicSimilarity], result of:
           3.3520954 = score(doc=2293,freq=2.0), product of:
             0.8305932 = queryWeight, product of:
               1.2212752 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.074475236 = queryNorm
             4.0357847 = fieldWeight in 2293, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.3125 = fieldNorm(doc=2293)
       0.5 = coord(1/2)
   

5. Papazoglou, M.P.; Heuvel, W.-J. van den: From business processes to cooperative information systems : an information agents perspective (1999) 1.42

   1.4221735 = sum of:
     1.4221735 = product of:
       2.844347 = sum of:
         2.844347 = weight(author_txt:heuvel in 3974) [ClassicSimilarity], result of:
           2.844347 = score(doc=3974,freq=1.0), product of:
             0.8305932 = queryWeight, product of:
               1.2212752 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.074475236 = queryNorm
             3.4244766 = fieldWeight in 3974, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.375 = fieldNorm(doc=3974)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Sun, M.; Danfa, J.B.; Teplitskiy, M.: Does double-blind peer review reduce bias? : evidence from a top computer science conference (2022) 0.18

   0.18307683 = sum of:
     0.18307683 = product of:
       0.7628201 = sum of:
         0.01688253 = weight(abstract_txt:impact in 562) [ClassicSimilarity], result of:
           0.01688253 = score(doc=562,freq=1.0), product of:
             0.05887767 = queryWeight, product of:
               1.0047867 = boost
               4.5878253 = idf(docFreq=1222, maxDocs=44218)
               0.012772322 = queryNorm
             0.28673908 = fieldWeight in 562, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5878253 = idf(docFreq=1222, maxDocs=44218)
               0.0625 = fieldNorm(doc=562)
         0.030106382 = weight(abstract_txt:conference in 562) [ClassicSimilarity], result of:
           0.030106382 = score(doc=562,freq=1.0), product of:
             0.08658274 = queryWeight, product of:
               1.2184684 = boost
               5.563489 = idf(docFreq=460, maxDocs=44218)
               0.012772322 = queryNorm
             0.34771806 = fieldWeight in 562, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.563489 = idf(docFreq=460, maxDocs=44218)
               0.0625 = fieldNorm(doc=562)
         0.12186549 = weight(abstract_txt:biases in 562) [ClassicSimilarity], result of:
           0.12186549 = score(doc=562,freq=1.0), product of:
             0.25173476 = queryWeight, product of:
               2.5445783 = boost
               7.7456436 = idf(docFreq=51, maxDocs=44218)
               0.012772322 = queryNorm
             0.48410273 = fieldWeight in 562, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.7456436 = idf(docFreq=51, maxDocs=44218)
               0.0625 = fieldNorm(doc=562)
         0.14177631 = weight(abstract_txt:review in 562) [ClassicSimilarity], result of:
           0.14177631 = score(doc=562,freq=8.0), product of:
             0.16507378 = queryWeight, product of:
               2.6601593 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.012772322 = queryNorm
             0.85886633 = fieldWeight in 562, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.0625 = fieldNorm(doc=562)
         0.19190851 = weight(abstract_txt:peer in 562) [ClassicSimilarity], result of:
           0.19190851 = score(doc=562,freq=4.0), product of:
             0.2362536 = queryWeight, product of:
               2.846445 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.012772322 = queryNorm
             0.8122988 = fieldWeight in 562, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.0625 = fieldNorm(doc=562)
         0.2602809 = weight(abstract_txt:bias in 562) [ClassicSimilarity], result of:
           0.2602809 = score(doc=562,freq=3.0), product of:
             0.34321013 = queryWeight, product of:
               3.8357356 = boost
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.012772322 = queryNorm
             0.7583719 = fieldWeight in 562, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.0625 = fieldNorm(doc=562)
       0.24 = coord(6/25)
   

2. García, J.A.; Rodriguez-Sánchez, R.; Fdez-Valdivia, J.: Bias and effort in peer review (2015) 0.17

   0.16762169 = sum of:
     0.16762169 = product of:
       1.0476356 = sum of:
         0.10852489 = weight(abstract_txt:review in 2121) [ClassicSimilarity], result of:
           0.10852489 = score(doc=2121,freq=3.0), product of:
             0.16507378 = queryWeight, product of:
               2.6601593 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.012772322 = queryNorm
             0.6574326 = fieldWeight in 2121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
         0.20774706 = weight(abstract_txt:peer in 2121) [ClassicSimilarity], result of:
           0.20774706 = score(doc=2121,freq=3.0), product of:
             0.2362536 = queryWeight, product of:
               2.846445 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.012772322 = queryNorm
             0.8793392 = fieldWeight in 2121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
         0.32535115 = weight(abstract_txt:bias in 2121) [ClassicSimilarity], result of:
           0.32535115 = score(doc=2121,freq=3.0), product of:
             0.34321013 = queryWeight, product of:
               3.8357356 = boost
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.012772322 = queryNorm
             0.94796485 = fieldWeight in 2121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
         0.40601248 = weight(abstract_txt:referees in 2121) [ClassicSimilarity], result of:
           0.40601248 = score(doc=2121,freq=1.0), product of:
             0.5737533 = queryWeight, product of:
               4.9594235 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.012772322 = queryNorm
             0.707643 = fieldWeight in 2121, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.078125 = fieldNorm(doc=2121)
       0.16 = coord(4/25)
   

3. Cabanac, G.; Preuss, T.: Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees (2013) 0.16

   0.16420266 = sum of:
     0.16420266 = product of:
       1.0262667 = sum of:
         0.021241082 = weight(abstract_txt:reviews in 619) [ClassicSimilarity], result of:
           0.021241082 = score(doc=619,freq=1.0), product of:
             0.06861884 = queryWeight, product of:
               1.0847266 = boost
               4.952828 = idf(docFreq=848, maxDocs=44218)
               0.012772322 = queryNorm
             0.30955175 = fieldWeight in 619, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.952828 = idf(docFreq=848, maxDocs=44218)
               0.0625 = fieldNorm(doc=619)
         0.08681991 = weight(abstract_txt:review in 619) [ClassicSimilarity], result of:
           0.08681991 = score(doc=619,freq=3.0), product of:
             0.16507378 = queryWeight, product of:
               2.6601593 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.012772322 = queryNorm
             0.5259461 = fieldWeight in 619, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.0625 = fieldNorm(doc=619)
         0.19190851 = weight(abstract_txt:peer in 619) [ClassicSimilarity], result of:
           0.19190851 = score(doc=619,freq=4.0), product of:
             0.2362536 = queryWeight, product of:
               2.846445 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.012772322 = queryNorm
             0.8122988 = fieldWeight in 619, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.0625 = fieldNorm(doc=619)
         0.7262972 = weight(abstract_txt:referees in 619) [ClassicSimilarity], result of:
           0.7262972 = score(doc=619,freq=5.0), product of:
             0.5737533 = queryWeight, product of:
               4.9594235 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.012772322 = queryNorm
             1.2658702 = fieldWeight in 619, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.0625 = fieldNorm(doc=619)
       0.16 = coord(4/25)
   

4. Lee, C.J.; Sugimoto, C.R.; Zhang, G.; Cronin, B.: Bias in peer review (2013) 0.14

   0.13576794 = sum of:
     0.13576794 = product of:
       0.8485496 = sum of:
         0.030071221 = weight(abstract_txt:self in 525) [ClassicSimilarity], result of:
           0.030071221 = score(doc=525,freq=1.0), product of:
             0.086515315 = queryWeight, product of:
               1.2179939 = boost
               5.561322 = idf(docFreq=461, maxDocs=44218)
               0.012772322 = queryNorm
             0.34758264 = fieldWeight in 525, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.561322 = idf(docFreq=461, maxDocs=44218)
               0.0625 = fieldNorm(doc=525)
         0.1326196 = weight(abstract_txt:review in 525) [ClassicSimilarity], result of:
           0.1326196 = score(doc=525,freq=7.0), product of:
             0.16507378 = queryWeight, product of:
               2.6601593 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.012772322 = queryNorm
             0.8033959 = fieldWeight in 525, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.0625 = fieldNorm(doc=525)
         0.23503898 = weight(abstract_txt:peer in 525) [ClassicSimilarity], result of:
           0.23503898 = score(doc=525,freq=6.0), product of:
             0.2362536 = queryWeight, product of:
               2.846445 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.012772322 = queryNorm
             0.9948588 = fieldWeight in 525, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.0625 = fieldNorm(doc=525)
         0.4508198 = weight(abstract_txt:bias in 525) [ClassicSimilarity], result of:
           0.4508198 = score(doc=525,freq=9.0), product of:
             0.34321013 = queryWeight, product of:
               3.8357356 = boost
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.012772322 = queryNorm
             1.3135387 = fieldWeight in 525, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.0625 = fieldNorm(doc=525)
       0.16 = coord(4/25)
   

5. Campanario, J.M.; Acedo, E.: Rejecting highly cited papers : the views of scientists who encounter resistance to their discoveries from other scientists (2007) 0.12

   0.12446259 = sum of:
     0.12446259 = product of:
       0.7778912 = sum of:
         0.021103162 = weight(abstract_txt:impact in 273) [ClassicSimilarity], result of:
           0.021103162 = score(doc=273,freq=1.0), product of:
             0.05887767 = queryWeight, product of:
               1.0047867 = boost
               4.5878253 = idf(docFreq=1222, maxDocs=44218)
               0.012772322 = queryNorm
             0.35842386 = fieldWeight in 273, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5878253 = idf(docFreq=1222, maxDocs=44218)
               0.078125 = fieldNorm(doc=273)
         0.06265687 = weight(abstract_txt:review in 273) [ClassicSimilarity], result of:
           0.06265687 = score(doc=273,freq=1.0), product of:
             0.16507378 = queryWeight, product of:
               2.6601593 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.012772322 = queryNorm
             0.3795689 = fieldWeight in 273, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.078125 = fieldNorm(doc=273)
         0.11994282 = weight(abstract_txt:peer in 273) [ClassicSimilarity], result of:
           0.11994282 = score(doc=273,freq=1.0), product of:
             0.2362536 = queryWeight, product of:
               2.846445 = boost
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.012772322 = queryNorm
             0.50768673 = fieldWeight in 273, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.49839 = idf(docFreq=180, maxDocs=44218)
               0.078125 = fieldNorm(doc=273)
         0.57418835 = weight(abstract_txt:referees in 273) [ClassicSimilarity], result of:
           0.57418835 = score(doc=273,freq=2.0), product of:
             0.5737533 = queryWeight, product of:
               4.9594235 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.012772322 = queryNorm
             1.0007583 = fieldWeight in 273, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.078125 = fieldNorm(doc=273)
       0.16 = coord(4/25)