Document (#39273)

Author
Zhao, Y.W.
Chi. C.-H.
Heuvel, W.J. van den
Title
Imperfect referees : reducing the impact of multiple biases in peer review
Source
Journal of the Association for Information Science and Technology. 66(2015) no.11, S.2340-2356
Year
2015
Abstract
Bias in peer review entails systematic prejudice that prevents accurate and objective assessment of scientific studies. The disparity between referees' opinions on the same paper typically makes it difficult to judge the paper's quality. This article presents a comprehensive study of peer review biases with regard to 2 aspects of referees: the static profiles (factual authority and self-reported confidence) and the dynamic behavioral context (the temporal ordering of reviews by a single reviewer), exploiting anonymized, real-world review reports of 2 different international conferences in information systems / computer science. Our work extends conventional bias research by considering multiple biases occurring simultaneously. Our findings show that the referees' static profiles are more dominant in peer review bias when compared to their dynamic behavioral context. Of the static profiles, self-reported confidence improved both conference fitness and impact-based bias reductions, while factual authority could only contribute to conference fitness-based bias reduction. Our results also clearly show that the reliability of referees' judgments varies along their static profiles and is contingent on the temporal interval between 2 consecutive reviews.
Content
Vgl.: http://onlinelibrary.wiley.com/doi/10.1002/asi.23317/abstract.

Similar documents (author)

  1. Heuvel, C. van den: Web 2.0 and the Semantic Web in research from a historical perspective : the designs of Paul Otlet (1868-1944) for telecommunication and machine readable documentation to organize research and society (2009) 1.86
    1.8596053 = sum of:
      1.8596053 = product of:
        3.7192106 = sum of:
          3.7192106 = weight(author_txt:heuvel in 252) [ClassicSimilarity], result of:
            3.7192106 = score(doc=252,freq=1.0), product of:
              0.8113658 = queryWeight, product of:
                1.178153 = boost
                9.167778 = idf(docFreq=11, maxDocs=42306)
                0.07511919 = queryNorm
              4.583889 = fieldWeight in 252, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.167778 = idf(docFreq=11, maxDocs=42306)
                0.5 = fieldNorm(doc=252)
        0.5 = coord(1/2)
    
  2. Heuvel, C. van den: Multidimensional classifications : past and future conceptualizations and visualizations (2012) 1.86
    1.8596053 = sum of:
      1.8596053 = product of:
        3.7192106 = sum of:
          3.7192106 = weight(author_txt:heuvel in 2633) [ClassicSimilarity], result of:
            3.7192106 = score(doc=2633,freq=1.0), product of:
              0.8113658 = queryWeight, product of:
                1.178153 = boost
                9.167778 = idf(docFreq=11, maxDocs=42306)
                0.07511919 = queryNorm
              4.583889 = fieldWeight in 2633, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.167778 = idf(docFreq=11, maxDocs=42306)
                0.5 = fieldNorm(doc=2633)
        0.5 = coord(1/2)
    
  3. Van den Heuvel, C. -> Heuvel, C. van den: 1.64
    1.6436745 = sum of:
      1.6436745 = product of:
        3.287349 = sum of:
          3.287349 = weight(author_txt:heuvel in 2293) [ClassicSimilarity], result of:
            3.287349 = score(doc=2293,freq=2.0), product of:
              0.8113658 = queryWeight, product of:
                1.178153 = boost
                9.167778 = idf(docFreq=11, maxDocs=42306)
                0.07511919 = queryNorm
              4.051624 = fieldWeight in 2293, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                9.167778 = idf(docFreq=11, maxDocs=42306)
                0.3125 = fieldNorm(doc=2293)
        0.5 = coord(1/2)
    
  4. Zhao, L.: Save space for "newcomers" : analyzing problems in book number assignment under the LCC system (2004) 1.42
    1.4214307 = sum of:
      1.4214307 = product of:
        2.8428614 = sum of:
          2.8428614 = weight(author_txt:zhao in 4082) [ClassicSimilarity], result of:
            2.8428614 = score(doc=4082,freq=1.0), product of:
              0.5845387 = queryWeight, product of:
                7.781483 = idf(docFreq=47, maxDocs=42306)
                0.07511919 = queryNorm
              4.863427 = fieldWeight in 4082, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.781483 = idf(docFreq=47, maxDocs=42306)
                0.625 = fieldNorm(doc=4082)
        0.5 = coord(1/2)
    
  5. Zhao, L.: How librarians used e-resources : an analysis of citations in CCQ (2006) 1.42
    1.4214307 = sum of:
      1.4214307 = product of:
        2.8428614 = sum of:
          2.8428614 = weight(author_txt:zhao in 767) [ClassicSimilarity], result of:
            2.8428614 = score(doc=767,freq=1.0), product of:
              0.5845387 = queryWeight, product of:
                7.781483 = idf(docFreq=47, maxDocs=42306)
                0.07511919 = queryNorm
              4.863427 = fieldWeight in 767, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.781483 = idf(docFreq=47, maxDocs=42306)
                0.625 = fieldNorm(doc=767)
        0.5 = coord(1/2)
    

Similar documents (content)

  1. García, J.A.; Rodriguez-Sánchez, R.; Fdez-Valdivia, J.: Bias and effort in peer review (2015) 0.23
    0.23112404 = sum of:
      0.23112404 = product of:
        1.1556202 = sum of:
          0.08164017 = weight(abstract_txt:reviewer in 4122) [ClassicSimilarity], result of:
            0.08164017 = score(doc=4122,freq=1.0), product of:
              0.114989504 = queryWeight, product of:
                1.0082217 = boost
                9.087735 = idf(docFreq=12, maxDocs=42306)
                0.012550079 = queryNorm
              0.7099793 = fieldWeight in 4122, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.087735 = idf(docFreq=12, maxDocs=42306)
                0.078125 = fieldNorm(doc=4122)
          0.11141589 = weight(abstract_txt:review in 4122) [ClassicSimilarity], result of:
            0.11141589 = score(doc=4122,freq=3.0), product of:
              0.16773999 = queryWeight, product of:
                2.722891 = boost
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.012550079 = queryNorm
              0.6642178 = fieldWeight in 4122, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.078125 = fieldNorm(doc=4122)
          0.21951437 = weight(abstract_txt:peer in 4122) [ClassicSimilarity], result of:
            0.21951437 = score(doc=4122,freq=3.0), product of:
              0.24472435 = queryWeight, product of:
                2.941681 = boost
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.012550079 = queryNorm
              0.89698625 = fieldWeight in 4122, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.078125 = fieldNorm(doc=4122)
          0.34475404 = weight(abstract_txt:bias in 4122) [ClassicSimilarity], result of:
            0.34475404 = score(doc=4122,freq=3.0), product of:
              0.35618705 = queryWeight, product of:
                3.9678116 = boost
                7.1528745 = idf(docFreq=89, maxDocs=42306)
                0.012550079 = queryNorm
              0.9679017 = fieldWeight in 4122, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                7.1528745 = idf(docFreq=89, maxDocs=42306)
                0.078125 = fieldNorm(doc=4122)
          0.39829576 = weight(abstract_txt:referees in 4122) [ClassicSimilarity], result of:
            0.39829576 = score(doc=4122,freq=1.0), product of:
              0.5656087 = queryWeight, product of:
                5.0 = boost
                9.013627 = idf(docFreq=13, maxDocs=42306)
                0.012550079 = queryNorm
              0.7041896 = fieldWeight in 4122, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.013627 = idf(docFreq=13, maxDocs=42306)
                0.078125 = fieldNorm(doc=4122)
        0.2 = coord(5/25)
    
  2. Cabanac, G.; Preuss, T.: Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees (2013) 0.16
    0.16405874 = sum of:
      0.16405874 = product of:
        1.0253671 = sum of:
          0.020962628 = weight(abstract_txt:reviews in 2620) [ClassicSimilarity], result of:
            0.020962628 = score(doc=2620,freq=1.0), product of:
              0.06791558 = queryWeight, product of:
                1.0957884 = boost
                4.9385138 = idf(docFreq=823, maxDocs=42306)
                0.012550079 = queryNorm
              0.3086571 = fieldWeight in 2620, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.9385138 = idf(docFreq=823, maxDocs=42306)
                0.0625 = fieldNorm(doc=2620)
          0.08913271 = weight(abstract_txt:review in 2620) [ClassicSimilarity], result of:
            0.08913271 = score(doc=2620,freq=3.0), product of:
              0.16773999 = queryWeight, product of:
                2.722891 = boost
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.012550079 = queryNorm
              0.5313743 = fieldWeight in 2620, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.0625 = fieldNorm(doc=2620)
          0.20277871 = weight(abstract_txt:peer in 2620) [ClassicSimilarity], result of:
            0.20277871 = score(doc=2620,freq=4.0), product of:
              0.24472435 = queryWeight, product of:
                2.941681 = boost
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.012550079 = queryNorm
              0.82860047 = fieldWeight in 2620, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.0625 = fieldNorm(doc=2620)
          0.7124931 = weight(abstract_txt:referees in 2620) [ClassicSimilarity], result of:
            0.7124931 = score(doc=2620,freq=5.0), product of:
              0.5656087 = queryWeight, product of:
                5.0 = boost
                9.013627 = idf(docFreq=13, maxDocs=42306)
                0.012550079 = queryNorm
              1.2596927 = fieldWeight in 2620, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                9.013627 = idf(docFreq=13, maxDocs=42306)
                0.0625 = fieldNorm(doc=2620)
        0.16 = coord(4/25)
    
  3. Lee, C.J.; Sugimoto, C.R.; Zhang, G.; Cronin, B.: Bias in peer review (2013) 0.14
    0.14294675 = sum of:
      0.14294675 = product of:
        0.8934172 = sum of:
          0.031207288 = weight(abstract_txt:self in 2526) [ClassicSimilarity], result of:
            0.031207288 = score(doc=2526,freq=1.0), product of:
              0.0885475 = queryWeight, product of:
                1.2512097 = boost
                5.638969 = idf(docFreq=408, maxDocs=42306)
                0.012550079 = queryNorm
              0.35243556 = fieldWeight in 2526, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.638969 = idf(docFreq=408, maxDocs=42306)
                0.0625 = fieldNorm(doc=2526)
          0.13615246 = weight(abstract_txt:review in 2526) [ClassicSimilarity], result of:
            0.13615246 = score(doc=2526,freq=7.0), product of:
              0.16773999 = queryWeight, product of:
                2.722891 = boost
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.012550079 = queryNorm
              0.8116876 = fieldWeight in 2526, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.0625 = fieldNorm(doc=2526)
          0.24835218 = weight(abstract_txt:peer in 2526) [ClassicSimilarity], result of:
            0.24835218 = score(doc=2526,freq=6.0), product of:
              0.24472435 = queryWeight, product of:
                2.941681 = boost
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.012550079 = queryNorm
              1.0148242 = fieldWeight in 2526, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.0625 = fieldNorm(doc=2526)
          0.47770524 = weight(abstract_txt:bias in 2526) [ClassicSimilarity], result of:
            0.47770524 = score(doc=2526,freq=9.0), product of:
              0.35618705 = queryWeight, product of:
                3.9678116 = boost
                7.1528745 = idf(docFreq=89, maxDocs=42306)
                0.012550079 = queryNorm
              1.341164 = fieldWeight in 2526, product of:
                3.0 = tf(freq=9.0), with freq of:
                  9.0 = termFreq=9.0
                7.1528745 = idf(docFreq=89, maxDocs=42306)
                0.0625 = fieldNorm(doc=2526)
        0.16 = coord(4/25)
    
  4. Campanario, J.M.; Acedo, E.: Rejecting highly cited papers : the views of scientists who encounter resistance to their discoveries from other scientists (2007) 0.12
    0.12414851 = sum of:
      0.12414851 = product of:
        0.7759282 = sum of:
          0.021590274 = weight(abstract_txt:impact in 2274) [ClassicSimilarity], result of:
            0.021590274 = score(doc=2274,freq=1.0), product of:
              0.059690386 = queryWeight, product of:
                1.0272926 = boost
                4.629816 = idf(docFreq=1121, maxDocs=42306)
                0.012550079 = queryNorm
              0.36170438 = fieldWeight in 2274, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.629816 = idf(docFreq=1121, maxDocs=42306)
                0.078125 = fieldNorm(doc=2274)
          0.064325996 = weight(abstract_txt:review in 2274) [ClassicSimilarity], result of:
            0.064325996 = score(doc=2274,freq=1.0), product of:
              0.16773999 = queryWeight, product of:
                2.722891 = boost
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.012550079 = queryNorm
              0.38348633 = fieldWeight in 2274, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.078125 = fieldNorm(doc=2274)
          0.1267367 = weight(abstract_txt:peer in 2274) [ClassicSimilarity], result of:
            0.1267367 = score(doc=2274,freq=1.0), product of:
              0.24472435 = queryWeight, product of:
                2.941681 = boost
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.012550079 = queryNorm
              0.5178753 = fieldWeight in 2274, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.078125 = fieldNorm(doc=2274)
          0.5632752 = weight(abstract_txt:referees in 2274) [ClassicSimilarity], result of:
            0.5632752 = score(doc=2274,freq=2.0), product of:
              0.5656087 = queryWeight, product of:
                5.0 = boost
                9.013627 = idf(docFreq=13, maxDocs=42306)
                0.012550079 = queryNorm
              0.99587446 = fieldWeight in 2274, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                9.013627 = idf(docFreq=13, maxDocs=42306)
                0.078125 = fieldNorm(doc=2274)
        0.16 = coord(4/25)
    
  5. Mutz, R.; Bornmann, L.; Daniel, H.-D.: Testing for the fairness and predictive validity of research funding decisions : a multilevel multiple imputation for missing data approach using ex-ante and ex-post peer evaluation data from the Austrian science fund (2015) 0.12
    0.123104244 = sum of:
      0.123104244 = product of:
        0.76940155 = sum of:
          0.17537616 = weight(abstract_txt:biases in 4271) [ClassicSimilarity], result of:
            0.17537616 = score(doc=4271,freq=2.0), product of:
              0.25429612 = queryWeight, product of:
                2.5969136 = boost
                7.8025365 = idf(docFreq=46, maxDocs=42306)
                0.012550079 = queryNorm
              0.6896533 = fieldWeight in 4271, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.8025365 = idf(docFreq=46, maxDocs=42306)
                0.0625 = fieldNorm(doc=4271)
          0.07277655 = weight(abstract_txt:review in 4271) [ClassicSimilarity], result of:
            0.07277655 = score(doc=4271,freq=2.0), product of:
              0.16773999 = queryWeight, product of:
                2.722891 = boost
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.012550079 = queryNorm
              0.43386525 = fieldWeight in 4271, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.908625 = idf(docFreq=848, maxDocs=42306)
                0.0625 = fieldNorm(doc=4271)
          0.20277871 = weight(abstract_txt:peer in 4271) [ClassicSimilarity], result of:
            0.20277871 = score(doc=4271,freq=4.0), product of:
              0.24472435 = queryWeight, product of:
                2.941681 = boost
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.012550079 = queryNorm
              0.82860047 = fieldWeight in 4271, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.6288037 = idf(docFreq=151, maxDocs=42306)
                0.0625 = fieldNorm(doc=4271)
          0.31847015 = weight(abstract_txt:bias in 4271) [ClassicSimilarity], result of:
            0.31847015 = score(doc=4271,freq=4.0), product of:
              0.35618705 = queryWeight, product of:
                3.9678116 = boost
                7.1528745 = idf(docFreq=89, maxDocs=42306)
                0.012550079 = queryNorm
              0.8941093 = fieldWeight in 4271, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                7.1528745 = idf(docFreq=89, maxDocs=42306)
                0.0625 = fieldNorm(doc=4271)
        0.16 = coord(4/25)