Document (#37621)

Author
Cabanac, G.
Preuss, T.
Title
Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees
Source
Journal of the American Society for Information Science and Technology. 64(2013) no.2, S.405-415
Year
2013
Abstract
Peer review supports scientific conferences in selecting high-quality papers for publication. Referees are expected to evaluate submissions equitably according to objective criteria (e.g., originality of the contribution, soundness of the theory, validity of the experiments). We argue that the submission date of papers is a subjective factor playing a role in the way they are evaluated. Indeed, program committee (PC) chairs and referees process submission lists that are usually sorted by paperIDs. This order conveys chronological information, as papers are numbered sequentially upon reception. We show that order effects lead to unconscious favoring of early-submitted papers to the detriment of later-submitted papers. Our point is supported by a study of 42 peer-reviewed conferences in Computer Science showing a decrease in the number of bids placed on submissions with higher paperIDs. It is advised to counterbalance order effects during the bidding phase of peer review by promoting the submissions with fewer bids to potential referees. This manipulation intends to better share bids out among submissions in order to attract qualified referees for all submissions. This would secure reviews from confident referees, who are keen on voicing sharp opinions and recommendations (acceptance or rejection) about submissions. This work contributes to the integrity of peer review, which is mandatory to maintain public trust in science.

Similar documents (author)

  1. Cabanac, G.: Shaping the landscape of research in information systems from the perspective of editorial boards : a scientometric study of 77 leading journals (2012) 2.19
    2.1905942 = sum of:
      2.1905942 = product of:
        4.3811884 = sum of:
          4.3811884 = weight(author_txt:cabanac in 1707) [ClassicSimilarity], result of:
            4.3811884 = score(doc=1707,freq=1.0), product of:
              0.73054785 = queryWeight, product of:
                1.0343274 = boost
                9.595404 = idf(docFreq=7, maxDocs=43254)
                0.073608406 = queryNorm
              5.9971275 = fieldWeight in 1707, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.595404 = idf(docFreq=7, maxDocs=43254)
                0.625 = fieldNorm(doc=1707)
        0.5 = coord(1/2)
    
  2. Cabanac, G.: Bibliogifts in LibGen? : a study of a text-sharing platform driven by biblioleaks and crowdsourcing (2016) 2.19
    2.1905942 = sum of:
      2.1905942 = product of:
        4.3811884 = sum of:
          4.3811884 = weight(author_txt:cabanac in 4315) [ClassicSimilarity], result of:
            4.3811884 = score(doc=4315,freq=1.0), product of:
              0.73054785 = queryWeight, product of:
                1.0343274 = boost
                9.595404 = idf(docFreq=7, maxDocs=43254)
                0.073608406 = queryNorm
              5.9971275 = fieldWeight in 4315, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.595404 = idf(docFreq=7, maxDocs=43254)
                0.625 = fieldNorm(doc=4315)
        0.5 = coord(1/2)
    
  3. Preuss, V.: User-centered counseling in online retrieval : practise and training (1983) 1.98
    1.9796475 = sum of:
      1.9796475 = product of:
        3.959295 = sum of:
          3.959295 = weight(author_txt:preuss in 5165) [ClassicSimilarity], result of:
            3.959295 = score(doc=5165,freq=1.0), product of:
              0.6828615 = queryWeight, product of:
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.073608406 = queryNorm
              5.798094 = fieldWeight in 5165, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.625 = fieldNorm(doc=5165)
        0.5 = coord(1/2)
    
  4. Preuss, V.: Benutzerorientierte Informationsberatung (1984) 1.98
    1.9796475 = sum of:
      1.9796475 = product of:
        3.959295 = sum of:
          3.959295 = weight(author_txt:preuss in 5166) [ClassicSimilarity], result of:
            3.959295 = score(doc=5166,freq=1.0), product of:
              0.6828615 = queryWeight, product of:
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.073608406 = queryNorm
              5.798094 = fieldWeight in 5166, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.625 = fieldNorm(doc=5166)
        0.5 = coord(1/2)
    
  5. Preuss, L.: Grundlage für eine quantitative Logik (1995) 1.98
    1.9796475 = sum of:
      1.9796475 = product of:
        3.959295 = sum of:
          3.959295 = weight(author_txt:preuss in 2207) [ClassicSimilarity], result of:
            3.959295 = score(doc=2207,freq=1.0), product of:
              0.6828615 = queryWeight, product of:
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.073608406 = queryNorm
              5.798094 = fieldWeight in 2207, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.27695 = idf(docFreq=10, maxDocs=43254)
                0.625 = fieldNorm(doc=2207)
        0.5 = coord(1/2)
    

Similar documents (content)

  1. Zhao, Y.W.; Chi. C.-H.; Heuvel, W.J. van den: Imperfect referees : reducing the impact of multiple biases in peer review (2015) 0.23
    0.22943923 = sum of:
      0.22943923 = product of:
        0.9559968 = sum of:
          0.004105324 = weight(abstract_txt:this in 3736) [ClassicSimilarity], result of:
            0.004105324 = score(doc=3736,freq=1.0), product of:
              0.027014744 = queryWeight, product of:
                1.0846492 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0102434205 = queryNorm
              0.15196605 = fieldWeight in 3736, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0625 = fieldNorm(doc=3736)
          0.024526259 = weight(abstract_txt:reviews in 3736) [ClassicSimilarity], result of:
            0.024526259 = score(doc=3736,freq=2.0), product of:
              0.056032084 = queryWeight, product of:
                1.1045675 = boost
                4.952215 = idf(docFreq=830, maxDocs=43254)
                0.0102434205 = queryNorm
              0.43771812 = fieldWeight in 3736, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.952215 = idf(docFreq=830, maxDocs=43254)
                0.0625 = fieldNorm(doc=3736)
          0.04979291 = weight(abstract_txt:review in 3736) [ClassicSimilarity], result of:
            0.04979291 = score(doc=3736,freq=4.0), product of:
              0.0816234 = queryWeight, product of:
                1.6327769 = boost
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.0102434205 = queryNorm
              0.6100323 = fieldWeight in 3736, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.0625 = fieldNorm(doc=3736)
          0.07023412 = weight(abstract_txt:conferences in 3736) [ClassicSimilarity], result of:
            0.07023412 = score(doc=3736,freq=1.0), product of:
              0.16296278 = queryWeight, product of:
                2.3070858 = boost
                6.8957214 = idf(docFreq=118, maxDocs=43254)
                0.0102434205 = queryNorm
              0.4309826 = fieldWeight in 3736, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.8957214 = idf(docFreq=118, maxDocs=43254)
                0.0625 = fieldNorm(doc=3736)
          0.17526543 = weight(abstract_txt:peer in 3736) [ClassicSimilarity], result of:
            0.17526543 = score(doc=3736,freq=3.0), product of:
              0.24646941 = queryWeight, product of:
                3.6629028 = boost
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.0102434205 = queryNorm
              0.7111042 = fieldWeight in 3736, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.0625 = fieldNorm(doc=3736)
          0.63207275 = weight(abstract_txt:referees in 3736) [ClassicSimilarity], result of:
            0.63207275 = score(doc=3736,freq=4.0), product of:
              0.55961716 = queryWeight, product of:
                6.0461655 = boost
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.0102434205 = queryNorm
              1.1294734 = fieldWeight in 3736, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.0625 = fieldNorm(doc=3736)
        0.24 = coord(6/25)
    
  2. Casnici, N.; Grimaldo, F.; Gilbert, N.; Squazzoni, F.: Attitudes of referees in a multidisciplinary journal : an empirical analysis (2017) 0.16
    0.1571447 = sum of:
      0.1571447 = product of:
        0.78572345 = sum of:
          0.004105324 = weight(abstract_txt:this in 5157) [ClassicSimilarity], result of:
            0.004105324 = score(doc=5157,freq=1.0), product of:
              0.027014744 = queryWeight, product of:
                1.0846492 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0102434205 = queryNorm
              0.15196605 = fieldWeight in 5157, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0625 = fieldNorm(doc=5157)
          0.017342683 = weight(abstract_txt:reviews in 5157) [ClassicSimilarity], result of:
            0.017342683 = score(doc=5157,freq=1.0), product of:
              0.056032084 = queryWeight, product of:
                1.1045675 = boost
                4.952215 = idf(docFreq=830, maxDocs=43254)
                0.0102434205 = queryNorm
              0.30951345 = fieldWeight in 5157, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.952215 = idf(docFreq=830, maxDocs=43254)
                0.0625 = fieldNorm(doc=5157)
          0.03270015 = weight(abstract_txt:reviewed in 5157) [ClassicSimilarity], result of:
            0.03270015 = score(doc=5157,freq=1.0), product of:
              0.0855183 = queryWeight, product of:
                1.3645939 = boost
                6.1180167 = idf(docFreq=258, maxDocs=43254)
                0.0102434205 = queryNorm
              0.38237604 = fieldWeight in 5157, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.1180167 = idf(docFreq=258, maxDocs=43254)
                0.0625 = fieldNorm(doc=5157)
          0.024896456 = weight(abstract_txt:review in 5157) [ClassicSimilarity], result of:
            0.024896456 = score(doc=5157,freq=1.0), product of:
              0.0816234 = queryWeight, product of:
                1.6327769 = boost
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.0102434205 = queryNorm
              0.30501616 = fieldWeight in 5157, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.0625 = fieldNorm(doc=5157)
          0.7066788 = weight(abstract_txt:referees in 5157) [ClassicSimilarity], result of:
            0.7066788 = score(doc=5157,freq=5.0), product of:
              0.55961716 = queryWeight, product of:
                6.0461655 = boost
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.0102434205 = queryNorm
              1.2627897 = fieldWeight in 5157, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.0625 = fieldNorm(doc=5157)
        0.2 = coord(5/25)
    
  3. Research and advanced technology for digital libraries : 11th European conference, ECDL 2007 / Budapest, Hungary, September 16-21, 2007, proceedings (2007) 0.15
    0.14696169 = sum of:
      0.14696169 = product of:
        0.73480844 = sum of:
          0.006157986 = weight(abstract_txt:this in 4431) [ClassicSimilarity], result of:
            0.006157986 = score(doc=4431,freq=1.0), product of:
              0.027014744 = queryWeight, product of:
                1.0846492 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0102434205 = queryNorm
              0.22794908 = fieldWeight in 4431, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.09375 = fieldNorm(doc=4431)
          0.049050223 = weight(abstract_txt:reviewed in 4431) [ClassicSimilarity], result of:
            0.049050223 = score(doc=4431,freq=1.0), product of:
              0.0855183 = queryWeight, product of:
                1.3645939 = boost
                6.1180167 = idf(docFreq=258, maxDocs=43254)
                0.0102434205 = queryNorm
              0.57356405 = fieldWeight in 4431, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.1180167 = idf(docFreq=258, maxDocs=43254)
                0.09375 = fieldNorm(doc=4431)
          0.13767001 = weight(abstract_txt:papers in 4431) [ClassicSimilarity], result of:
            0.13767001 = score(doc=4431,freq=3.0), product of:
              0.16012686 = queryWeight, product of:
                2.9524055 = boost
                5.2947226 = idf(docFreq=589, maxDocs=43254)
                0.0102434205 = queryNorm
              0.8597558 = fieldWeight in 4431, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                5.2947226 = idf(docFreq=589, maxDocs=43254)
                0.09375 = fieldNorm(doc=4431)
          0.21465544 = weight(abstract_txt:peer in 4431) [ClassicSimilarity], result of:
            0.21465544 = score(doc=4431,freq=2.0), product of:
              0.24646941 = queryWeight, product of:
                3.6629028 = boost
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.0102434205 = queryNorm
              0.87092125 = fieldWeight in 4431, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.09375 = fieldNorm(doc=4431)
          0.32727483 = weight(abstract_txt:submissions in 4431) [ClassicSimilarity], result of:
            0.32727483 = score(doc=4431,freq=1.0), product of:
              0.4371333 = queryWeight, product of:
                5.3436923 = boost
                7.9859657 = idf(docFreq=39, maxDocs=43254)
                0.0102434205 = queryNorm
              0.7486843 = fieldWeight in 4431, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.9859657 = idf(docFreq=39, maxDocs=43254)
                0.09375 = fieldNorm(doc=4431)
        0.2 = coord(5/25)
    
  4. Meho, L.I.; Sonnenwald, D.H.: Citation ranking versus peer evaluation of senior faculty research performance : a case study of Kurdish scholarship (2000) 0.14
    0.14194131 = sum of:
      0.14194131 = product of:
        0.70970654 = sum of:
          0.008888287 = weight(abstract_txt:this in 6383) [ClassicSimilarity], result of:
            0.008888287 = score(doc=6383,freq=3.0), product of:
              0.027014744 = queryWeight, product of:
                1.0846492 = boost
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.0102434205 = queryNorm
              0.32901615 = fieldWeight in 6383, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.4314568 = idf(docFreq=10335, maxDocs=43254)
                0.078125 = fieldNorm(doc=6383)
          0.021678355 = weight(abstract_txt:reviews in 6383) [ClassicSimilarity], result of:
            0.021678355 = score(doc=6383,freq=1.0), product of:
              0.056032084 = queryWeight, product of:
                1.1045675 = boost
                4.952215 = idf(docFreq=830, maxDocs=43254)
                0.0102434205 = queryNorm
              0.3868918 = fieldWeight in 6383, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.952215 = idf(docFreq=830, maxDocs=43254)
                0.078125 = fieldNorm(doc=6383)
          0.031120569 = weight(abstract_txt:review in 6383) [ClassicSimilarity], result of:
            0.031120569 = score(doc=6383,freq=1.0), product of:
              0.0816234 = queryWeight, product of:
                1.6327769 = boost
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.0102434205 = queryNorm
              0.3812702 = fieldWeight in 6383, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.078125 = fieldNorm(doc=6383)
          0.25297388 = weight(abstract_txt:peer in 6383) [ClassicSimilarity], result of:
            0.25297388 = score(doc=6383,freq=4.0), product of:
              0.24646941 = queryWeight, product of:
                3.6629028 = boost
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.0102434205 = queryNorm
              1.0263906 = fieldWeight in 6383, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.078125 = fieldNorm(doc=6383)
          0.39504546 = weight(abstract_txt:referees in 6383) [ClassicSimilarity], result of:
            0.39504546 = score(doc=6383,freq=1.0), product of:
              0.55961716 = queryWeight, product of:
                6.0461655 = boost
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.0102434205 = queryNorm
              0.70592093 = fieldWeight in 6383, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.078125 = fieldNorm(doc=6383)
        0.2 = coord(5/25)
    
  5. Campanario, J.M.; Acedo, E.: Rejecting highly cited papers : the views of scientists who encounter resistance to their discoveries from other scientists (2007) 0.13
    0.12959339 = sum of:
      0.12959339 = product of:
        0.8099587 = sum of:
          0.031120569 = weight(abstract_txt:review in 2274) [ClassicSimilarity], result of:
            0.031120569 = score(doc=2274,freq=1.0), product of:
              0.0816234 = queryWeight, product of:
                1.6327769 = boost
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.0102434205 = queryNorm
              0.3812702 = fieldWeight in 2274, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.8802586 = idf(docFreq=892, maxDocs=43254)
                0.078125 = fieldNorm(doc=2274)
          0.09367257 = weight(abstract_txt:papers in 2274) [ClassicSimilarity], result of:
            0.09367257 = score(doc=2274,freq=2.0), product of:
              0.16012686 = queryWeight, product of:
                2.9524055 = boost
                5.2947226 = idf(docFreq=589, maxDocs=43254)
                0.0102434205 = queryNorm
              0.5849897 = fieldWeight in 2274, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.2947226 = idf(docFreq=589, maxDocs=43254)
                0.078125 = fieldNorm(doc=2274)
          0.12648694 = weight(abstract_txt:peer in 2274) [ClassicSimilarity], result of:
            0.12648694 = score(doc=2274,freq=1.0), product of:
              0.24646941 = queryWeight, product of:
                3.6629028 = boost
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.0102434205 = queryNorm
              0.5131953 = fieldWeight in 2274, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.5688996 = idf(docFreq=164, maxDocs=43254)
                0.078125 = fieldNorm(doc=2274)
          0.5586786 = weight(abstract_txt:referees in 2274) [ClassicSimilarity], result of:
            0.5586786 = score(doc=2274,freq=2.0), product of:
              0.55961716 = queryWeight, product of:
                6.0461655 = boost
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.0102434205 = queryNorm
              0.9983229 = fieldWeight in 2274, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                9.035788 = idf(docFreq=13, maxDocs=43254)
                0.078125 = fieldNorm(doc=2274)
        0.16 = coord(4/25)