Search (3 results, page 1 of 1)

0.016424898 = product of:
  0.032849796 = sum of:
    0.032849796 = product of:
      0.06569959 = sum of:
        0.06569959 = weight(_text_:n in 4132) [ClassicSimilarity], result of:
          0.06569959 = score(doc=4132,freq=4.0), product of:
            0.19504215 = queryWeight, product of:
              4.3116565 = idf(docFreq=1611, maxDocs=44218)
              0.045236014 = queryNorm
            0.33684817 = fieldWeight in 4132, product of:
              2.0 = tf(freq=4.0), with freq of:
                4.0 = termFreq=4.0
              4.3116565 = idf(docFreq=1611, maxDocs=44218)
              0.0390625 = fieldNorm(doc=4132)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

In a comprehensive research project, we investigated the predictive validity of selection decisions and reviewers' ratings at the open access journal Atmospheric Chemistry and Physics (ACP). ACP is a high-impact journal publishing papers on the Earth's atmosphere and the underlying chemical and physical processes. Scientific journals have to deal with the following question concerning the predictive validity: Are in fact the "best" scientific works selected from the manuscripts submitted? In this study we examined whether selecting the "best" manuscripts means selecting papers that after publication show top citation performance as compared to other papers in this research area. First, we appraised the citation impact of later published manuscripts based on the percentile citedness rank classes of the population distribution (scaling in a specific subfield). Second, we analyzed the association between the decisions (n = 677 accepted or rejected, but published elsewhere manuscripts) or ratings (reviewers' ratings for n = 315 manuscripts), respectively, and the citation impact classes of the manuscripts. The results confirm the predictive validity of the ACP peer review system.

0.016424898 = product of:
  0.032849796 = sum of:
    0.032849796 = product of:
      0.06569959 = sum of:
        0.06569959 = weight(_text_:n in 3792) [ClassicSimilarity], result of:
          0.06569959 = score(doc=3792,freq=4.0), product of:
            0.19504215 = queryWeight, product of:
              4.3116565 = idf(docFreq=1611, maxDocs=44218)
              0.045236014 = queryNorm
            0.33684817 = fieldWeight in 3792, product of:
              2.0 = tf(freq=4.0), with freq of:
                4.0 = termFreq=4.0
              4.3116565 = idf(docFreq=1611, maxDocs=44218)
              0.0390625 = fieldNorm(doc=3792)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

Scientific journals publish an increasing number of articles every year. To steer readers' attention to the most important papers, journals use several techniques (e.g., lead paper). Angewandte Chemie International Edition (AC), a leading international journal in chemistry, signals high-quality papers through designating them as a "very important paper" (VIP). This study aims to investigate the citation impact of Communications in AC receiving the special feature VIP, both cumulated and over time. Using propensity score matching, treatment group (VIP) and control group (non-VIP) were balanced for 14 covariates to estimate the unconfounded "average treatment effect on the treated" for the VIP designation. Out of N = 3,011 Communications published in 2007 and 2008, N = 207 received the special feature VIP. For each Communication, data were collected from AC (e.g., referees' ratings) and from the databases Chemical Abstracts (e.g., sections) and the Web of Science (e.g., citations). The estimated unconfounded average treatment effect on the treated (that is, Communications designated as a VIP) was statistically significant and amounted to 19.83 citations. In addition, the special feature VIP fostered the cumulated annual citation growth. For instance, the time until a Communication reached its maximum annual number of citations, was reduced.

0.011614156 = product of:
  0.023228312 = sum of:
    0.023228312 = product of:
      0.046456624 = sum of:
        0.046456624 = weight(_text_:n in 2270) [ClassicSimilarity], result of:
          0.046456624 = score(doc=2270,freq=2.0), product of:
            0.19504215 = queryWeight, product of:
              4.3116565 = idf(docFreq=1611, maxDocs=44218)
              0.045236014 = queryNorm
            0.23818761 = fieldWeight in 2270, product of:
              1.4142135 = tf(freq=2.0), with freq of:
                2.0 = termFreq=2.0
              4.3116565 = idf(docFreq=1611, maxDocs=44218)
              0.0390625 = fieldNorm(doc=2270)
      0.5 = coord(1/2)
  0.5 = coord(1/2)

Abstract

It is essential for research funding organizations to ensure both the validity and fairness of the grant approval procedure. The ex-ante peer evaluation (EXANTE) of N?=?8,496 grant applications submitted to the Austrian Science Fund from 1999 to 2009 was statistically analyzed. For 1,689 funded research projects an ex-post peer evaluation (EXPOST) was also available; for the rest of the grant applications a multilevel missing data imputation approach was used to consider verification bias for the first time in peer-review research. Without imputation, the predictive validity of EXANTE was low (r?=?.26) but underestimated due to verification bias, and with imputation it was r?=?.49. That is, the decision-making procedure is capable of selecting the best research proposals for funding. In the EXANTE there were several potential biases (e.g., gender). With respect to the EXPOST there was only one real bias (discipline-specific and year-specific differential prediction). The novelty of this contribution is, first, the combining of theoretical concepts of validity and fairness with a missing data imputation approach to correct for verification bias and, second, multilevel modeling to test peer review-based funding decisions for both validity and fairness in terms of potential and real biases.