Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers’ evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers’ ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers “translated” a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.

由于过去十年来资助的成功率下降了，因此从美国国立卫生研究院（NIH）获得赠款的竞争越来越激烈。要分配相对稀缺的资金，科学的同行评审员必须将最佳申请与相对较弱的申请区分开来。尽管此决定很重要，但很少有研究探索审阅者如何为他们审阅的应用程序分配评分，以及审阅者对同一应用程序的评估是否一致。复制了NIH同行评审过程的所有方面，我们检查了43名个人评审的评分，并对同一组25项NIH资助申请书进行了书面评论。结果表明，审阅者在定性或定量评估中对申请质量均未达成一致。尽管所有审阅者都收到了有关如何对申请进行评分和撰写书面评论的方式的相同指示，但我们也没有就审阅者如何将给定数量的优缺点“转化”为数字评分达成一致。看来，拨款审查的结果更多地取决于拨款授予的审查者，而不是拨款中提出的研究。这项研究重复了NIH同行评审过程，以详细审查审查同一申请的不同评审员的定性和定量判断，我们的结果与科学资助的同行评审具有广泛的相关性。看来，拨款审查的结果更多地取决于拨款授予的审查者，而不是拨款中提出的研究。这项研究重复了NIH同行评审过程，以详细审查审查同一申请的不同评审员的定性和定量判断，我们的结果与科学资助的同行评审具有广泛的相关性。看来，拨款审查的结果更多地取决于拨款授予的审查者，而不是拨款中提出的研究。这项研究重复了NIH同行评审过程，以详细审查审查同一申请的不同评审员的定性和定量判断，我们的结果与科学资助的同行评审具有广泛的相关性。