Background

Vast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers.

Methods

A total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018.

Results

A total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups.

Conclusions

We did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed.

Trial registration

The study was preregistered at OSF.io/n4fq3.

中文翻译：

---

个人与一般结构化反馈以提高拨款同行评审的一致性：一项随机对照试验

背景

巨额资金是根据同行评审来分配的，但研究表明，评估者之间的可靠性往往很低。在这项研究中，我们测试了收到两份简短的个人反馈报告与一份简短的一般反馈报告对审稿人之间一致性的影响。

方法

挪威基金会大坝共有 42 名评审员被随机分配接收一般反馈报告或个人反馈报告。一般反馈小组在评审开始前收到一份报告，其中包含有关评审员参与的上一次通话的一般信息。在个人反馈小组中，审核人员收到两份报告，一份在审核期之前（基于之前的通话），一份在审核期间（基于当前的通话）。在个人反馈小组中，在审核之前和审核期间，向审核者提供了与整个审核委员会相比的评分的详细信息。主要结果是资格评估中的一致比例以及评估同一提案的评审员之间的平均分数差异。结果于 2017 年进行测量，并于 2018 年提供反馈后进行。

结果

分析中总共包含 2398 条配对评论。两组对提案是否符合资助计划的绝对同意比例存在显着差异，一般反馈组的同意率较高。两组之间的平均分数差异没有差异。然而，对于两个小组来说，关于提案分数的一致性仍然非常低。

结论

我们没有观察到 2017 年至 2018 年收到不同反馈的审稿人的提案分数一致性发生变化。一致程度低仍然是资助同行评审的一个主要问题，仍然需要进行研究以确定影响因素以及开发和测试干预措施以提高一致率。

试用注册

该研究已在 OSF.io/n4fq3 上预先注册。