To test a simplified evaluation of fellowship proposals by analyzing the agreement of funding decisions with the official evaluation, and to examine the use of a lottery-based decision for proposals of similar quality. The study involved 134 junior fellowship proposals (Postdoc.Mobility). The official method used two panel reviewers who independently scored the application, followed by triage and discussion of selected applications in a panel. Very competitive/uncompetitive proposals were directly funded/rejected without discussion. The simplified procedure used the scores of the two panel members, with or without the score of an additional, third expert. Both methods could further use a lottery to decide on applications of similar quality close to the funding threshold. The same funding rate was applied, and the agreement between the two methods analyzed. The simplified procedure based on three reviews agreed in 80.6% (95% CI 73.9-87.3) with the official funding outcome. The agreement was 86.6% (95% CI 80.8-92.4) when using the two reviews of the panel members. The agreement between the two methods was lower for the group of applications discussed in the panel (64.2% and 73.1%, respectively), and higher for directly funded/rejected applications (range 96.7% to 100%). The lottery was used in eight (6.0%) of 134 applications (official method), 19 (14.2%) applications (simplified, three reviewers) and 23 (17.2%) applications (simplified, two reviewers). With the simplified procedure, evaluation costs could have been halved and 31 hours of meeting time saved for the two 2019 calls. Agreement between the two methods was high. The simplified procedure could represent a viable evaluation method for the Postdoc.Mobility early career instrument at the SNSF.

中文翻译：

---

如何最好地评估初级奖学金的申请？比较远程评估和面对面的小组会议

通过分析与官方评估的资助决策的一致性来测试研究金提案的简化评估，并检查对类似质量提案的基于彩票的决策的使用。该研究涉及134个初级研究金计划（Postdoc.Mobility）。官方方法使用两名专家评审员对申请进行独立评分，然后对专家小组中选定的申请进行分类和讨论。非常有竞争力/无竞争力的提案未经讨论直接获得资助/被拒绝。简化程序使用两个小组成员的分数，有或没有其他第三位专家的分数。两种方法都可以进一步使用彩票来决定质量接近资金阈值的类似应用。采用相同的资助率，并分析了两种方法之间的一致性。基于三项审查的简化程序获得了官方资助结果的80.6％（95％CI 73.9-87.3）同意。使用小组成员的两次审核时，协议达成率为86.6％（95％CI 80.8-92.4）。对于小组讨论的一组申请，这两种方法之间的协议较低（分别为64.2％和73.1％），而对于直接资助/拒绝的申请则较高（范围为96.7％至100％）。134个申请（官方方法）中有8个（6.0％）使用了彩票，19个（14.2％）申请（简化了，三个审阅者）和23个（17.2％）申请（简化了，两个审阅者）中使用了彩票。通过简化的流程，本来可以为2019年的两次电话会议节省一半的评估成本并节省31个小时的会议时间。两种方法之间的一致性很高。