Abstract

In this study, we analyze the two-phase bottom-up procedure applied by the Future and Emerging Technologies Program (FET-Open) at the Research Executive Agency (REA) of the European Commission (EC), for the evaluation of highly interdisciplinary, multi-beneficiary research proposals which request funding. In the first phase, remote experts assess the proposals and draft comments addressing the pre-defined (by FET-Open) evaluation criteria. In the second phase, a new set of additional experts (of more general expertise and different from the remote ones), after cross reading the proposals and their remote evaluation reports, they convene in an on-site panel where they discuss the proposals. They complete the evaluation by reinforcing per proposal and per criterion one or another assessment, as assigned remotely during the first phase. We analyze the level of the inter-rater agreement among the remote experts and we identify its relative correlation with the funded proposals resulted after the end of the evaluation. Our study also provides comparative figures of the evolution of the proposals` scores during the two phases of the evaluation process. Finally, by carrying out an appropriate quantitative and qualitative analysis of all scores from the seven past cut-offs, we elaborate on the significant contribution of the panel (the second phase of the evaluation) in identifying and promoting the best proposals for funding.

中文翻译：

---

在同行评审中评估众多，非结构化和高度跨学科的研究建议时得出的小组意见

摘要

在这项研究中，我们分析了欧洲委员会（EC）研究执行机构（REA）的未来和新兴技术计划（FET-Open）所采用的两阶段自下而上的程序，用于评估高度跨学科的，要求资助的多受益人研究计划。在第一阶段，远程专家评估提案和评论草案，以解决预先定义的（通过FET-Open）评估标准。在第二阶段中，在交叉阅读了提案及其远程评估报告后，又有一组新的专家（具有更广泛的专业知识，与远程专家有所不同），然后召集一个现场小组讨论该提案。他们通过加强每个提案和每个准则的一项或另一项评估来完成评估，这是在第一阶段中进行的远程分配。我们分析了远程专家之间的评估者之间协议的水平，并确定了其与评估结束后获得的资助提案的相对相关性。我们的研究还提供了在评估过程的两个阶段中提案分数演变的比较数据。最后，通过对过去七个截止日期的所有分数进行适当的定量和定性分析，我们详细阐述了专家组（评估的第二阶段）在确定和推广最佳资助建议方面的重大贡献。我们的研究还提供了在评估过程的两个阶段中提案分数演变的比较数据。最后，通过对过去七个截止日期的所有分数进行适当的定量和定性分析，我们详细阐述了专家组（评估的第二阶段）在确定和推广最佳资助建议方面的重大贡献。我们的研究还提供了在评估过程的两个阶段中提案分数演变的比较数据。最后，通过对过去七个截止日期的所有分数进行适当的定量和定性分析，我们详细阐述了专家组（评估的第二阶段）在确定和推广最佳资助建议方面的重大贡献。