Background

Funding agencies have long used panel discussion in the peer review of research grant proposals as a way to utilize a set of expertise and perspectives in making funding decisions. Little research has examined the quality of panel discussions and how effectively they are facilitated.

Methods

Here, we present a mixed-method analysis of data from a survey of reviewers focused on their perceptions of the quality, effectiveness, and influence of panel discussion from their last peer review experience.

Results

Reviewers indicated that panel discussions were viewed favorably in terms of participation, clarifying differing opinions, informing unassigned reviewers, and chair facilitation. However, some reviewers mentioned issues with panel discussions, including an uneven focus, limited participation from unassigned reviewers, and short discussion times. Most reviewers felt the discussions affected the review outcome, helped in choosing the best science, and were generally fair and balanced. However, those who felt the discussion did not affect the outcome were also more likely to evaluate panel communication negatively, and several reviewers mentioned potential sources of bias related to the discussion. While respondents strongly acknowledged the importance of the chair in ensuring appropriate facilitation of the discussion to influence scoring and to limit the influence of potential sources of bias from the discussion on scoring, nearly a third of respondents did not find the chair of their most recent panel to have performed these roles effectively.

Conclusions

It is likely that improving chair training in the management of discussion as well as creating review procedures that are informed by the science of leadership and team communication would improve review processes and proposal review reliability.

中文翻译：

---

授予审稿人对小组讨论的质量、有效性和影响的看法

背景

资助机构长期以来一直在研究资助提案的同行评审中使用小组讨论，作为在做出资助决策时利用一系列专业知识和观点的一种方式。很少有研究检查小组讨论的质量以及如何有效地促进它们。

方法

在这里，我们对评审员的调查数据进行了混合方法分析，重点关注他们对上次同行评审经验中小组讨论的质量、有效性和影响的看法。

结果

审稿人表示，小组讨论在参与、澄清不同意见、通知未指派的审稿人和主持会议方面受到好评。然而，一些审稿人提到了小组讨论的问题，包括焦点不均、未指定审稿人的参与有限以及讨论时间短。大多数审稿人认为讨论影响了审稿结果，有助于选择最好的科学，并且总体上是公平和平衡的。然而，那些认为讨论不影响结果的人也更有可能对小组交流进行负面评价，一些评论者提到了与讨论相关的潜在偏见来源。

结论

改进讨论管理方面的主席培训以及创建以领导科学和团队沟通为依据的审查程序很可能会改善审查流程和提案审查的可靠性。