Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others. A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process by increasing the burden on the pool of qualified reviewers which is growing at a much slower rate. In this work, we consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. Specifically, we design a procedure for (i) recruiting reviewers from the population not typically covered by major conferences and (ii) guiding them through the reviewing pipeline. In conjunction with ICML 2020 -- a large, top-tier machine learning conference -- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers. Our experiment reveals that a combination of the recruiting and guiding mechanisms allows for a principled enhancement of the reviewer pool and results in reviews of superior quality compared to the conventional pool of reviews as evaluated by senior members of the program committee (meta-reviewers).

中文翻译：

---

解决大型会议中合格审稿人短缺的新审稿人实验

会议同行评审构成了一个人为讨论的过程，其重要性不能被夸大：它不仅确定了最佳的接受方案，而且最终通过提倡某些想法和限制其他想法影响了整个研究领域的未来。领先的AI会议收到的投稿数量激增，通过增加合格审稿人的负担而使审稿过程的可持续性面临挑战，而这种负担的增长速度要慢得多。在这项工作中，我们考虑了审稿人的招聘问题，重点是大型会议中合格审稿人的匮乏。具体而言，我们设计了一种程序（i）从通常不会在主要会议中涵盖的人群中招募审稿人，以及（ii）引导他们通过审阅渠道。结合大型顶级机器学习会议ICML 2020，我们通过我们的程序招募了一小批审稿人，并将他们的表现与ICML审稿人的总数进行比较。我们的实验表明，招募和指导机制相结合，可以从原则上增强审阅者库，并导致与计划委员会高级成员（元审阅者）所评估的常规审阅者库相比，审稿质量更高。