Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent reviewers is growing at a much slower rate. To curb this trend and reduce the burden on reviewers, several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. Such initiatives have been met with skepticism among authors, who raise the concern about a potential bias in reviewers' recommendations induced by this information. In this work, we investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue, focusing on a population of novice reviewers who constitute a large fraction of the reviewer pool in leading machine learning and computer science conferences. We design and conduct a randomized controlled trial closely replicating the relevant components of the peer-review pipeline with $133$ reviewers (master's, junior PhD students, and recent graduates of top US universities) writing reviews for $19$ papers. The analysis reveals that reviewers indeed become negatively biased when they receive a signal about paper being a resubmission, giving almost 1 point lower overall score on a 10-point Likert item ($\Delta = -0.78, \ 95\% \ \text{CI} = [-1.30, -0.24]$) than reviewers who do not receive such a signal. Looking at specific criteria scores (originality, quality, clarity and significance), we observe that novice reviewers tend to underrate quality the most.

中文翻译：

---

偏见和偏见：新手审稿人对会议同行审稿中重新提交的偏见

现代机器学习和计算机科学会议的提交数量激增，这对同行评审的质量提出了挑战，因为有能力的评审员数量的增长速度要慢得多。为了抑制这种趋势并减轻审稿人的负担，一些会议已经开始鼓励甚至要求作者声明其论文以前的提交历史。这些举措引起了作者的怀疑，他们对此信息引起了审稿人建议中潜在的偏见的担忧。在这项工作中，我们调查审稿人是否表现出偏见，原因是该人先前知道被审稿在类似的地点被拒绝，重点关注在领先的机器学习和计算机科学会议中，占审阅者群很大一部分的新手审阅者。我们设计并进行了一项随机对照试验，由$ 133 $的审稿人（硕士，初级博士学位学生和美国顶尖大学的最近毕业生）紧密复制同业审稿渠道的相关组成部分，撰写$ 19 $的论文审稿。分析显示，当审稿人收到有关重新提交论文的信号时，他们的确会产生负面的偏见，这会使李克特（Likert）项目的10分得分降低近1分（$ \ Delta = -0.78，\ 95 \％\ \ text { CI} = [-1.30，-0.24] $）。查看特定的标准分数（原始性，质量，清晰度和重要性），