Peer review is one of the most important processes of scientific evaluation. This paper proposes a consistency index to measure the performances of scholar journal reviewers on recognizing papers with higher scientific influences with respect to citations. The index is initially designed based on three consistencies, i.e., expert consistency, editor consistency and academic community consistency, to capture both the knowledge implied in the review data and the bibliometric data. An iterative computation process is then proposed to make the consistency flow among experts based on their historical performances. Further, a quantitative indicator is designed to characterize the ability of an expert of recognizing papers with higher quality during review process. Experiments are conducted to verify the convergence of the iterative computation, and to evaluate the effectiveness of the consistency index. The experimental results show that the performance of the top experts (with regard to consistency index) is always better than the tail experts, and the performance can be maintained over time. We believe that the proposed index can help journal editors get deeper understanding to their reviewers, and provide evidence in the process of reviewer selection.

同行评审是科学评价最重要的过程之一。本文提出了一个一致性指标来衡量学术期刊审稿人在识别引用方面具有较高科学影响力的论文方面的表现。该索引最初是基于三个一致性设计的，即专家一致性、编辑一致性和学术界一致性，以捕获评论数据和文献计量数据中隐含的知识。然后提出了一个迭代计算过程，以根据专家的历史表现在专家之间进行一致性流动。此外，还设计了一个定量指标来表征专家在审查过程中识别高质量论文的能力。进行实验以验证迭代计算的收敛性，并评估一致性指标的有效性。实验结果表明，顶级专家的性能（在一致性指标方面）总是优于尾专家，并且可以随着时间的推移保持性能。我们相信，提议的索引可以帮助期刊编辑更深入地了解他们的审稿人，并在审稿人选择过程中提供证据。