We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. Such evaluations are of interest when raters rate numerous comparable items on the same scale, as occurs in hiring, college admissions, and peer review. Our exposition takes the context of peer review, which involves univariate and multivariate cardinal ratings. We draw on this context to motivate an information-theoretic measure of the refinement of a set of ratings – entropic refinement – as well as two secondary measures. A mathematical analysis of the three measures reveals that only the first, which captures the information content of the ratings, possesses properties appropriate to a refinement metric. Finally, we analyse refinement in real-world grant-review data, finding evidence that overall merit scores are more refined than criterion scores.

中文翻译：

---

改进：在没有黄金标准的情况下衡量评级的信息量

我们提出了一个新的指标，用于评估给定范围内单个评分者的一组评分的信息量。当评分者以相同的尺度对许多可比较的项目进行评分时，这种评估很有意义，如在招聘、大学录取和同行评审中发生的那样。我们的阐述以同行评审为背景，其中涉及单变量和多变量基数评级。我们利用这种背景来激发改进的信息理论测量一组评级——熵细化——以及两个次要措施。对这三个度量的数学分析表明，只有第一个捕获评级的信息内容，才具有适合细化度量的属性。最后，我们分析了真实世界的资助审查数据中的细化，发现了整体优点分数比标准分数更细化的证据。