Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta-analytical interrater reliability (IRR) coefficients are underestimates and do not reflect the reliability of interest to most practitioners and researchers-the reliability of an employee's direct supervisor. To establish the reliability of direct supervisor ratings, those making PA ratings must directly supervise employee job performance instead of nonparallel rater designs (e.g., direct supervisor ratings correlated with ratings from a more senior leader). The current meta-analysis identified 22 independent samples that met this more restrictive study inclusion criterion, finding an average observed IRR of .65. We also report reliability estimates for several important contextual moderators, including whether ratings were completed in operational settings (.60) or for research purposes (.67). In sum, we argue that this study's meta-analytical IRR estimates are the best available estimates of direct supervisor reliability and should be used to guide future research and practice. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

中文翻译：

---

直接主管绩效评级的评估者间可靠性的元分析估计：最佳测量设计下的乐观。

绩效评估（PA）用于各种组织目的，对人力资源实践至关重要。尽管如此，目前对 PA 可靠性的估计较低，导致数十年来对在组织环境中使用 PA 的批评。在本文中，我们认为当前的元分析者间可靠性（IRR）系数被低估，并且没有反映大多数从业者和研究人员感兴趣的可靠性——员工直接主管的可靠性。为了建立直接主管评级的可靠性，进行 PA 评级的人员必须直接监督员工的工作绩效，而不是非平行的评级者设计（例如，直接主管评级与更高级领导的评级相关）。当前的荟萃分析确定了 22 个独立样本符合这一更严格的研究纳入标准，发现平均观察到的 IRR 为 0.65。我们还报告了几个重要上下文调节器的可靠性估计，包括评级是在操作环境中完成的 (.60) 还是出于研究目的 (.67)。总之，我们认为本研究的元分析 IRR 估计是直接主管可靠性的最佳可用估计，应该用于指导未来的研究和实践。（PsycInfo 数据库记录 (c) 2023 APA，保留所有权利）。