ABSTRACT

Criterion measures vary greatly in terms of their psychometric quality and ease of use. This paper serves two purposes. First, it provides a general summary of different approaches to criterion measurement in a military context. Second, it provides an extensive review of 16 specific types of criterion measurement methods (e.g., job performance rating scales, self-report questionnaires, job knowledge tests) on nine psychometric and ease-of-use evaluation factors. Eight criterion measurement experts read a summary of extant research and made ratings to evaluate each measurement method on the evaluation factors. Rater intra-class correlations (ICCs) were high, ranging from .75 to .95 across the evaluation dimensions with a median of .91. Data showed a quality-feasibility tradeoff, where criterion data that are easy to obtain often have technical flaws. Recommendations for military services and future directions in criterion measurement (e.g., applications of machine learning) are discussed.

摘要

标准测量在心理测量质量和易用性方面差异很大。本文有两个目的。首先，它概述了军事背景下标准测量的不同方法。其次，它对九个心理测量和易用性评估因素的 16 种特定类型的标准测量方法（例如，工作绩效评级量表、自我报告问卷、工作知识测试）进行了广泛的回顾。八位标准测量专家阅读了现有研究的总结，并对每种测量方法的评价因素进行了评分。评估者的组内相关性 (ICC) 很高，整个评估维度的范围为 0.75 至 0.95，中位数为 0.91。数据显示质量与可行性之间的权衡，其中容易获得的标准数据往往存在技术缺陷。讨论了对军事服务的建议和标准测量的未来方向（例如机器学习的应用）。