Health service psychology (HSP) graduate programs are shifting from knowledge- to competency-based assessments of trainees' psychotherapy skills. This study used Generalizability Theory to test the dependability of psychotherapy competence assessments based on video observation of trainees. A 10-item rating form was developed from a collection of forms used by graduate programs (n = 102) in counseling and clinical psychology, and a review of the common factors research literature. This form was then used by 11 licensed psychologists to rate eight graduate trainees while viewing 129, approximately 5-min video clips from their psychotherapy sessions with clients (n = 22) at a graduate program's training clinic. Generalizability analyses were used to forecast how the number of raters and clients, and length of observation time impact the dependability of ratings in various rating designs. Raters were the primary source of error variance in ratings, with rater main effects (leniency bias) and dyadic effects (rater-target interactions) contributing 24% and 7% of variance, respectively. Variance due to segments (video clips) was also substantial, suggesting that therapist performance varies within the same counseling session. Generalizability coefficients (G) were highest for crossed rating designs and reached maximum levels (G > .50) after four raters watched each therapist working with three clients and observed 15 min per dyad. These findings suggest that expert raters show consensus in ratings even without rater training and only limited direct observation. Future research should investigate the validity of competence ratings as predictors of outcome. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

中文翻译：

---

评估心理治疗师能力：测试研究生实习生临床能力评估的普遍性。

健康服务心理学（HSP）研究生课程正在从对学员心理治疗技能的基于知识的评估转向基于能力的评估。本研究使用普遍性理论来测试基于受训者视频观察的心理治疗能力评估的可靠性。包含 10 项的评级表是根据咨询和临床心理学研究生项目 (n = 102) 使用的一系列表格以及对常见因素研究文献的回顾而制定的。然后，11 名持有执照的心理学家使用该表格对 8 名研究生学员进行评分，同时观看他们在研究生项目培训诊所与客户 (n = 22) 进行心理治疗的 129 个大约 5 分钟的视频剪辑。普遍性分析用于预测评级者和客户的数量以及观察时间的长度如何影响各种评级设计中评级的可靠性。评分者是评分误差方差的主要来源，评分者主效应（宽大偏差）和二元效应（评分者-目标相互作用）分别贡献了 24% 和 7% 的方差。由于片段（视频剪辑）而产生的差异也很大，这表明治疗师的表现在同一咨询疗程中有所不同。交叉评级设计的普遍性系数 (G) 最高，并在四位评级者观看每位治疗师与三位客户的工作并每对观察 15 分钟后达到最高水平 (G > .50)。这些研究结果表明，即使没有对评估员进行培训并且仅进行有限的直接观察，专家评估员也会在评级方面表现出共识。未来的研究应该调查能力评级作为结果预测因素的有效性。（PsycInfo 数据库记录 (c) 2022 APA，保留所有权利）。