The consensual assessment technique (CAT) is a reliable and valid method to measure (product) creativity and often considered the gold standard of creativity assessment. The reliability measure traditionally applied in CAT studies—inter-rater reliability—cannot capture time-sampling error, which is a particular relevant source of error for specific applications of the CAT. Therefore, the present study intended to investigate the test–retest reliability of CAT ratings. We asked raters (N = 61) for their creativity assessment of the same set of 90 fashion outfits at an initial rating session and a follow-up session either 2 or 4 weeks later. We found that mean product ratings—the actual focus of interest in the CAT—were highly stable over time, as evidenced by consistency and agreement ICCs clearly exceeding levels of .90. However, individual raters (partially) lacked temporal stability, indicating a drift in rater tendencies over time. Our findings support the CAT’s reputation as a highly reliable measurement method, but question the temporal rating stability of the CAT’s actual “measurement instrument,” namely individual judges.

中文翻译：

---

随时间推移的创造力评估：检查 CAT 评级的可靠性

在协商一致的评估技术（CAT）是衡量（产品）创新可靠和有效的方法，通常被认为的创造力评估的金标准。CAT 研究中传统上应用的可靠性度量——评分者间的可靠性——无法捕获时间采样误差，这是 CAT 特定应用的特定相关误差源。因此，本研究旨在调查 CAT 评分的重测信度。我们询问了评分者（N = 61），因为他们在初始评级会议和 2 或 4 周后的后续会议中对同一组 90 套时装的创造力评估。我们发现平均产品评级——CAT 中实际关注的焦点——随着时间的推移高度稳定，一致性和一致性 ICC 明显超过 0.90 的水平就证明了这一点。然而，个体评分者（部分）缺乏时间稳定性，表明评分者倾向随时间推移而发生漂移。我们的研究结果支持 CAT 作为高度可靠的衡量方法的声誉，但质疑 CAT 实际“衡量工具”（即个别法官）的时间评级稳定性。