Researchers and practitioners sometimes presume that using a previously “validated” instrument will produce “valid” scores; however, contemporary views of validity suggest that there are many reasons this assumption can be faulty. In order to demonstrate just some of the problems with this view, and to support comparisons of different observation protocols across contexts, we introduce and define the conceptual tool of an observation system. We then describe psychometric evidence of a popular teacher observation instrument, Charlotte Danielson’s Framework for Teaching, in three use contexts—a lower-stakes research context, a lower-stakes practice-based context, and a higher-stakes practice-based context. Despite sharing a common instrument, we find the three observation systems and their associated use contexts combine to produce different average teacher scores, variation in score distributions, and different levels of precision in scores. However, all three systems produce higher average scores in the classroom environment domain than the instructional domain and all three sets of scores support a one-factor model, whereas the Framework posits four factors. We discuss how the dependencies between aspects of observation systems and practical constraints leave researchers with significant validation challenges and opportunities.

中文翻译：

---

情境中的课堂观察系统：观察系统验证的案例

研究人员和从业者有时会假设使用先前“验证过的”工具会产生“有效”的分数；然而，当代关于有效性的观点表明，这种假设有很多错误的原因。为了仅证明这种观点的一些问题，并支持跨上下文比较不同观察协议，我们引入并定义了观察系统的概念工具。然后，我们描述了一种流行的教师观察工具 Charlotte Danielson 的教学框架在三个使用环境中的心理测量证据——低风险的研究环境、低风险的基于实践的环境和高风险的基于实践的环境。尽管拥有共同的乐器，我们发现三个观察系统及其相关的使用上下文结合起来产生不同的平均教师分数、分数分布的变化以及分数的不同精度水平。然而，所有三个系统在课堂环境领域产生的平均分数都高于教学领域，并且所有三组分数都支持单因素模型，而框架假设有四个因素。我们讨论了观测系统各方面与实际约束之间的依赖关系如何使研究人员面临重大的验证挑战和机遇。所有三个系统在课堂环境领域产生的平均分数都高于教学领域，并且所有三组分数都支持单因素模型，而框架假设有四个因素。我们讨论了观测系统各方面与实际约束之间的依赖关系如何使研究人员面临重大的验证挑战和机遇。所有三个系统在课堂环境领域产生的平均分数都高于教学领域，并且所有三组分数都支持单因素模型，而框架假设有四个因素。我们讨论了观测系统各方面与实际约束之间的依赖关系如何使研究人员面临重大的验证挑战和机遇。