Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.

观察被广泛用于研究和评估以表征教学和学习活动。因为进行观察通常是资源密集型的，所以从观察数据中可靠地做出推断是很重要的。虽然注意力集中在评价者间的可靠性上，但一个学期内单一类别测量的可靠性受到的关注较少。我们研究了观察在评估教学实践中的使用和局限性，以及在典型课程中需要多少观察才能对教学实践做出自信的推断。我们基于泛化理论进行了两项研究，以计算给定一个学期教学中班级间差异的可靠性。需要对一个学期内的课堂时间进行 11 次观察才能获得可靠的测量结果，远远超过文献中通常观察到的一到四节课。调查结果表明，从业者可能需要投入比预期更多的资源来实现可靠的测量和比较。