The methodological limitations education researchers face in the evaluation of reformed instruction have led to debates as to the evidence advancing evidence-based practices. To conduct more effective research, methodological pluralism in the evaluation of educational reforms can be used to complement the strengths and limitations of a corpus of literature informing the impact of an evidence-based practice. This study seeks to introduce randomization tests, a nonparametric statistical analysis incorporating a random-assignment component that can be applied to a single-subject (N = 1) research design, as a methodology to be counted amongst evaluations of instructional reforms. To demonstrate the utility of this approach, an evaluation of peer-led team learning (PLTL) for classes of second-semester general chemistry spanning 7 semesters was conducted using randomization tests. The design contributes novel understandings of PLTL including differences in effectiveness across instructors, trends in effectiveness over time, and a perspective as to the appropriateness of assumptions concerning statistical independence when applied to educational settings. At the research setting, four instructors (each constituting an individual case) alternated implementing lecture-based instruction and PLTL by term. Across these four instructors, the treatment effects of peer-led team learning when compared to lecture-based instruction ranged in impact (from d = 0.233 to 2.09). For two instructors, PLTL provided a means by which to significantly reduce the differential performances observed of students with variable preparations in mathematics, thereby advancing the equitability of their courses. Implications of this work include the incorporation of single-subject research designs in establishing evidence-based instructional practices, the effectiveness of PLTL as interpreted in a methodologically pluralistic context of the research literature, and enacting measurements of equity when gauging the success of instructional reforms in science. Further, this introduction to randomization tests offers another methodology for the evaluation of instructional reforms more widely applicable in educational settings with smaller sample sizes (e.g., reforms conducted within a single classroom or upper-level courses with small class sizes).

中文翻译：

---

通过对本科生化学课程中的同伴主导的团队学习进行评估，引入随机化测试

教育研究人员在评估改革后的教学法时面临的方法学局限性引发了关于证据推进循证实践的争论。为了进行更有效的研究，可以在教育改革的评估中采用方法论上的多元性，以补充文学作品集的优势和局限性，从而告知循证实践的影响。这项研究旨在介绍随机化测试，这是一种非参数统计分析，该分析结合了可以应用于单个对象（N= 1）研究设计，作为在教学改革评估中应计入的方法。为了证明这种方法的实用性，我们使用随机测试对跨7个学期的第二学期普通化学班进行了同行领导的团队学习（PLTL）评估。该设计有助于对PLTL进行新颖的理解，包括不同教员的有效性差异，有效性随时间的变化趋势以及对应用于教育环境时有关统计独立性的假设是否适当的看法。在研究环境中，四位讲师（每人组成一个案例）交替实施基于讲义的讲授和PLTL。在这四位教练中，d = 0.233至2.09）。对于两名教师，PLTL提供了一种方法，通过该方法可以显着降低在数学上有不同准备的学生所观察到的差异表现，从而提高课程的公平性。这项工作的含义包括在建立基于证据的教学实践中纳入单科目的研究设计，在研究文献的方法论多元化背景下解释的PLTL的有效性，以及在衡量教学改革是否成功时制定公平性的衡量标准。科学。此外，这种对随机化测试的介绍为评估教学改革提供了另一种方法，该方法更适用于样本量较小的教育环境（例如，，在单一教室或小班制高级课程中进行的改革）。