We present lessons learned from an ongoing attempt to conceptualize, develop, and refine a way for teachers to gather formative assessment evidence about classroom argumentation as it happens. The system—named DiALoG (Diagnosing Argumentation Levels of Groups)—includes a digital scoring tool that allows teachers to assess oral classroom argumentation across two primary dimensions: one to capture the Intrapersonal, discipline-specific features of scientific arguments, and another to capture the Interpersonal, group regulatory features of argumentation as a dynamic social act. Coupled with the digital assessment are responsive mini-lessons (RMLs), which provide follow-up curriculum for teachers to respond to different levels of classroom argumentation proficiency for each item assessed. We use classroom observations, interviews, and surveys from piloting science teachers in two different states to iteratively refine this multifaceted formative assessment system of oral classroom argumentation. Lessons learned include the realization by pilot teachers that using the DiALoG system fine-tunes their professional vision to notice student practices they had not previously considered, and the accompanying RMLs help fill gaps in their pedagogical content knowledge and repertoire. Furthermore, while the DiALoG system is intended to be a formative assessment, we learned that the mere presence of numerical scores can queue teacher schema for summative assessment. This prompted us to do away with numbers entirely in the latest version of our digital scoring tool. Such lessons learned from teacher experiences with the development of a novel formative assessment system like DiALoG can be instructive to the development of science educational technology more broadly.

我们提供了从正在进行的尝试中汲取的教训，这些尝试正在进行中，以概念化，发展和完善的方式让教师在课堂辩论发生时收集有关课堂辩论的形成性评估证据。该系统名为DiALoG（组的论证诊断水平），包括一个数字评分工具，该工具可让教师评估两个主要方面的口头课堂论证：一个捕捉科学论证的个人内部，学科特有的特征，另一个捕捉捕捉科学论证的个人内在特征。作为动态社会行为的论证的人际，团体监管特征。响应式迷你课程（RML）与数字评估相结合，可为教师提供后续课程，以针对所评估的每个项目对课堂论证水平的不同水平做出响应。我们使用课堂观察，访谈，以及来自两个州的飞行员科学教师的调查，以迭代地完善这种口头课堂论证的多方面形成性评估系统。所学到的经验包括飞行员老师意识到使用DiALoG系统可以对其进行微调。以专业眼光注意到他们以前从未考虑过的学生行为，并且随附的RML帮助填补了他们在教学内容知识和曲目方面的空白。此外，尽管DiALoG系统旨在作为一种形成性评估，但我们了解到，仅存在数字分数可将教师模式排入汇总评估。这促使我们完全删除了最新版本的数字评分工具中的数字。从教师的经验中学到的经验教训，如像DiALoG这样的新型形成性评估系统的发展，可以对更广泛的科学教育技术的发展具有指导意义。