Autograding systems are being increasingly deployed to meet the challenges of teaching programming at scale. Studies show that formative feedback can greatly help novices learn programming. This work extends an autograder, enabling it to provide formative feedback on programming assignment submissions. Our methodology starts with the design of a knowledge map, which is the set of concepts and skills that are necessary to complete an assignment, followed by the design of the assignment and that of a comprehensive test suite for identifying logical errors in the submitted code. Test cases are used to test the student submissions and learn classes of common errors. For each assignment, we train a classifier that automatically categorizes errors in a submission based on the outcome of the test suite. The instructor maps the errors to corresponding concepts and skills and writes hints to help students find their misconceptions and mistakes. We apply this methodology to two assignments in our Introduction to Computer Science course and find that the automatic error categorization has a 90% average accuracy. We report and compare data from two semesters, one semester when hints are given for the two assignments and one when hints are not given. Results show that the percentage of students who successfully complete the assignments after an initial erroneous submission is three times greater when hints are given compared to when hints are not given. However, on average, even when hints are provided, almost half of the students fail to correct their code so that it passes all the test cases. The initial implementation of the framework focuses on the functional correctness of the programs as reflected by the outcome of the test cases. In our future work, we will explore other kinds of feedback and approaches to automatically generate feedback to better serve the educational needs of the students.

中文翻译：

---

CSF：自动评分中的形成性反馈

越来越多地部署自动评分系统来应对大规模编程教学的挑战。研究表明，形成性反馈可以极大地帮助新手学习编程。这项工作扩展了自动评分器，使其能够为提交的编程作业提供形成性反馈。我们的方法从设计知识图开始，这是完成作业所必需的一组概念和技能，然后是作业的设计和用于识别提交代码中的逻辑错误的综合测试套件的设计。测试用例用于测试学生提交的内容并学习常见错误的类别。对于每个作业，我们训练一个分类器，该分类器根据测试套件的结果自动对提交中的错误进行分类。教师将错误映射到相应的概念和技能，并编写提示以帮助学生找到他们的误解和错误。我们将此方法应用于计算机科学导论课程中的两个作业，发现自动错误分类的平均准确率为 90%。我们报告并比较了两个学期的数据，一个学期对两项作业给出提示，一个学期没有给出提示。结果表明，在初始错误提交后成功完成作业的学生百分比在给出提示时是未给出提示时的三倍。然而，平均而言，即使提供了提示，几乎一半的学生也未能纠正他们的代码以使其通过所有测试用例。框架的初始实现侧重于测试用例结果所反映的程序的功能正确性。在我们未来的工作中，我们将探索其他类型的反馈和自动生成反馈的方法，以更好地满足学生的教育需求。