ABSTRACT

Human raters and machine scoring systems potentially have complementary strengths in evaluating language ability; specifically, it has been suggested that automated systems might be used to make consistent measurements of specific linguistic phenomena, whilst humans evaluate more global aspects of performance. We report on an empirical study that explored the possibility of combining human and machine scores using responses from the speaking section of the TOEFL iBT® test. Human raters awarded scores for three sub-constructs: delivery, language use and topic development. The SpeechRater^SM automated scoring system produced scores for delivery and language use. Composite scores computed from three different combinations of human and automated analytic scores were equally or more reliable than human holistic scores, probably due to the inclusion of multiple observations in composite scores. However, composite scores calculated solely from human analytic scores showed the highest reliability and reliability steadily decreased as more machine scores replaced human scores.

摘要

人工评分者和机器评分系统在评估语言能力方面可能具有互补优势；具体来说，有人建议可以使用自动化系统对特定语言现象进行一致的测量，而人类则可以评估更全面的表现方面。我们报告了一项实证研究，该研究探索了使用 TOEFL iBT ®考试口语部分的回答将人类和机器分数相结合的可能性。人工评分员为三个子结构评分：交付、语言使用和主题发展。SpeechRater ^SM自动评分系统为交付和语言使用生成分数。从人类和自动分析分数的三种不同组合计算得出的综合分数与人类整体分数相同或更可靠，这可能是由于综合分数中包含了多项观察结果。然而，仅根据人类分析分数计算的综合分数显示出最高的可靠性，并且随着更多机器分数取代人工分数，可靠性稳步下降。