A significant progress has been made in deep-learning based Automatic Essay Scoring (AES) systems in the past two decades. The performance commonly measured by the standard performance metrics like Quadratic Weighted Kappa (QWK), and accuracy points to the same. However, testing on common-sense adversarial examples of these AES systems reveal their lack of natural language understanding capability. Inspired by common student behaviour during examinations, we propose a task agnostic adversarial evaluation scheme for AES systems to test their natural language understanding capabilities and overall robustness.

中文翻译：

---

呼唤虚张声势：用简单的对抗性测试攻击自动评分系统的稳健性

在过去的二十年中，基于深度学习的自动论文评分 (AES) 系统取得了重大进展。通常由标准性能指标（如二次加权 Kappa (QWK)）衡量的性能和准确性指向相同。然而，对这些 AES 系统的常识对抗样本的测试揭示了它们缺乏自然语言理解能力。受考试期间常见学生行为的启发，我们为 AES 系统提出了一种与任务无关的对抗性评估方案，以测试其自然语言理解能力和整体稳健性。