Recent research analyzing the sensitivity of natural language understanding models to word-order perturbations have shown that the state-of-the-art models in several language tasks may have a unique way to understand the text that could seldom be explained with conventional syntax and semantics. In this paper, we investigate the insensitivity of natural language models to word-order by quantifying perturbations and analysing their effect on neural models' performance on language understanding tasks in GLUE benchmark. Towards that end, we propose two metrics - the Direct Neighbour Displacement (DND) and the Index Displacement Count (IDC) - that score the local and global ordering of tokens in the perturbed texts and observe that perturbation functions found in prior literature affect only the global ordering while the local ordering remains relatively unperturbed. We propose perturbations at the granularity of sub-words and characters to study the correlation between DND, IDC and the performance of neural language models on natural language tasks. We find that neural language models - pretrained and non-pretrained Transformers, LSTMs, and Convolutional architectures - require local ordering more so than the global ordering of tokens. The proposed metrics and the suite of perturbations allow a systematic way to study the (in)sensitivity of neural language understanding models to varying degree of perturbations.

中文翻译：

---

揭开神经语言模型对词序不敏感的神秘面纱

最近分析自然语言理解模型对词序扰动的敏感性的研究表明，几种语言任务中的最新模型可能有一种独特的方式来理解文本，而这种方式很少用传统的句法和语义来解释. 在本文中，我们通过量化扰动并分析扰动对神经模型在 GLUE 基准测试中的语言理解任务性能的影响来研究自然语言模型对词序的不敏感性。为此，我们提出了两个指标——直接邻居位移（DND）和索引位移计数（IDC）——它们对扰动文本中标记的局部和全局排序进行评分，并观察到先前文献中发现的扰动函数仅影响全局排序，而本地排序仍然相对不受干扰。我们建议在子词和字符的粒度上进行扰动，以研究 DND、IDC 与神经语言模型在自然语言任务上的性能之间的相关性。我们发现神经语言模型——预训练和非预训练的 Transformer、LSTM 和卷积架构——比令牌的全局排序更需要局部排序。