Transformers are the current state-of-the-art of natural language processing in many domains and are using traction within software engineering research as well. Such models are pre-trained on large amounts of data, usually from the general domain. However, we only have a limited understanding regarding the validity of transformers within the software engineering domain, i.e., how good such models are at understanding words and sentences within a software engineering context and how this improves the state-of-the-art. Within this article, we shed light on this complex, but crucial issue. We compare BERT transformer models trained with software engineering data with transformers based on general domain data in multiple dimensions: their vocabulary, their ability to understand which words are missing, and their performance in classification tasks. Our results show that for tasks that require understanding of the software engineering context, pre-training with software engineering data is valuable, while general domain models are sufficient for general language understanding, also within the software engineering domain.

中文翻译：

---

关于软件工程领域自然语言处理的预训练转换器的有效性

Transformer 是许多领域中自然语言处理的最新技术，并且也在软件工程研究中发挥着重要作用。这些模型在大量数据上进行了预训练，这些数据通常来自一般领域。然而，我们对软件工程领域内转换器的有效性了解有限，即这些模型在软件工程环境中理解单词和句子方面有多好，以及这如何改进最先进的技术。在本文中，我们阐明了这个复杂但至关重要的问题。我们将使用软件工程数据训练的 BERT 转换器模型与基于通用领域数据的转换器在多个维度进行比较：它们的词汇、它们理解哪些单词缺失的能力，以及它们在分类任务中的表现。