Linked text representation is critical for many intelligent web applications, such as online advertisement and recommender systems. Recent breakthroughs on pretrained language models and graph neural networks facilitate the development of corresponding techniques. However, the existing works mainly rely on cascaded model structures: the texts are independently encoded by language models at first, and the textual embeddings are further aggregated by graph neural networks. We argue that the neighbourhood information is insufficiently utilized within the above process, which restricts the representation quality. In this work, we propose GraphFormers, where graph neural networks are nested alongside each transformer layer of the language models. On top of the above architecture, the linked texts will iteratively extract neighbourhood information for the enhancement of their own semantics. Such an iterative workflow gives rise to more effective utilization of neighbourhood information, which contributes to the representation quality. We further introduce an adaptation called unidirectional GraphFormers, which is much more efficient and comparably effective; and we leverage a pretraining strategy called the neighbourhood-aware masked language modeling to enhance the training effect. We perform extensive experiment studies with three large-scale linked text datasets, whose results verify the effectiveness of our proposed methods.

中文翻译：

---

GraphFormers：用于链接文本表示的GNN嵌套语言模型

链接文本表示对于许多智能Web应用程序（例如在线广告和推荐系统）至关重要。预训练语言模型和图神经网络的最新突破促进了相应技术的发展。但是，现有的工作主要依靠级联模型结构：首先，文本是由语言模型独立编码的，而文本嵌入则是由图神经网络进一步聚合的。我们认为邻域信息在上述过程中没有得到充分利用，这限制了表示质量。在这项工作中，我们提出了GraphFormers，其中图神经网络嵌套在语言模型的每个转换器层旁边。在上述架构之上，链接的文本将迭代提取邻域信息，以增强其自身的语义。这样的迭代工作流引起对邻域信息的更有效利用，这有助于表示质量。我们进一步介绍了一种称为单向GraphFormers的改编，它效率更高，效率更高。我们利用一种称为邻域感知的屏蔽语言建模的预培训策略来增强培训效果。我们对三个大型链接文本数据集进行了广泛的实验研究，其结果证明了我们提出的方法的有效性。我们进一步介绍了一种称为单向GraphFormers的改编，它效率更高，效率更高。我们利用一种称为邻域感知的屏蔽语言建模的预培训策略来增强培训效果。我们对三个大型链接文本数据集进行了广泛的实验研究，其结果证明了我们提出的方法的有效性。我们进一步介绍了一种称为单向GraphFormers的改编，它效率更高，效率更高。我们利用一种称为邻域感知的屏蔽语言建模的预培训策略来增强培训效果。我们对三个大型链接文本数据集进行了广泛的实验研究，其结果验证了我们提出的方法的有效性。