Motivation Biomedical language models produce meaningful concept representations that are useful for a variety of biomedical natural language processing (bioNLP) applications such as named entity recognition, relationship extraction and question answering. Recent research trends have shown that the contextualized language models (e.g. BioBERT, BioELMo) possess tremendous representational power and are able to achieve impressive accuracy gains. However, these models are still unable to learn high-quality representations for concepts with low context information (i.e. rare words). Infusing the complementary information from knowledge-bases (KBs) is likely to be helpful when the corpus-specific information is insufficient to learn robust representations. Moreover, as the biomedical domain contains numerous KBs, it is imperative to develop approaches that can integrate the KBs in a continual fashion. Results We propose a new representation learning approach that progressively fuses the semantic information from multiple KBs into the pretrained biomedical language models. Since most of the KBs in the biomedical domain are expressed as parent-child hierarchies, we choose to model the hierarchical KBs and propose a new knowledge modeling strategy that encodes their topological properties at a granular level. Moreover, the proposed continual learning technique efficiently updates the concepts representations to accommodate the new knowledge while preserving the memory efficiency of contextualized language models. Altogether, the proposed approach generates knowledge-powered embeddings with high fidelity and learning efficiency. Extensive experiments conducted on bioNLP tasks validate the efficacy of the proposed approach and demonstrates its capability in generating robust concept representations.

中文翻译：

---

将知识持续注入预训练的生物医学语言模型

动机 生物医学语言模型产生有意义的概念表示，可用于各种生物医学自然语言处理 (bioNLP) 应用程序，例如命名实体识别、关系提取和问答。最近的研究趋势表明，语境化语言模型（例如 BioBERT、BioELMo）具有巨大的表征能力，并且能够实现令人印象深刻的准确性提升。然而，这些模型仍然无法为具有低上下文信息（即罕见词）的概念学习高质量的表示。当特定于语料库的信息不足以学习稳健的表示时，从知识库 (KB) 中注入补充信息可能会有所帮助。此外，由于生物医学领域包含大量知识库，开发能够以持续方式整合知识库的方法势在必行。结果我们提出了一种新的表示学习方法，该方法逐步将来自多个知识库的语义信息融合到预训练的生物医学语言模型中。由于生物医学领域中的大多数知识库都表示为父子层次结构，我们选择对层次结构的知识库进行建模，并提出一种新的知识建模策略，在粒度级别对其拓扑属性进行编码。此外，所提出的持续学习技术有效地更新概念表示以适应新知识，同时保持上下文语言模型的记忆效率。总而言之，所提出的方法生成了具有高保真度和学习效率的知识驱动嵌入。