In recent years, the performance of deep neural network in extractive summarization task has been improved significantly compared with traditional methods. However, in the field of biomedical extractive summarization, existing methods can not make good use of the domain-aware external knowledge; furthermore, the document structural feature is omitted by existing deep neural network model. In this paper, we propose a novel model called BioBERTSum to better capture token-level and sentence-level contextual representation, which uses a domain-aware bidirectional language model pre-trained on large-scale biomedical corpora as encoder, and further fine-tunes the language model for extractive text summarization task on single biomedical document. Especially, we adopt a sentence position embedding mechanism, which enables the model to learn the position information of sentences and achieve the structural feature of document. To the best of our knowledge, this is the first work to use the pre-trained language model and fine-tuning strategy for extractive summarization task in the biomedical domain. Experiments on PubMed dataset show that our proposed model outperforms the recent SOTA (state-of-the-art) model by ROUGE-1/2/L.

近年来，与传统方法相比，深度神经网络在提取摘要任务中的性能已得到显着改善。但是，在生物医学提取总结领域，现有方法不能充分利用领域感知的外部知识。此外，现有的深度神经网络模型省略了文档的结构特征。在本文中，我们提出了一种新的名为BioBERTSum的模型，以更好地捕获令牌级和句子级的上下文表示，它使用在大型生物医学语料库上预训练的领域感知双向语言模型作为编码器，并进行进一步的微调。用于单个生物医学文档的提取文本摘要任务的语言模型。特别是，我们采用了句子位置嵌入机制，这使模型能够学习句子的位置信息并实现文档的结构特征。据我们所知，这是将预训练语言模型和微调策略用于生物医学领域提取摘要任务的第一项工作。在PubMed数据集上进行的实验表明，我们提出的模型的性能优于ROUGE-1 / 2 / L的最新SOTA（最新技术）模型。