Multi-turn response selection is a major task in building intelligent dialogue systems. Most existing works focus on modeling the semantic relationship between the utterances and the candidate response with neural networks like RNNs and various attention mechanisms. In this paper, we study how to leverage the advantage of pre-trained language models (PTMs) to multi-turn response selection in retrieval-based chatbots. We propose a deep context modeling architecture (DCM) for multi-turn response selection by utilizing BERT as the context encoder. DCM is formulated as a four-module architecture, namely contextual encoder, utterance-to-response interaction, features aggregation, and response selection. Moreover, in DCM, we introduce the next utterance prediction as a pre-training scheme based on BERT, aiming to adapt general BERT to accommodate the inherent context continuity underlying the multi-turn dialogue. Taking BERT as the backbone encoder, we then investigate a variety of strategies to perform response selection with comprehensive comparisons. Empirical results on three public datasets from two different languages show that our proposed model outperforms existing promising models significantly, pushing recall to 86.8% (+5.2% improvement over BERT) on Ubuntu Dialogue corpus, recall to 68.5% (+6.4% improvement over BERT) on E-Commerce Dialogue corpus, MAP and MRR to 61.6% and 64.9% respectively (+2.3% and 1.8% improvement over BERT) on Douban Conversation corpus, achieving new state-of-the-art performance for multi-turn response selection.

多回合响应选择是构建智能对话系统的主要任务。现有的大多数工作都着重于利用神经网络（如RNN）和各种注意机制对发声与候选响应之间的语义关系进行建模。在本文中，我们研究了如何在基于检索的聊天机器人中利用预训练语言模型（PTM）的优势进行多回合响应选择。我们提出了一个d EEP Ç ontext米利用BERT作为上下文编码器odeling架构（DCM）对多圈响应选择。DCM被公式化为四个模块的体系结构，即上下文编码器，话语-响应交互，功能聚合和响应选择。此外，在DCM中，我们引入了下一个语音预测作为基于BERT的预训练方案，旨在适应通用BERT以适应多回合对话背后的固有上下文连续性。以BERT为骨干编码器，然后我们通过综合比较研究了各种策略来执行响应选择。来自两种不同语言的三个公开数据集的经验结果表明，我们提出的模型显着优于现有的有前途的模型，从而使Ubuntu Dialogue语料库的召回率达到86.8％（比BERT提高+ 5.2％），召回率达到68.5％（比BERT提高+ 6.4％） ）在电子商务对话语料库上，豆瓣对话语料库的MAP和MRR分别达到61.6％和64.9％（比BERT分别提高了2.3％和1.8％），