Dialogue policy training for composite tasks, such as restaurant reservation in multiple places, is a practically important and challenging problem. Recently, hierarchical deep reinforcement learning (HDRL) methods have achieved good performance in composite tasks. However, in vanilla HDRL, both top-level and low-level policies are all represented by multi-layer perceptrons (MLPs) which take the concatenation of all observations from the environment as the input for predicting actions. Thus, traditional HDRL approach often suffers from low sampling efficiency and poor transferability. In this paper, we address these problems by utilizing the flexibility of graph neural networks (GNNs). A novel ComNet is proposed to model the structure of a hierarchical agent. The performance of ComNet is tested on composited tasks of the PyDial benchmark. Experiments show that ComNet outperforms vanilla HDRL systems with performance close to the upper bound. It not only achieves sample efficiency but also is more robust to noise while maintaining the transferability to other composite tasks.

中文翻译：

---

使用图神经网络的结构化分层对话策略

复合任务的对话策略训练，例如多地餐厅预订，是一个实际重要且具有挑战性的问题。最近，分层深度强化学习（HDRL）方法在复合任务中取得了良好的性能。然而，在普通的 HDRL 中，顶级和低级策略都由多层感知器 (MLP) 表示，该多层感知器将环境中所有观察结果的串联作为预测动作的输入。因此，传统的 HDRL 方法通常存在采样效率低和可转移性差的问题。在本文中，我们通过利用图神经网络 (GNN) 的灵活性来解决这些问题。提出了一种新颖的 ComNet 来对分层代理的结构进行建模。ComNet 的性能在 PyDial 基准测试的复合任务上进行了测试。实验表明，ComNet 在性能接近上限的情况下优于普通的 HDRL 系统。它不仅实现了样本效率，而且在保持对其他复合任务的可转移性的同时，对噪声也更加鲁棒。