Creation of task-oriented dialog/virtual agent (VA) capable of managing complex domain-specific user queries pertaining to multiple intents is difficult since the agent must deal with several subtasks simultaneously. Most end-to-end dialogue systems, however, only provide user semantics as inputs from texts into the learning process and neglect other useful user behaviour and information from other modalities such as images. This stresses the benefit of incorporating multi-modal inputs for eliciting user preference in the task. Also, sentiment of the user plays a significant role in achieving maximum user/customer satisfaction during the conversation. Thus, it is also important to incorporate users’ sentiments during policy learning, especially when serving user’s composite goals. For the creation of multi-modal VA aided with sentiment for conversations encompassing multi-intents, this paper introduces a new dataset, named Vis-SentiVA: Visual and Sentiment aided VA created from open-accessed conversational dataset. We present a hierarchical reinforcement learning (HRL) typically options-based VA to learn policies for serving multi-intent dialogues. Multi-modal information (texts and images) extraction to identify user’s preference is incorporated in the learning framework. A combination of task-based and sentiment-based rewards is integrated in the hierarchical value functions for the VA to be user adaptive. Empirically, we show that all these aspects induced together in the learning framework play a vital role in acquiring higher dialogue task success and increased user contentment in the process of creating composite-natured VAs. This is the first effort in integrating sentiment-aware rewards in the multi-modal HRL framework. The paper highlights that it is indeed essential to include other modes of information extraction such as images and behavioural cues of the user such as sentiment to secure greater user contentment. This also helps in improving success of composite-natured VAs serving task-oriented dialogues.

由于代理必须同时处理多个子任务，因此难以创建能够管理与多个意图有关的复杂的特定于域的用户查询的面向任务的对话框/虚拟代理（VA）。然而，大多数端到端对话系统仅提供用户语义作为从文本到学习过程的输入，而忽略了其他有用的用户行为和其他方式（例如图像）的信息。这强调了合并多模式输入以在任务中引起用户偏爱的好处。同样，用户的情感在对话期间实现最大的用户/客户满意度方面也起着重要作用。因此，在策略学习过程中纳入用户情绪也很重要，尤其是在服务于用户的综合目标时。Vis-SentiVA：从开放访问的对话数据集中创建的视觉和情感辅助VA。我们提出一种典型的分层强化学习（HRL）选项基于VA的语言，以学习用于多目标对话的策略。学习框架中集成了用于识别用户偏好的多模式信息（文本和图像）提取。基于任务的奖励和基于情感的奖励的组合被集成到VA的分层值函数中，以使其成为用户自适应的。从经验上讲，我们表明在学习框架中一起诱发的所有这些方面在创建更高级别的对话任务成功和在创建复合性质的VA的过程中增加用户的满意度方面都起着至关重要的作用。这是在多模式HRL框架中整合感知情感的奖励的第一步。该论文强调，包括其他信息提取模式（例如图像和用户行为提示，例如情感）以确保更大的用户满意度确实是必不可少的。