With the proliferation of large-scale knowledge graphs (KGs), multi-hop knowledge graph reasoning has been a capstone that enables machines to be able to handle intelligent tasks, especially where some explicit reasoning path is appreciated for decision making. To train a KG reasoner, supervised learning-based methods suffer from false-negative issues, i.e., unseen paths during training are not to be found in prediction; in contrast, reinforcement learning (RL)-based methods do not require labeled paths, and can explore to cover many appropriate reasoning paths. In this connection, efforts have been dedicated to investigating several RL formulations for multi-hop KG reasoning. Particularly, current RL-based methods generate rewards at the very end of the reasoning process, due to which short paths of hops less than a given threshold are likely to be overlooked, and the overall performance is impaired. To address the problem, we propose RL-MHR, a revised RL formulation of multi-hop KG reasoning that is characterized by two novel designs—the stop signal and the worth-trying signal. The stop signal instructs the agent of RL to stay at the entity after finding the answer, preventing from hopping further even if the threshold is not reached; meanwhile, the worth-trying signal encourages the agent to try to learn some partial patterns from the paths that fail to lead to the answer. To validate the design of our model RL-MHR, comprehensive experiments are carried out on three benchmark knowledge graphs, and the results and analysis suggest the superiority of RL-MHR over state-of-the-art methods.

随着大规模知识图 (KG) 的激增，多跳知识图推理已成为使机器能够处理智能任务的顶峰，尤其是在某些明确的推理路径可用于决策的情况下。为了训练 KG 推理器，基于监督学习的方法存在假阴性问题，即在预测中找不到训练期间看不见的路径；相比之下，基于强化学习 (RL) 的方法不需要标记路径，并且可以探索覆盖许多合适的推理路径。在这方面，一直致力于研究用于多跳 KG 推理的几种 RL 公式。特别是，当前基于 RL 的方法会在推理过程的最后产生奖励，因此，小于给定阈值的短跳路径可能会被忽略，从而损害整体性能。为了解决这个问题，我们建议RL-MHR是多跳 KG 推理的修订 RL 公式，其特点是两种新颖的设计——停止信号和值得尝试的信号。停止信号指示 RL 的代理在找到答案后留在实体，即使没有达到阈值也防止进一步跳跃；同时，值得尝试的信号鼓励代理尝试从无法通向答案的路径中学习一些部分模式。为了验证我们模型RL-MHR的设计，在三个基准知识图谱上进行了综合实验，结果和分析表明RL-MHR优于最先进的方法。