Reinforcement learning (RL) has emerged as an alternative approach for optimizing the traffic signal control system. However, there is a restricted exploration problem encountered when a signal control model is trained with a predefined demand scenario in the traffic simulation. With the restricted exploration, the model learns a policy based only on partial experiences in the search space, which yields a partially-trained policy. Partially-trained policy fails to adapt to some unexperienced (‘unexplored’, ‘never-before-seen’) dataset that have different distributions from the training dataset. Although this issue has critical effects on training a signal control model, it has not been considered in the literature. Therefore, this research aims to obtain a transferable policy to enhance the model’s applicability on unexperienced traffic states. The key idea is to represent the state as graph-structured data, and train it using a graph neural network (GNN). Since this approach enables to learn the relationship between the features resulting from the spatial structure of the intersection, it is able to transfer the already-learned knowledge of the relationship to the unexperienced data. In order to investigate the transferability, an experiment is conducted on five unexperienced test demand scenarios. For the evaluation, the performance of the proposed GNN model is compared with the conventional DQN model that is based on vector-valued state. At first, the models are trained with only a single dataset (training demand scenario). Then, they are tested with different unexperienced dataset (test demand scenarios) without additional trainings. The results show that the proposed GNN model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional DQN model fails.

强化学习 (RL) 已成为优化交通信号控制系统的替代方法。然而，当在交通模拟中使用预定义的需求场景训练信号控制模型时，会遇到一个受限的探索问题。通过受限探索，该模型仅基于搜索空间中的部分经验学习策略，从而产生部分训练的策略。部分训练的策略无法适应一些与训练数据集具有不同分布的未经验（“未探索”、“从未见过”）数据集。虽然这个问题对训练信号控制模型有关键影响，但文献中并未考虑到这一点。因此，本研究旨在获得一种可转移的策略，以增强模型对无经验交通状态的适用性。关键思想是将状态表示为图结构数据，并使用图神经网络 (GNN) 对其进行训练。由于这种方法能够学习由交叉点的空间结构产生的特征之间的关系，因此能够将已经学习的关系知识转移到没有经验的数据中。为了研究可转移性，对五个没有经验的测试需求场景进行了实验。为了评估，将所提出的 GNN 模型的性能与基于向量值状态的传统 DQN 模型进行比较。起初，模型仅使用单个数据集进行训练（训练需求场景）。然后，他们在没有额外培训的情况下使用不同的未经验数据集（测试需求场景）进行测试。