Abstract To benefit from the accurate simulation and high-throughput data contributed by advanced digital twin technologies in modern smart plants, the deep reinforcement learning (DRL) method is an appropriate choice to generate a self-optimizing scheduling policy. This study employs the deep Q-network (DQN), which is a successful DRL method, to solve the dynamic scheduling problem of flexible manufacturing systems (FMSs) involving shared resources, route flexibility, and stochastic arrivals of raw products. To model the system in consideration of both manufacturing efficiency and deadlock avoidance, we use a class of Petri nets combining timed-place Petri nets and a system of simple sequential processes with resources (S3PR), which is named as the timed S3PR. The dynamic scheduling problem of the timed S3PR is defined as a Markov decision process (MDP) that can be solved by the DQN. For constructing deep neural networks to approximate the DQN action-value function that maps the timed S3PR states to scheduling rewards, we innovatively employ a graph convolutional network (GCN) as the timed S3PR state approximator by proposing a novel graph convolution layer called a Petri-net convolution (PNC) layer. The PNC layer uses the input and output matrices of the timed S3PR to compute the propagation of features from places to transitions and from transitions to places, thereby reducing the number of parameters to be trained and ensuring robust convergence of the learning process. Experimental results verify that the proposed DQN with a PNC network can provide better solutions for dynamic scheduling problems in terms of manufacturing performance, computational efficiency, and adaptability compared with heuristic methods and a DQN with basic multilayer perceptrons.

中文翻译：

---

基于Petri网的柔性制造系统动态调度，通过深度强化学习和图卷积网络

摘要 为了从现代智能工厂中先进的数字孪生技术提供的精确模拟和高吞吐量数据中受益，深度强化学习（DRL）方法是生成自优化调度策略的合适选择。本研究采用深度 Q 网络 (DQN)，这是一种成功的 DRL 方法，解决了涉及共享资源、路线灵活性和原材料随机到达的柔性制造系统 (FMS) 的动态调度问题。为了在考虑制造效率和避免死锁的情况下对系统进行建模，我们使用了一类 Petri 网，结合了定时地点 Petri 网和具有资源的简单顺序过程系统（S3PR），称为定时 S3PR。定时 S3PR 的动态调度问题被定义为可以通过 DQN 解决的马尔可夫决策过程 (MDP)。为了构建深度神经网络来逼近将定时 S3PR 状态映射到调度奖励的 DQN 动作值函数，我们创新地采用图卷积网络 (GCN) 作为定时 S3PR 状态逼近器，提出了一种称为 Petri 的新型图卷积层。网络卷积 (PNC) 层。PNC 层使用定时 S3PR 的输入和输出矩阵来计算特征从位置到转换以及从转换到位置的传播，从而减少要训练的参数数量并确保学习过程的稳健收敛。