Abstract This work creates a framework for solving highly non-linear satellite formation control problems by using model-free policy optimisation deep reinforcement learning (DRL) methods. This work considers, believed to be for the first time, DRL methods, such as advantage actor-critic method (A2C) and proximal policy optimisation (PPO), to solve the example satellite formation problem of propellantless planar phasing of multiple satellites. Three degree-of-freedom simulations, including a novel surrogate propagation model, are used to train the deep reinforcement learning agents. During training, the agents actuated their motion through cross-sectional area changes which altered the environmental accelerations acting on them. The DRL framework designed in this work successfully coordinated three spacecraft to achieve a propellantless planar phasing manoeuvre. This work has created a DRL framework that can be used to solve complex satellite formation flying problems, such as planar phasing of multiple satellites and in doing so provides key insights into achieving optimal and robust formation control using reinforcement learning.

中文翻译：

---

使用深度强化学习对多颗卫星进行无推进平面定相

摘要 这项工作创建了一个框架，通过使用无模型策略优化深度强化学习 (DRL) 方法来解决高度非线性的卫星编队控制问题。这项工作被认为是第一次考虑 DRL 方法，例如优势参与者-评论家方法 (A2C) 和近端策略优化 (PPO)，以解决多卫星无推进剂平面定相的示例卫星形成问题。三个自由度模拟，包括一个新的代理传播模型，用于训练深度强化学习代理。在训练期间，代理通过横截面积变化来驱动它们的运动，这改变了作用于它们的环境加速度。在这项工作中设计的 DRL 框架成功地协调了三个航天器，以实现无推进剂平面定相机动。这项工作创建了一个 DRL 框架，可用于解决复杂的卫星编队飞行问题，例如多颗卫星的平面定相，并为此提供了使用强化学习实现最佳和稳健编队控制的关键见解。