Abstract

The problem of general non-linear stochastic optimal control with small Wiener noise is studied. The problem is approximated by a Markov Decision Process. Bellman Equation is solved using Value Iteration (VI) algorithm in the low rank Tensor Train format (TT-VI). In this paper a modification of the TT-VI algorithm called TT-Q-Iteration (TT-QI) is proposed by authors. In it, the nonlinear Bellman Optimality Operator is iteratively applied to the solution as a composition of internal Tensor Train algebraic operations and TT-CROSS algorithm. We show that it has lower asymptotic complexity per iteration than the method existing in the literature, provided that TT-ranks of transition probabilities are small. In test examples of an underpowered inverted pendulum and Dubins cars our method shows up to 3–10 times faster convergence in terms of wall clock time compared with the original method.

中文翻译：

---

TT-QI：用于随机优化控制的张量训练格式的更快值迭代

摘要

研究了具有小维纳噪声的一般非线性随机最优控制问题。该问题由马尔可夫决策过程近似。贝尔曼方程使用低秩张量训练格式 (TT-VI) 中的值迭代 (VI) 算法求解。在本文中，作者提出了一种对 TT-VI 算法的修改，称为 TT-Q-Iteration (TT-QI)。其中，非线性贝尔曼最优算子作为内部 Tensor Train 代数运算和 TT-CROSS 算法的组合被迭代应用于解。我们表明，只要转移概率的 TT 秩很小，它每次迭代的渐进复杂度比文献中现有的方法低。