Abstract
This paper presents several numerical applications of deep learning-based algorithms for discrete-time stochastic control problems in finite time horizon that have been introduced in Huré et al. (2018). Numerical and comparative tests using TensorFlow illustrate the performance of our different algorithms, namely control learning by performance iteration (algorithms NNcontPI and ClassifPI), control learning by hybrid iteration (algorithms Hybrid-Now and Hybrid-LaterQ), on the 100-dimensional nonlinear PDEs examples from Weinan et al. (2017) and on quadratic backward stochastic differential equations as in Chassagneux and Richou (2016). We also performed tests on low-dimension control problems such as an option hedging problem in finance, as well as energy storage problems arising in the valuation of gas storage and in microgrid management. Numerical results and comparisons to quantization-type algorithms Qknn, as an efficient algorithm to numerically solve low-dimensional control problems, are also provided.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Auer P, Cesa-Bianchi N, Fischer P (2002) Finite-time analysis of the multiarmed bandit problem. In: Machine Learning 47.2. ISSN: 1573-0565. https://doi.org/10.1023/A:1013689704352, pp 235–256
Alasseur C, Balata A, Aziza SB, Maheshwari A, Tankov P, Warin X (2019) Regression Monte Carlo for microgrid management. In: ESAIM proceedings and surveys, CEMRACS 2017, pp 46–67
Balata A, Huré C, Laurière M, Pham H, Pimentel I (2019) A class of finite-dimensional numerically solvable McKean-Vlasov control problems. In: ESAIM Proceedings and surveys, CEMRACS 2017, vol 19, pp 114–144
Bertsimas D, Kogan L, Lo AW (2001) Hedging derivative securities and incomplete markets: an ε-arbitrage approach. In: Operations research 49.3, pp 372–397
Carmona R, Ludkovski M (2010) Valuation of energy storage: an optimal switching approach. In: Quantitative finance 26.1, pp 262–304
Chassagneux J-F, Richou A (2016) Numerical simulation of quadratic BSDEs. In: The annals of applied probabilities 26.1, pp 262–304
Weinan E, Han J, Jentzen A (2017) Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. In: Communications in mathematics and statistics 5, vol 5, pp 349–380
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press
Heymann B, Bonnans JF, Martinon P, Silva FJ, Lanas F, Jiménez-Estévez G (2018) Continuous optimal control approaches to microgrid energy management. In: Energy Systems 9.1, pp 59–77
Henry-Labordere P (2017) Deep primal-dual algorithm for BSDEs: Applications of machine learning to CVA and IM. In: SSRN:3071506
Huré C, Pham H, Bachouch A, Langrené N (2018) Deep neural networks algorithms for stochastic control problems on finite horizon, part I: convergence analysis. In: arXiv:1812.04300
Jiang DR, Powell WB (2015) An approximate dynamic programming algorithm for monotone value functions. In: Operations research 63.6, pp 1489–1511
Kou S, Peng X, Xu X (2018) A general Monte Carlo algorithm with monotonicity for stochastic control problems. 2018 IMS Annual meeting on probability and statistics
Ludkovski M, Maheshwari A (2019) Simulation methods for stochastic storage problems: a statistical learning perspective. In: Energy systems. issn: 1868-3975. https://doi.org/10.1007/s12667-018-0318-4
Pagès G, Pham H, Printems J (2004) Optimal quantization methods and applications to numerical problems in finance. In: Handbook of computational and numerical methods in finance, pp 253–297
Richou A (2010) Etude théorique et numérique des équations différentielles stochastiques rétrogrades. PhD thesis. Université, de Rennes 1
Richou A (2011) Numerical simulation of BSDEs with drivers of quadratic growth. In: The annals of applied probability 21.5, pp 1933–1964
Sutton RS, Barto AG (1998) Reinforcement learning. The MIT Press
Wai-Nam QC, Mikael J, Warin X (2019) Machine learning for semi linear PDEs. In: Journal of scientific computing 79.3, pp 1667–1712
Yong J, Zhou X (1999) Stochastic controls hamiltonian systems and HJB equations. Springer
Acknowledgments
We are grateful to both referees for helpful comments and remarks.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Achref Bachouch’s research is carried out with support of the Norwegian Research Council, within the research project Challenges in Stochastic Control, Information and Applications (STOCONINF), project number 250768/F20
The work of Huyên Pham is supported by the ANR project CAESARS (ANR-15-CE05-0024), and also by FiME and the “Finance and Sustainable Development” EDF - CACIB Chair
Rights and permissions
About this article
Cite this article
Bachouch, A., Huré, C., Langrené, N. et al. Deep Neural Networks Algorithms for Stochastic Control Problems on Finite Horizon: Numerical Applications. Methodol Comput Appl Probab 24, 143–178 (2022). https://doi.org/10.1007/s11009-019-09767-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11009-019-09767-9