Elsevier

Applied Numerical Mathematics

Volume 157, November 2020, Pages 470-489
Applied Numerical Mathematics

Strong-order conditions of Runge-Kutta method for stochastic optimal control problems

https://doi.org/10.1016/j.apnum.2020.07.002Get rights and content

Abstract

In this work, we obtain strong-order conditions of the Runge-Kutta method for the optimal control of the stochastic differential equations. We follow the discretize-then-optimize approach in order to get the optimality system. We compare Stratonovich-Taylor expansions of the exact solution and approximation method of the stochastic optimal control problem defined by the Runge-Kutta scheme successively to get the strong-order conditions. We derive the strong order-1.5 conditions and verify by a numerical example.

Introduction

Stochastic optimal control theory has been challenging over many years in science, engineering, finance and economics. Since it is difficult to find the analytical solutions of stochastic optimal control problems, recent works have been concentrated on the numerical solutions. Recently, there have been many investigations related to stochastic Runge-Kutta methods [2], [3], [4], [13], [14]. Runge-Kutta method is one of the most famous numerical schemes for the numerical solution of deterministic optimal control problems [1], [6], [8], [10], [11]. It is for the first time that Runge-Kutta scheme was applied to stochastic optimal control problems [16].

We let (Wt)t0tT be a 1-dimensional Brownian motion on the filtered probability space (Ω,F,(Ft)t[t0,T],P), where t0>0 and Ω=[t0,T] is a fixed finite interval. On this probability space, the space of real-valued square-integrable (Ft)-adapted processes is defined over L2(t0,T).

We consider a controlled stochastic differential equation (SDE)dyt=f(yt,ut)dt+h(yt)dWt(t[t0,T]),y(t0)=y0, where f and h are continuously differentiable functions with respect to (y,u) and y, respectively, and their derivatives are uniformly bounded. Under these assumptions, we assure that Eqn. (1) has a unique solution [9]. Also, u=(ut)t[t0,T] is a control process in A, which is a closed convex set in the control space L2(t0,T).

Here, we note that the diffusion term does not contain the control process. However, in the general case the control process could appear in the diffusion term.

Moreover, the objective of the optimal control problem is to minimize the cost functionalJ(u)=E[ϕ(yT)+t0Tg(yt,ut)dt], where ϕ and g are continuously differentiable functions. A control process u that solves this problem is called an optimal control.

In [16], we have obtained the Runge-Kutta scheme for stochastic optimal control problems by using discretize-then-optimize approach. Whenever Runge-Kutta discretization of the state variable y is given, Runge-Kutta discretization of the adjoint pair (p,q) is obtained by means of Lagrangian method. The beauty of this method is that the Runge-Kutta discretization of q is derived directly.

Furthermore, it is important to measure the accuracy of the Runge-Kutta scheme by either using the strong-order convergence or weak-order convergence criteria. In this work, our aim is to get strong-order conditions of the Runge-Kutta scheme for stochastic optimal control problems.

We let ζ(T) be a numerical approximation to X(tN) after N steps with constant step size Δ=(tNt0)/N; then ζ(T) is said to converge strongly to X with order r>0 if there exists a constant C>0, which does not depend on Δ, and a Δ0>0 such thatE[|ζ(T)X(tN)|]CΔr,Δ(t0,Δ0). Here, by assuming exact initial values, Stratonovich-Taylor expansions of the exact solution and the solution based on the Runge-Kutta scheme are compared to find the order of accuracy. In the Runge-Kutta scheme for stochastic optimal control problems, Runge-Kutta coefficients of the adjoint process have been obtained in terms of the Runge-Kutta coefficients of the state process in [16]. This yields additional order conditions to classical Runge-Kutta method of SDEs [2], [3] for the strong-order of accuracy. In this work, such order conditions are derived explicitly.

The paper is organized as follows. In Section 2, we give the optimality conditions and discretization with Runge-Kutta scheme. Then, we state the Runge-Kutta scheme and obtain the strong order-1.5 conditions in Section 3. In Section 4, we give a numerical experiment to confirm convergence order. Finally, we conclude and give an outlook to future studies.

Section snippets

Stochastic optimal control problem and discretization

Now, we state our optimal control problem as:(Pc){minimizeuL2(t0,T)J(u)=E[ϕ(yT)+t0Tg(yt,ut)dt]subject to dyt=f(yt,ut)dt+h(yt)dWt(t[t0,T]),yt0=y0. We assume that f, h, ϕ and g are continuously differentiable functions and the problem (P) has a unique solution [17].

Strong-order conditions of the Runge-Kutta scheme for the stochastic optimal control problems

Now, we have a discrete-state equation and a discrete-cost functional. In the following proposition, we get the discrete optimality conditions of (Pd).

Proposition 3.1

[16] If αi0 and βi0, i=1,2,,s, in the system problem (Pd), then the discrete first-order optimality conditions of problem (Pd) are obtained as(OCd){yk+1=yk+Δi=1sαif_(yki,uki)+ΔWi=1sβih(yki),yki=yk+Δj=1saijf_(ykj,ukj)+ΔWj=1sbijh(ykj),pk+1=pkΔi=1sα˜iH_y(yki,uki,pki,qki)+ΔWi=1sβ˜ih(yki)qki,pki=pkΔj=1sa˜ijHy_(ykj,ukj,pkj,qkj)+ΔWj=1sb˜

Numerical application

In this section, we choose a numerical example whose exact solution is known. Herewith, convergence rates can be computed explicitly. To solve the optimization problem, we employ gradient-descent method with a stopping criterion 1e7. We also use 1000 paths of Monte-Carlo simulation.

Example

We consider the following Black-Scholes type of optimal control problem [7]:{minimizeuL2(0,T)J(u)=120TE[(ytyt)2]dt+120Tut2dt,subject to dyt=utytdt+σytdWt,y(0)=y0, where σ>0 is a constant and yt is given.

Also

Conclusion and outlook

In this study, we have derived strong-order conditions of the Runge-Kutta scheme for stochastic optimal control problems. For this reason, we have compared Stratonovich-Taylor expansion of the exact solution and Stratonovich-Taylor expansion of the approximation method defined by the Runge-Kutta scheme successively. It is the first time that stochastic Runge-Kutta scheme applied to optimal control of SDEs is analyzed for the strong order-1.5 conditions. Compared to the Runge-Kutta schemes of

References (17)

  • K. Burrage et al.

    High strong order explicit Runge-Kutta methods for stochastic ordinary differential equations

    Appl. Numer. Math.

    (1996)
  • J.F. Bonnans et al.

    Computation of order conditions for symplectic partitioned Runge-Kutta schemes with application to optimal control

    Numer. Math.

    (2006)
  • P.M. Burrage

    Runge-Kutta Methods for Stochastic Differential Equations

    (1999)
  • K. Burrage et al.

    Order conditions of stochastic Runge-Kutta methods by B-series

    SIAM J. Numer. Anal.

    (2000)
  • J.C. Butcher

    Numerical Methods for Ordinary Differential Equations

    (2003)
  • A.L. Dontchev et al.

    Second-order Runge-Kutta approximations in control constrained optimal control

    SIAM J. Numer. Anal.

    (2001)
  • N. Du et al.

    An effective gradient projection method for stochastic optimal control

    Int. J. Numer. Anal. Model.

    (2013)
  • W.W. Hager

    Runge-Kutta methods in optimal control and the transformed adjoint system

    Numer. Math.

    (2000)
There are more references available in the full text version of this article.

Cited by (0)

View full text