Stochastic Dynamic Cutting Plane for Multistage Stochastic Convex Programs

Abstract

We introduce Stochastic Dynamic Cutting Plane (StoDCuP), an extension of the Stochastic Dual Dynamic Programming (SDDP) algorithm to solve multistage stochastic convex optimization problems. At each iteration, the algorithm builds lower bounding affine functions not only for the cost-to-go functions, as SDDP does, but also for some or all nonlinear cost and constraint functions. We show the almost sure convergence of StoDCuP. We also introduce an inexact variant of StoDCuP where all subproblems are solved approximately (with bounded errors) and show the almost sure convergence of this variant for vanishing errors. Finally, numerical experiments are presented on nondifferentiable multistage stochastic programs where Inexact StoDCuP computes a good approximate policy quicker than StoDCuP while SDDP and the previous inexact variant of SDDP combined with Mosek library to solve subproblems were not able to solve the differentiable reformulation of the problem.

Appendix

To prove (3.40) and (3.53), we will need the following lemma (the proof of (ii) of this lemma was given in [5] for a more general sampling scheme and the proof of (i), that we detail, is similar to the proof of (ii)):

Lemma A.1

Assume that Assumptions (H0), (H1)-Sto, and (H2) hold for StoDCuP. Define random variables $y_n^k = 1(k \in {\mathcal {S}}_n)$.

• (i) Let $\varepsilon >0$, $t \in \{1,\ldots ,T\}$, $n \in {\mathtt{Nodes}(t-1)}$, $m \in C(n)$, $i \in \{1,\ldots ,p\}$ and set

  $$\begin{aligned} K_{\varepsilon , m , i}=\left\{ k \ge 1 : g_{t i}( x_m^{k}, x_{n}^{k} , \xi _m ) - g_{t i j_t(m)}^{k-1}( x_m^{k}, x_{n}^{k} ) \ge \varepsilon \right\} . \end{aligned}$$

  Let

  $$\begin{aligned} \Omega _0( \varepsilon )=\{ \omega \in \Omega : |K_{\varepsilon , m , i}(\omega )| \text{ is } \text{ infinite } \} \end{aligned}$$

  and assume that $\Omega _0( \varepsilon ) \ne \emptyset$. Define on the sample space $\Omega _0 ( \varepsilon )$ the random variables ${\mathcal {I}}_{\varepsilon , m , i}(j), j \ge 1$, where ${\mathcal {I}}_{\varepsilon , m , i}(1)=\min \{k \ge 1: k \in K_{\varepsilon , m , i}(\omega ) \}$ and for $j \ge 2$

  $$\begin{aligned} {\mathcal {I}}_{\varepsilon , m , i}(j)=\min \{ k > {\mathcal {I}}_{\varepsilon , m , i}(j-1) : k \in K_{\varepsilon , m , i}(\omega )\}, \end{aligned}$$

  i.e., ${\mathcal {I}}_{\varepsilon , m , i}(j)(\omega )$ is the index of jth iteration k such that $g_{t i}( x_m^{k}, x_{n}^{k} , \xi _m ) - g_{t i j_t (m)}^{k-1}( x_m^{k}, x_{n}^{k} ) \ge \varepsilon$. Then random variables $(y_n^{{\mathcal {I}}_{\varepsilon , m , i}(j)})_{j \ge 1}$ defined on sample space $\Omega _0(\varepsilon )$ are independent, have the distribution of $y_n^1$ and therefore by the Strong Law of Large numbers we have

  $$\begin{aligned} {\mathbb {P}}\left( \lim _{N \rightarrow +\infty } \frac{1}{N} \sum _{j=1}^N y_n^{{\mathcal {I}}_{\varepsilon , m , i}(j)}= {\mathbb {E}}[y_n^1] \right) =1. \end{aligned}$$

  (A.1)

• (ii) Let $\varepsilon >0$, $t \in \{1,\ldots ,T\}$, $n \in {\mathtt{Nodes}(t-1)}$, and set

  $$\begin{aligned} K_{\varepsilon , n}=\left\{ k \ge 1 : {\mathcal {Q}}_t ( x_{n}^{k} ) - {\mathcal {Q}}_t^k ( x_{n}^{k} ) \ge \varepsilon \right\} . \end{aligned}$$

  Let

  $$\begin{aligned} \Omega _1( \varepsilon )=\{\omega \in \Omega : |K_{\varepsilon , n}(\omega )| \text{ is } \text{ infinite } \} \end{aligned}$$

  and assume that $\Omega _1( \varepsilon ) \ne \emptyset$. Define on the sample space $\Omega _1( \varepsilon )$ the random variables ${\mathcal {I}}_{\varepsilon , n}(j), j \ge 1$, where ${\mathcal {I}}_{\varepsilon , n}(1)=\min \{k \ge 1: k \in K_{\varepsilon , n}(\omega ) \}$ and for $j \ge 2$

  $$\begin{aligned} {\mathcal {I}}_{\varepsilon , n}(j)=\min \{ k > {\mathcal {I}}_{\varepsilon , n}(j-1) : k \in K_{\varepsilon , n}(\omega )\}, \end{aligned}$$

  i.e., ${\mathcal {I}}_{\varepsilon , n}(j)(\omega )$ is the index of jth iteration k such that ${\mathcal {Q}}_t ( x_{n}^{k} ) - {\mathcal {Q}}_t^k ( x_{n}^{k} ) \ge \varepsilon$. Then random variables $(y_n^{{\mathcal {I}}_{\varepsilon , n}(j)})_{j \ge 1}$ defined on sample space $\Omega _1( \varepsilon )$ are independent, have the distribution of $y_n^1$ and therefore by the Strong Law of Large numbers we have

  $$\begin{aligned} {\mathbb {P}}\left( \lim _{N \rightarrow +\infty } \frac{1}{N} \sum _{j=1}^N y_n^{{\mathcal {I}}_{\varepsilon , n}(j)}= {\mathbb {E}}[y_n^1] \right) =1. \end{aligned}$$

  (A.2)

Proof

(i) Define on the sample space $\Omega _0( \varepsilon )$ the random variables $(w_{\varepsilon , m , i}^k)_k$ by

$$\begin{aligned} w_{\varepsilon , m , i}^k(\omega ) =\left\{ \begin{array}{ll} 1 &{} \quad \text{ if } k \in K_{\varepsilon , m , i}(\omega ) \\ 0 &{} \quad \text{ otherwise. } \end{array} \right. \end{aligned}$$

To alleviate notation ($\varepsilon , m , n, i$ being fixed), let us put $w^k := w_{\varepsilon , m , i}^k$, ${\mathcal {I}}(j):={\mathcal {I}}_{\varepsilon , m , i}(j)$, For ${{\overline{y}}}_j \in \{0,1\}$, we have

$$\begin{aligned} {\mathbb {P}}\Big (y_n^{{\mathcal {I}}(j)} = {{\overline{y}}}_j \Big ) = \sum _{{\overline{{\mathcal {I}}}}_j =1}^{\infty } {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_j} = {{\overline{y}}}_j ; {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j \Big ). \end{aligned}$$

(A.3)

Observe that the event ${\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j$ can be written as the union $\bigcup _{1 \le {\overline{{\mathcal {I}}}}_1< {\overline{{\mathcal {I}}}}_2< \ldots < {\overline{{\mathcal {I}}}}_j} E({\overline{{\mathcal {I}}}}_1,\ldots ,{\overline{{\mathcal {I}}}}_j)$ of events

$$\begin{aligned} E({\overline{{\mathcal {I}}}}_1,\ldots ,{\overline{{\mathcal {I}}}}_j):= \left\{ \begin{array}{l} w^{\overline{{\mathcal {I}}}_1} =\ldots =w^{\overline{{\mathcal {I}}}_j}=1,\\ w^{\ell } = 0, 1 \le \ell < \overline{{\mathcal {I}}}_j, \ell \notin \{ \overline{{\mathcal {I}}}_1,\ldots , \overline{{\mathcal {I}}}_j \} \end{array} \right\} . \end{aligned}$$

Due to Assumption (H2) observe that random variable $y_n^{{\overline{{\mathcal {I}}}}_j}$ is independent of random variables $w^{i}, i=1,\ldots ,\overline{{\mathcal {I}}}_j$, and therefore events $\{y_n^{{\overline{{\mathcal {I}}}}_j} = {{\overline{y}}}_j \}$ and $\{ {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j \}$ are independent which gives

$$\begin{aligned} \begin{aligned} {\mathbb {P}}\Big (y_n^{{\mathcal {I}}(j)} = {{\overline{y}}}_j \Big )&=\displaystyle \sum _{{\overline{{\mathcal {I}}}}_j =1}^{\infty } {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_j} = {{\overline{y}}}_j ; {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j \Big ) \\&=\displaystyle \sum _{{\overline{{\mathcal {I}}}}_j =1}^{\infty } {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_j} = {{\overline{y}}}_j \Big ) {\mathbb {P}}\Big ( {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j \Big )\\&= \displaystyle {\mathbb {P}}\Big (y_n^{1} = {{\overline{y}}}_j \Big ) \sum _{{\overline{{\mathcal {I}}}}_j =1}^{\infty } {\mathbb {P}}\Big ( {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j \Big )= {\mathbb {P}}\Big (y_n^{1} = {{\overline{y}}}_j \Big ) \end{aligned} \end{aligned}$$

(A.4)

where we have used the fact that $y_n^{1}$ and $y_n^{{\overline{{\mathcal {I}}}}_j}$ have the same distribution (from (H2)).

Next for ${{\overline{y}}}_1,\ldots ,{{\overline{y}}}_p \in \{0,1\}$, we have

$$\begin{aligned} \begin{aligned}&{\mathbb {P}}\Big (y_n^{{\mathcal {I}}(1)} = {{\overline{y}}}_1, \ldots , y_n^{{\mathcal {I}}(p)} = {{\overline{y}}}_p \Big ) \\&\quad = \displaystyle \sum _{1 \le {\overline{{\mathcal {I}}}}_1< {\overline{{\mathcal {I}}}}_2< \ldots < {\overline{{\mathcal {I}}}}_p}^{\infty } {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_1} = {{\overline{y}}}_1 ; \ldots ;y_n^{{\overline{{\mathcal {I}}}}_p} = {{\overline{y}}}_p ; {\mathcal {I}}(1) = {\overline{{\mathcal {I}}}}_1 ;\ldots ;{\mathcal {I}}(p) = {\overline{{\mathcal {I}}}}_p \Big ). \end{aligned} \end{aligned}$$

By the same reasoning as above, the event

$$\begin{aligned} \left\{ y_n^{{\overline{{\mathcal {I}}}}_1} = {{\overline{y}}}_1 ; \ldots ;y_n^{{\overline{{\mathcal {I}}}}_{p-1}} = {{\overline{y}}}_{p-1} ; {\mathcal {I}}(1) = {\overline{{\mathcal {I}}}}_1 ;\ldots ;{\mathcal {I}}(p) = {\overline{{\mathcal {I}}}}_p \right\} \end{aligned}$$

can be expressed in terms of random variables $y_n^{{\overline{{\mathcal {I}}}}_1},\ldots ,y_n^{{\overline{{\mathcal {I}}}}_{p-1}},w_n^{{\overline{{\mathcal {I}}}}_{1}},\ldots ,w_n^{{\overline{{\mathcal {I}}}}_{p}}$, and is therefore independent of event $\{ y_n^{{\overline{{\mathcal {I}}}}_p} = {{\overline{y}}}_p \}$. It follows that

$$\begin{aligned} \begin{aligned}&{\mathbb {P}}\Big (y_n^{{\mathcal {I}}(j)} = {{\overline{y}}}_j, 1 \le j \le p \Big )\\&\quad = \displaystyle \sum _{1 \le {\overline{{\mathcal {I}}}}_1< {\overline{{\mathcal {I}}}}_2< \ldots< {\overline{{\mathcal {I}}}}_p}^{\infty } {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_p} = {{\overline{y}}}_p \Big ) {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_j} = {{\overline{y}}}_j, 1 \le j \le p-1 ; {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j, 1 \le j \le p \Big )\\&\quad = {\mathbb {P}}\Big (y_n^{1} = {{\overline{y}}}_p \Big ) \displaystyle \sum _{1 \le {\overline{{\mathcal {I}}}}_1< {\overline{{\mathcal {I}}}}_2< \ldots< {\overline{{\mathcal {I}}}}_p}^{\infty } {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_j} = {{\overline{y}}}_j, 1 \le j \le p-1 ; {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j, 1 \le j \le p \Big )\\&\quad = {\mathbb {P}}\Big (y_n^{1} = {{\overline{y}}}_p \Big ) \displaystyle \sum _{1 \le {\overline{{\mathcal {I}}}}_1< {\overline{{\mathcal {I}}}}_2< \ldots < {\overline{{\mathcal {I}}}}_{p-1}}^{\infty } {\mathbb {P}}\Big (y_n^{{\overline{{\mathcal {I}}}}_j} = {{\overline{y}}}_j, 1 \le j \le p-1 ; {\mathcal {I}}(j) = {\overline{{\mathcal {I}}}}_j, 1 \le j \le p-1 \Big )\\&\quad = {\mathbb {P}}\Big (y_n^{1} = {{\overline{y}}}_p \Big ) {\mathbb {P}}\Big (y_n^{{\mathcal {I}}(j)} = {{\overline{y}}}_j, 1 \le j \le p-1 \Big ). \end{aligned} \end{aligned}$$

(A.5)

By induction this implies

$$\begin{aligned} \begin{aligned} {\mathbb {P}}\Big ( y_n^{{\mathcal {I}}(j)} = {{\overline{y}}}_j, 1 \le j \le p \Big )&= \displaystyle \prod _{j=1}^p {\mathbb {P}}\Big (y_n^{1} = {{\overline{y}}}_j \Big ) {\mathop {=}\limits ^{(6.4)}} \displaystyle \prod _{j=1}^p {\mathbb {P}}\Big ( y_n^{{\mathcal {I}}(j)} = {{\overline{y}}}_j \Big ) \end{aligned} \end{aligned}$$

(A.6)

which shows that random variables $(y_n^{{\mathcal {I}}(j)})_{j \ge 1}$ are independent.

The proof of (ii) is similar to the proof of (i). $\square$

Proof of (3.40) and (3.53). As in [5], we can now use the previous lemma to prove (3.40) and (3.53). Let us prove (3.40). By contradiction, assume that (3.40) does not hold. Then, there is $\varepsilon >0$ such that the set $\Omega _0( \varepsilon )$ defined in Lemma A.1 is nonempty. By Lemma A.1, this implies that (A.1) holds. But due to (3.39), only a finite number of indices ${{\mathcal {I}}_{\varepsilon , m , i}(j)}$ can be in ${\mathcal {S}}_n$ (with corresponding variable $y_n^{{{\mathcal {I}}_{\varepsilon , m , i}(j)}}$ being one) and therefore ${\mathbb {P}}\left( \lim _{N \rightarrow +\infty } \frac{1}{N} \sum _{j=1}^N y_n^{{\mathcal {I}}_{\varepsilon , m , i}(j)}= 0 \right) =1$, which is a contradiction with (A.1).

The proof of (3.53) is similar to the proof of (3.40), by contradiction and using (3.52) and Lemma A.1-(ii) (see also [5, 6]). $\square$