On Optimization Techniques for the Construction of an Exponential Estimate for Delayed Recurrent Neural Networks

Abstract

This work is devoted to the modeling and investigation of the architecture design for the delayed recurrent neural network, based on the delayed differential equations. The usage of discrete and distributed delays makes it possible to model the calculation of the next states using internal memory, which corresponds to the artificial recurrent neural network architecture used in the field of deep learning. The problem of exponential stability of the models of recurrent neural networks with multiple discrete and distributed delays is considered. For this purpose, the direct method of stability research and the gradient descent method is used. The methods are used consequentially. Firstly we use the direct method in order to construct stability conditions (resulting in an exponential estimate), which include the tuple of positive definite matrices. Then we apply the optimization technique for these stability conditions (or of exponential estimate) with the help of a generalized gradient method with respect to this tuple of matrices. The exponential estimates are constructed on the basis of the Lyapunov–Krasovskii functional. An optimization method of improving estimates is offered, which is based on the notion of the generalized gradient of the convex function of the tuple of positive definite matrices. The search for the optimal exponential estimate is reduced to finding the saddle point of the Lagrange function.

1. Introduction

Breakthrough results in the field of deep machine learning are obtained nowadays using recurrent neural networks (RNN). In particular, the construction of machine learning models for problems of image recognition with captioning, natural language processing and translation, was made possible by recurrent neural networks with Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU). The paper [1] offers a description of such models using ordinary differential equations. Further research has to be related to the systems with time delays as they are modeling the memory within the network units. In [2] drawing from concepts in signal processing, they formally derived the canonical RNN formulation from differential equations.

Here our study of the RNN model is based on the system with multiple discrete and distributed time-varying delays

$$\begin{array}{cc}\hfill \dot{x}\left(t\right)=& -Ax\left(t\right)+\sum _{k=1}^{r_1}{W}_{1,k}g\left(x(t-h_k\left(t\right))\right)+\sum _{m=1}^{r_2}{W}_{2,m}{\int }_{t-{\tau }_m\left(t\right)}^{t}g\left(x\right(\theta \left)\right)d\theta ,\hfill \end{array}$$

(1)

where $x\left(t\right)\in {\mathbb{R}}^n$ is the state vector, $A=diag(a_1,a_2,\dots ,a_n)$ is a diagonal matrix with positive entries $a_i>0$. For the i-th neuron $1/a_i$ can be interpreted as the activity decay constant (or time constant). $W_{1,k}={\left(w_{ij}^{1,k}\right)}_{n\times n}$, $k=\overline{1,r_1}$, $W_{2,m}={\left(w_{ij}^{2,m}\right)}_{n\times n}$, $m=\overline{1,r_2}$ are the synaptic connection weight matrices. The entries of $W_{1,k}$ and $W_{2,m}$ may be positive (excitatory synapses) or negative (inhibitory synapses). $g\left(x\left(t\right)\right)={[g_1\left(x\left(t\right)\right),g_2\left(x\left(t\right)\right),\dots ,g_n\left(x\left(t\right)\right)]}^{\top }\in {\mathbb{R}}^n$ is the non-decreasing activation function, which belongs to sector non-linear function class defined by

$$\begin{array}{c}\hfill g_j\left(0\right)=0\mathrm{and}0\le \frac{g_j\left({\xi }_1\right)-g_j\left({\xi }_2\right)}{{\xi }_1-{\xi }_2}\le l_j,l_j>0,\end{array}$$

(2)

${\xi }_1,{\xi }_2\in \mathbb{R}$, ${\xi }_1\ne {\xi }_2$, $j\in \overline{1,n}$ and $x=0$ is a fixed point of Equation (1). We let $L=diag(l_1,l_2,\dots ,l_n)$ is a diagonal matrix with positive entries $l_j>0$.

The system (1) includes discrete and distributed time-varying delays, which are described with the help of the second and the third terms correspondingly.

The bounded differentiable functions $h_k\left(t\right)$ represent discrete delays of system with

$$0\le h_k\left(t\right)\le h_{M,k},$$

and

$${\dot{h}}_k\left(t\right)\le h_{D,k}<1,$$

(3)

$k=\overline{1,r_1}$, $t>0$. Delays $h_k\left(t\right)$ and ${\tau }_m\left(t\right)$ have physical meaning as “controllable memory” if previous states of neurons effects on output only during some time intervals. $h_{M,k}$ and $h_{D,k}$ are bounds of the delay and its derivative for discrete delays.

The bounded functions ${\tau }_m\left(t\right)$ represent distributed delays of system with $0\le {\tau }_m\left(t\right)\le {\tau }_{M,m}$, $m=\overline{1,r_2}$.

The bounded functions $h_k\left(t\right)$ and ${\tau }_m\left(t\right)$ represent axonal signal transmission delays. The condition (3) for derivative ${\dot{h}}_k\left(t\right)$ will be applied when estimating the upper right derivative of Lyapunov–Krasovskii functional (see, for example, [3]).

The initial conditions associated with system (1) are assumed to be

$$\begin{array}{cc}\hfill x\left(s\right)=& \varphi \left(s\right),s\in [-{\tau }_M,0],\hfill \\ \hfill & {\tau }_M:=max\left\{h_{M,k},k=\overline{1,r_1},{\tau }_{M,m},m=\overline{1,r_2}\right\},\hfill \end{array}$$

(4)

where $\varphi \left(s\right)\in C[-{\tau }_M,0]$.

Given any $\varphi \left(s\right)\in C[-{\tau }_M,0]$, under the assumption (2), there exists a unique trajectory of (1) starting from $\varphi$ [4].

Here we use the Hopfield neural network model, which includes a diagonal matrix A with positive entries, that shows the self-connection of the neuron. That is the next state of the neuron is dependent on its current state and outputs of eventually all neurons. Such a diagonal matrix is traditionally applied in stability research of continuous-time RNNs [5]. On the other hand, if we used arbitrary matrix A, we would assume the next state of a neuron is dependent on its current state as well as the states of all other neurons, which means that the internal states of all neurons are accessible from outside. It contradicts that, for example, in the case of the LSTM unit only the hidden state vector (also known as the output vector) is seen.

Following the work [2], we may give the interpretation of the model (1) from the viewpoint of signal processing, leading us to “canonical” and “non-canonical” RNNs. Namely, we have $x\left(t\right)$, the state signal vector; $g\left(x\right(t\left)\right)$, the readout signal vector, which is a warped version of the state signal vector; the bias parameters are omitted without loss of generality since they can be used in the transformation resulting in the homogeneous system (1). Initial state $\varphi \left(s\right)$, $s\in [-{\tau }_M,0]$ is considered as an input signal, thus, modeling one-to-many RNN architecture. In more general many-t0-many case, input signal vector $u\left(t\right)$, $t>0$ can be used during the work of the RNN as an “input sequence”.

Although RNNs can actually be described using difference equations, it makes sense to consider continuous-time equations that describe the operation of RNNs. This is due to the fact that differential equations make it possible to better describe and understand the dynamic processes that occur. In addition, with the help of differential equations it is possible to explicitly obtain the conditions for stabilization of recurrent neural networks. This is of great importance in the design of recurrent neural networks. Ref. [5] provide a comprehensive review of the research on the continuous-time recurrent neural networks focusing on the stability of Hopfield and Cohen–Grossberg neural networks.

Note also that the corresponding recurrent neural networks can be obtained by discretizing the models based on differential equations. Thus, work [2] shows the way to construct a RNN of the LSTM type starting from the corresponding model based on differential equations with delay and further through the discretization of the so-called canonical RNN.

Recurrent neural network models have been considered from the 1980s, after the pioneering work of Hopfield [6] modeling each neuron as a linear circuit consisting of a resistor and a capacitor. Two approaches can be differed when investigating the models of recurrent neural networks in the class of delayed differential equations. The first way means research of local stability with the help of comparison with the linearised system [7,8,9,10]. The conditions of the Hopf bifurcation were obtained in [10,11]. The second approach (which is called the direct Lyapunov’s method) uses Lyapunov–Krasovkii functionals [3]. It allows us to get stability conditions constructively, which are formulated in the form of linear-matrix inequalities (LMIs). These stability conditions can be improved with the help of optimization of parameters of Lyapunov–Krasovskii functionals.

Exponential estimates of the solutions are very important when investigating the models of RNNs because they show the rate of convergence of calculations when recognizing input data. In the previous works [12,13], indirect method was developed allowing us to get exponential estimates in some general cases of the RNN models. It results in the numerical solution of quasipolynomial equation. It gives us a clear value of exponential decay, which, unfortunately, does not admit optimization and so, cannot be improved. In order to overcome this shortcoming, here we develop an optimization technique, which is based on the direct method for Liapunov–Krasovskii functionals of the special kind.

2. Exponential Estimate

Let ${\Omega }_n\subset {\mathbb{R}}^{n\times n}$ be a set of all symmetric positive definite matrices. It is an open convex cone because:

(a)

convexity—for any $P_1\in {\Omega }_n$, $P_2\in {\Omega }_n$, $x\in {\mathbb{R}}^n$, and $\xi \in [0,1]$ we have $x^{\top }(\xi P_1+(1-\xi )P_2)x=\xi x^{\top }{P}_{1}x+(1-\xi )x^{\top }{P}_{2}x>0$;

(b)

cone—for any $P\in {\Omega }_n$, $x\in {\mathbb{R}}^n$, and $\eta >0$ we have $\eta x^{\top }Px>0$.

${\overline{\Omega }}_n^1$ is the part of the cone ${\Omega }_n$ contained inside the unit sphere, i.e., ${\overline{\Omega }}_n^1:=\{P\in {\Omega }_n:\parallel P\parallel \le 1\}$. Here $\parallel P\parallel$ is Frobenius norm of the matrix $P\in {\Omega }_n$.

Lemma 1.

Reference [14] For any constant matrix $U\in {\Omega }_n$, scalar $\beta >0$, vector function $u:[0,\beta ]\to {\mathbb{R}}^n$ such that the integrations concerned are well-defined, then

$$\left({\int }_0^{\beta }{u}^{\top }\left(s\right)ds\right)U\left({\int }_0^{\beta }u\left(s\right)ds\right)\le \beta {\int }_0^{\beta }{u}^{\top }\left(s\right)Uu\left(s\right)ds.$$

Lemma 2.

Reference [15] Given any real matrices $W_1$, $W_2$, $W_3$ with appropriate dimensions and a scalar $\beta >0$, $W_3\in {\Omega }_n$, then the following inequality holds

$$W_1^{\top }{W}_2+W_2^{\top }{W}_1\le \beta W_1^{\top }{W}_3{W}_1+{\beta }^{-1}{W}_2^{\top }{W}_3^{-1}{W}_2.$$

In the following definitions, we assume that the trivial solution of (1) be the unique equilibrium point of the model (1).

Definition 1.

The trivial solution of (1) is globally asymptotically stable if for every solution $x\left(t\right)$ to the initial value problem (1)–(4), we have $x\left(t\right)\to 0$ as $t\to \infty$.

Definition 2.

If there exist constants $\alpha >0$, $K>0$, and $T>0$ such that every solution $x\left(t\right)$ to the initial value problem (1)–(4) always satisfies $\parallel x\left(t\right)\parallel \le K{e}^{-\alpha t}$ for all $t>T$, then the trivial solution of (1) is said to be globally exponentially stable.

Our research is based on the following Lyapunov–Krasovskii functional, which is an extension of the one offered in the work [3] for the case of multiple delays

$$\begin{array}{cc}\hfill V\left[x_t(·)\right]& =e^{2\alpha t}{x}^{\top }\left(t\right)Px\left(t\right)+\sum _{k=1}^{r_1}{\int }_{t-h_k\left(t\right)}^t{e}^{2\alpha s}{g}^{\top }\left(x\left(s\right)\right)Q_{k}g\left(x\left(s\right)\right)ds\hfill \\ \hfill & +\sum _{m=1}^{r_2}{\tau }_{M,m}{\int }_{-{\tau }_{M,m}}^0{\int }_{t+\theta }^t{e}^{2\alpha s}{g}^{\top }\left(x\left(s\right)\right)S_{m}g\left(x\left(s\right)\right)dsd\theta ,\hfill \end{array}$$

(5)

where unknown constant $\alpha >0$ and matrices P, $Q_k$, $k=\overline{1,r_1}$, $S_m$, $m=\overline{1,r_2}$ belong to ${\Omega }_n$. Here we use traditional denotation of the element of the solution of (1) as the vector-interval $x_t(·):=\left\{x(t+\theta )\right|\theta \in [-{\tau }_M,0]\}\in C[-{\tau }_M,0]$.

Theorem 1.

We assume that system (1) satisfies the following condition.

H1. Let there exist constant $\alpha >0$ and matrices P, $Q_k$, $k=\overline{1,r_1}$, $S_m$, $m=\overline{1,r_2}$, which belong to $\mathit{relint}\left({\overline{\Omega }}_n^1\right)$, such that the symmetric matrix

$$\begin{array}{cc}\hfill & \Gamma :=\left[\begin{array}{cc}{\Gamma }_{11}& {\Gamma }_{12}\\ {\Gamma }_{21}& {\Gamma }_{22}\end{array}\right],\hfill \\ \hfill & {\Gamma }_{11}:=-2\alpha P+A^{\top }P+PA-L\left(\sum _{k=1}^{r_1}{Q}_k+{\tau }_{M,m}^2\sum _{m=1}^{r_2}{S}_m\right)L\in {\Omega }_n,\hfill \\ \hfill & {\Gamma }_{12}:=\left[\frac{e^{\alpha h_{M,1}}}{\sqrt{1-h_{D,1}}}P{W}_{1,1}\dots \frac{e^{\alpha h_{M,r_1}}}{\sqrt{1-h_{D,r_1}}}P{W}_{1,r_1}{e}^{\alpha {\tau }_{M,1}}P{W}_{2,1}\dots e^{\alpha {\tau }_{M,r_2}}P{W}_{2,r_2}\right]\in {\mathbb{R}}^{n\times n(r_1+r_2)},\hfill \\ \hfill & {\Gamma }_{21}:={\Gamma }_{12}^{\top }\in {\mathbb{R}}^{n(r_1+r_2)\times n},\hfill \\ \hfill & {\Gamma }_{22}:=\left[\begin{array}{cccccc}{Q}_1& & & & & \Theta \\ & \ddots & & & & \\ & & Q_{r_1}& & & \\ & & & S_1& & \\ & & & & \ddots & \\ \Theta & & & & & S_{r_2}\end{array}\right]\in {\Omega }_{n(r_1+r_2)},\hfill \\ \hfill & \Theta \in {\mathbb{R}}^{n\times n}\mathit{is}\mathit{matrix}\mathit{of}\mathit{zeroes},\hfill \end{array}$$

(6)

belong to ${\Omega }_{n(1+r_1+r_2)}$.

Then the trivial solution of (1) is globally asymptotically stable.

Proof. 

Estimating right upper derivative of the functional $V\left[x_t(·)\right]$ along the solution of the system (1), we get

$$\begin{array}{cc}\hfill & \frac{d{V}^{+}[x_t(·)]}{dt}\le e^{2\alpha t}\{x^{\top }\left(t\right)[2\alpha P-A^{\top }P-PA]x\left(t\right)\hfill \\ \hfill & +\left[\sum _{k=1}^{r_1}{g}^{\top }\left(x(t-h_k\left(t\right))\right)W_{1,k}^{\top }Px\left(t\right)+x^{\top }\left(t\right)P\sum _{k=1}^{r_1}{W}_{1,k}g\left(x(t-h_k\left(t\right))\right)\right]\hfill \\ \hfill & +\left[\sum _{m=1}^{r_2}{\int }_{t-{\tau }_m\left(t\right)}^t{g}^{\top }\left(x\left(\theta \right)d\theta \right)W_{2,m}^{\top }Px\left(t\right)+x^{\top }\left(t\right)P\sum _{m=1}^{r_2}{W}_{2,m}{\int }_{t-{\tau }_m\left(t\right)}^{t}g\left(x\right(\theta \left)\right)d\theta \right]\}\hfill \\ \hfill & +\sum _{k=1}^{r_1}{e}^{2\alpha t}\{g^{\top }\left(x\left(t\right)\right)Q_{k}g\left(x\left(t\right)\right)\hfill \\ \hfill & -e^{-2\alpha h_k\left(t\right)}{g}^{\top }\left(x(t-h_k\left(t\right))\right)Q_{k}g\left(x(t-h_k\left(t\right))\right)(1-h_{D,k})\}\hfill \\ \hfill & +\sum _{m=1}^{r_2}{\tau }_{M,m}\{{\tau }_{M,m}{e}^{2\alpha t}{g}^{\top }\left(x\left(t\right)\right)S_{m}g\left(x\left(t\right)\right)\hfill \\ \hfill & -e^{2\alpha (t-{\tau }_{M,m})}{\int }_{t-{\tau }_m\left(t\right)}^t{g}^{\top }\left(x\left(s\right)\right)S_{m}g\left(x\left(s\right)\right)ds\}.\hfill \end{array}$$

(7)

Applying Lemmas 1 and 2 for estimating counterparts of (7), we have

$$\begin{array}{cc}\hfill & \sum _{k=1}^{r_1}\{g^{\top }\left(x(t-h_k\left(t\right))\right)W_{1,k}^{\top }Px\left(t\right)+x^{\top }\left(t\right)P{W}_{1,k}g\left(x(t-h_k\left(t\right))\right)\}\hfill \\ \hfill & =\sum _{k=1}^{r_1}[e^{-\alpha h_{M,k}}{(1-h_{D,k})}^{1/2}{g}^{\top }\left(x(t-h_k\left(t\right))\right)][e^{\alpha h_{M,k}}{(1-h_{D,k})}^{-1/2}{W}_{1,k}^{\top }Px\left(t\right)]\hfill \\ \hfill & +[e^{\alpha h_{M,k}}{(1-h_{D,k})}^{-1/2}{x}^{\top }\left(t\right)P{W}_{1,k}][e^{-\alpha h_{M,k}}{(1-h_{D,k})}^{1/2}g\left(x(t-h_k\left(t\right))\right)]\hfill \\ \hfill & \le \sum _{k=1}^{r_1}\{e^{2\alpha h_{M,k}}{(1-h_{D,k})}^{-1}{x}^{\top }\left(t\right)P{W}_{1,k}{Q}_k^{-1}{W}_{1,k}^{\top }Px\left(t\right)\hfill \\ \hfill & +e^{-2\alpha h_{M,k}}(1-h_{D,k})g^{\top }\left(x(t-h_k\left(t\right))\right)Q_{k}g\left(x(t-h_k\left(t\right))\right)\},\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill & \sum _{m=1}^{r_2}\{{\int }_{t-{\tau }_m\left(t\right)}^t{g}^{\top }\left(x\left(\theta \right)\right)d\theta W_{2,m}\top Px\left(t\right)+x^{\top }\left(t\right)P{W}_{2,m}{\int }_{t-{\tau }_m\left(t\right)}^{t}g\left(x\right(\theta \left)\right)d\theta \}\hfill \\ \hfill & =\sum _{m=1}^{r_2}\{[e^{-\alpha {\tau }_{M,m}}{\int }_{t-{\tau }_m\left(t\right)}^t{g}^{\top }\left(x\left(\theta \right)\right)d\theta ][e^{\alpha {\tau }_{M,m}}{W}_{2,m}^{\top }Px\left(t\right)]\hfill \\ \hfill & +[e^{\alpha {\tau }_{M,m}}{x}^{\top }\left(t\right)P{W}_{2,m}][{\int }_{t-{\tau }_m\left(t\right)}^{t}g\left(x\right(\theta \left)\right)d\theta e^{-\alpha {\tau }_{M,m}}]\}\hfill \\ \hfill & \le \sum _{m=1}^{r_2}\{e^{-2\alpha {\tau }_{M,m}}\left({\int }_{t-{\tau }_m\left(t\right)}^t{g}^{\top }\left(x\left(\theta \right)\right)d\theta \right)S_m\left({\int }_{t-{\tau }_m\left(t\right)}^{t}g\left(x\right(\theta \left)\right)d\theta \right)\hfill \\ \hfill & +e^{2\alpha {\tau }_{M,m}}{x}^{\top }\left(t\right)P{W}_{2,m}{S}_m^{-1}{W}_{2,m}^{\top }Px\left(t\right)\}\hfill \\ \hfill & \le \sum _{m=1}^{r_2}\{{\tau }_{M,m}{e}^{-2\alpha {\tau }_{M,m}}{\int }_{t-{\tau }_m\left(t\right)}^t{g}^{\top }\left(x\left(\theta \right)\right)S_{m}g\left(x\left(\theta \right)\right)d\theta \hfill \\ \hfill & +e^{2\alpha {\tau }_{M,m}}{x}^{\top }\left(t\right)P{W}_{2,m}{S}_m^{-1}{W}_{2,m}^{\top }Px\left(t\right)\}.\hfill \end{array}$$

Finally we get

$$\begin{array}{cc}\hfill \frac{d{V}^{+}\left[x_t(·)\right]}{dt}& \le e^{2\alpha t}{x}^{\top }\left(t\right)\{2\alpha P-A^{\top }P-PA\hfill \\ \hfill & +\sum _{k=1}^{r_1}\left[e^{2\alpha h_{M,k}}{(1-h_{D,k})}^{-1}P{W}_{1,k}{Q}_k^{-1}{W}_{1,k}^{\top }P+L{Q}_{k}L\right]\hfill \\ \hfill & +\sum _{m=1}^{r_2}\left[e^{2\alpha {\tau }_{M,m}}P{W}_{2,m}{S}_m^{-1}{W}_{2,m}^{\top }P+{\tau }_{M,m}^{2}L{S}_{m}L\right]\}x\left(t\right)\hfill \\ \hfill & \le -e^{2\alpha t}{x}^{\top }\left(t\right)\{{\Gamma }_{11}-{\Gamma }_{12}{\Gamma }_{22}^{-1}{\Gamma }_{21}\}x\left(t\right)\hfill \\ \hfill & =-e^{2\alpha t}{x}^{\top }\left(t\right)\Gamma /{\Gamma }_{22}x\left(t\right),\hfill \end{array}$$

(8)

where $\Gamma /{\Gamma }_{22}:={\Gamma }_{11}-{\Gamma }_{12}{\Gamma }_{22}^{-1}{\Gamma }_{21}$ is the Schur complement of $\Gamma$ in ${\Gamma }_{22}$. From the Schur complement it follows that the right side of (8) is negative definite if and only if $\Gamma \in {\Omega }_{n(1+r_1+r_2)}$ [16]. ☐

Corollary 1.

Provided that the condition H1 holds the trivial solution of (1) is globally exponentially stable as follows

$$\parallel x\left(t\right)\parallel \le {\gamma \left(\alpha \right)\left|\varphi \right|}_{{\tau }_M}{e}^{-\alpha t},t>0,$$

(9)

where

$$\begin{array}{cc}\hfill \gamma \left(\alpha \right):=& {\lambda }_{min}^{-1/2}\left(P\right)({\lambda }_{max}\left(P\right)+\sum _{k=1}^{r_1}{\lambda }_{max}\left(Q_k\right)l_{max}^2\frac{1-e^{-2\alpha h_{M,k}}}{2\alpha }\hfill \\ & +\sum _{m=1}^{r_2}{\tau }_{M,m}{\lambda }_{max}\left(L{S}_{m}L\right)\frac{2\alpha {\tau }_{M,m}-1+e^{-2\alpha {\tau }_{M,m}}}{4{\alpha }^2}{)}^{1/2},\hfill \\ \hfill l_{max}:=& max\{l_1,\dots ,l_n\},\hfill \end{array}$$

${\lambda }_{min}(·)$ and ${\lambda }_{max}(·)$ are minimal and maximal eigenvalues of the matrix. Here we use denotions of $\parallel ·\parallel$ as Euclidean norm in ${\mathbb{R}}^n$ and ${\left|x(·)\right|}_{{\tau }_M}:={sup}_{s\in [-{\tau }_M,0]}\parallel x\left(s\right)\parallel$ as the uniform convergence norm in $C[{\tau }_M,0]$.

Proof. 

Firstly note that the inequality

$$2\alpha {\tau }_{M,m}+e^{-2\alpha {\tau }_{M,m}}\ge 1,$$

enables us that the square root expression in $\gamma \left(\alpha \right)$ is nonnegative for $\alpha >0$. From Theorem 1 it follows that $V\left[x_t(·)\right]\le V\left[\varphi (·)\right]$. Hence, we get

$$\begin{array}{cc}\hfill {\lambda }_{min}(P){\parallel x(t)\parallel }^2& \le e^{-2\alpha t}V\left[x_t(·)\right]\le e^{-2\alpha t}V\left[\varphi (·)\right]\hfill \\ \hfill & \le e^{-2\alpha t}({\varphi }^{\top }(0)P\varphi (0)+\sum _{k=1}^{r_1}{\int }_{-h_{M,k}}^0{e}^{2\alpha s}{g}^{\top }(x(s))Q_{k}g(x(s))ds\hfill \\ & +\sum _{m=1}^{r_2}{\tau }_{M,m}{\int }_{-{\tau }_{M,m}}^0{\int }_{\theta }^0{e}^{2\alpha s}{g}^{\top }(\varphi (s))S_{m}g(\varphi (s))dsd\theta )\hfill \\ \hfill & \le e^{-2\alpha t}({\lambda }_{max}(P)+\sum _{k=1}^{r_1}{\lambda }_{max}(Q_k)l_{max}^2{\int }_{-h_{M,k}}^0{e}^{2\alpha s}ds\hfill \\ & +\sum _{m=1}^{r_2}{\tau }_{M,m}{\lambda }_{max}(L{S}_{m}L){\int }_{-{\tau }_{M,m}}^0{\int }_{\theta }^0{e}^{2\alpha s}dsd\theta ){\left|\varphi \right|}_{{\tau }_M}^2\hfill \\ \hfill & =e^{-2\alpha t}({\lambda }_{max}(P)+\sum _{k=1}^{r_1}{\lambda }_{max}(Q_k)l_{max}^2\frac{1-e^{-2\alpha h_{M,k}}}{2\alpha }\hfill \\ & +\sum _{m=1}^{r_2}{\tau }_{M,m}{\lambda }_{max}(L{S}_{m}L)\frac{2\alpha {\tau }_{M,m}-1+e^{-2\alpha {\tau }_{M,m}}}{4{\alpha }^2}){\left|\varphi \right|}_{{\tau }_M}^2.\hfill \end{array}$$

Finally, it yields

$${\lambda }_{min}{\left(P\right)\parallel x\left(t\right)\parallel }^2\le e^{-2\alpha t}\gamma \left(\alpha \right){\left|\varphi \right|}_{{\tau }_M}^2.$$

☐

3. Optimization Method

The condition H1 describes the main stability result. The matrix $\Gamma$ presents operator, which is linear with the respect to the tuple of matrices $\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]$. At the same time, the dependence of $\Gamma$ on $\alpha$ is nonlinear. $\alpha$ is the parameter determining the exponential decay rate. Since optimization of $\Gamma$ with the respect to $\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]$ may be considered as linear matrix inequality, we can reduce the problem of the construction of exponential estimate like (9) to the convex programming. It is natural to assume that “the more positive definite” is the matrix $\Gamma$, the “more asymptotically stable” is the trivial solution. In turn, the solution is “more exponentially” stable. The positive definiteness of the matrix $\Gamma$ is described with the help of minimum eigenvalue. Thus we result in the following optimization problem.

Further we apply optimization technique developed earlier for linear systems in [17]. Given $\alpha >0$ we search the tuple of matrices $(P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2})$ as a solution of the optimization problem

$$\begin{array}{cc}\hfill & \left[P^{\star },Q_1^{\star },\dots ,Q_{r_1}^{\star },S_1^{\star },\dots ,S_{r_2}^{\star }\right]=arg\underset{\begin{array}{c}[P,Q_1,\dots ,Q_{r_1},\hfill \\ S_1,\dots ,S_{r_2}]\hfill \\ \in {\prod }_{i=1}^{1+r_1+r_2}{\overline{\Omega }}_n^1\hfill \end{array}}{inf}{\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right].\hfill \end{array}$$

(10)

Here ${\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]=-{\lambda }_{min}(\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right])$.

We give some general conditions of existence of a solution of problem (10).

Definition 3.

The inner product of the tuples of matrices $\left[P_1,Q_{1,1},\dots ,Q_{r_1,1},S_{1,1},\dots ,S_{r_2,1}\right]$, $\left[P_2,Q_{1,2},\dots ,Q_{r_1,2},S_{1,2},\dots ,S_{r_2,2}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\overline{\Omega }}_n^1$ is

$$\begin{array}{cc}\hfill & \langle \left[P_1,Q_{1,1},\dots ,Q_{r_1,1},S_{1,1},\dots ,S_{r_2,1}\right],\left[P_2,Q_{1,2},\dots ,Q_{r_1,2},S_{1,2},\dots ,S_{r_2,2}\right]\rangle \hfill \\ \hfill & :=\sum _{i,j=1}^n\left(p_{ij}^1{p}_{ij}^2+\sum _{k=1}^{r_1}{q}_{ij,k}^1{q}_{ij,k}^2+\sum _{m=1}^{r_2}{s}_{ij,m}^1{s}_{ij,m}^2\right),\hfill \end{array}$$

where $P_{\delta }=\{p_{ij}^{\delta }\}$, $Q_{k,\delta }=\{q_{ij,k}^{\delta }\}$, $S_{m,\delta }=\{s_{ij,m}^{\delta }\}$, $i,j=\overline{1,n}$, $k=\overline{1,r_1}$, $m=\overline{1,r_2}$, $\delta =1,2$.

Definition 4.

The generalized gradient of the convex function ${\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]$ at the interior point $\left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\overline{\Omega }}_n^1$ is the tuple of matrices $\left[D_0,E_{1,0},\dots ,E_{r_1,0},F_{1,0},\dots ,F_{r_2,0}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\mathbb{R}}^{n\times n}$ such that for all $(P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2})\in {\prod }_{i=1}^{1+r_1+r_2}{\overline{\Omega }}_n^1$ we have

$$\begin{array}{cc}\hfill {\psi }_0& \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]-{\psi }_0\left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\hfill \\ \hfill & \ge \langle \left[D_0,E_{1,0},\dots ,E_{r_1,0},F_{1,0},\dots ,F_{r_2,0}),(P-P_0,Q_1-Q_{1,0},\dots ,Q_{r_1}-Q_{r_1,0},S_1-S_{1,0},\dots ,S_{r_2}-S_{r_2,0}\right]\rangle .\hfill \end{array}$$

Let $\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]$ be a linear matrix-valued operator that maps the tuple of matrices $\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\overline{\Omega }}_n^1$ into the $n(1+r_1+r_2)\times n(1+r_1+r_2)$ symmetric matrices $\Gamma$.

Denote by ${\Delta }_{ij}\in {\mathbb{R}}^{n\times n}$ the matrix in which the entries at positions $(i,j)$ and $(j,i)$ are units, and all the rest are zeroes.

Lemma 3.

The generalized gradient of the function ${\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]=-{\lambda }_{min}(\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right])$ at the interior point $\left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\overline{\Omega }}_n^1$ is the tuple of matrices $\left[D_0,E_{1,0},\dots ,E_{r_1,0},F_{1,0},\dots ,F_{r_2,0}\right]$, where $D_0=\{d_{ij}^0\}$, $E_{k,0}=\{e_{ij,k}^0\}$, $F_{m,0}=\{f_{ij,m}^0\}$, $i,j=\overline{1,n}$, $k=\overline{1,r_1}$, $m=\overline{1,r_2}$ such that

$$\begin{array}{cc}\hfill & d_{ij}^0=\left\{\begin{array}{c}-z_0^{\top }\Gamma ({\Delta }_{ij},{\Theta }_1,\dots ,{\Theta }_{r_1+r_2})z_0,ifi=j,\hfill \\ -\frac{1}{2}{z}_0^{\top }\Gamma ({\Delta }_{ij},{\Theta }_1,\dots ,{\Theta }_{r_1+r_2})z_0,ifi\ne j,\hfill \end{array}\right.\hfill \\ \hfill & e_{ij,k}^0=\left\{\begin{array}{c}-z_0^{\top }\Gamma (\Theta ,{\Theta }_1,\dots ,{\Theta }_{k-1},{\Delta }_{ij},{\Theta }_{k+1},\dots ,{\Theta }_{r_1},{\Theta }_{r_1+1},\dots ,{\Theta }_{r_1+r_2})z_0,ifi=j,\hfill \\ -\frac{1}{2}{z}_0^{\top }\Gamma (\Theta ,{\Theta }_1,\dots ,{\Theta }_{k-1},{\Delta }_{ij},{\Theta }_{k+1},\dots ,{\Theta }_{r_1},{\Theta }_{r_1+1},\dots ,{\Theta }_{r_1+r_2})z_0,ifi\ne j,\hfill \end{array}\right.\hfill \\ \hfill & f_{ij,m}^0=\left\{\begin{array}{c}-z_0^{\top }\Gamma (\Theta ,{\Theta }_1,\dots ,{\Theta }_{r_1},{\Theta }_{r_1+1},\dots ,{\Theta }_{m-1},{\Delta }_{ij},{\Theta }_{m+1},\dots ,{\Theta }_{r_1+r_2})z_0,ifi=j,\hfill \\ -\frac{1}{2}{z}_0^{\top }\Gamma (\Theta ,{\Theta }_1,\dots ,{\Theta }_{r_1},{\Theta }_{r_1+1},\dots ,{\Theta }_{m-1},{\Delta }_{ij},{\Theta }_{m+1},\dots ,{\Theta }_{r_1+r_2})z_0,ifi\ne j,\hfill \end{array}\right.\hfill \end{array}$$

(11)

where ${\Theta }_v$ is matrix Θ at position v,

$z_0$ is the unit eigenvector corresponding to ${\lambda }_{min}(\Gamma (P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}))$.

Proof. 

Consider for $z\in {\mathbb{R}}^{n(1+r_1+r_2)}$

$$\begin{array}{cc}\hfill & {\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]-{\psi }_0\left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\hfill \\ \hfill & =-\underset{\parallel z\parallel =1}{min}\{z^{\top }\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]z\}+\underset{\parallel z\parallel =1}{min}\{z^{\top }\Gamma \left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]z\}.\hfill \end{array}$$

Assume the first quadratic form reaches its minimal value at $z=z_1$, and the second one at $z=z_0$. Then subtracting and adding the expression $z_0^{\top }\Gamma (P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2})z_0$, we get

$$\begin{array}{cc}\hfill & {\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]-{\psi }_0\left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\hfill \\ \hfill & =-z_0^{\top }\left(\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]-\Gamma \left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\right)z_0\hfill \\ \hfill & +z_0^{\top }\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]z_0-z_1^{\top }\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]z_1.\hfill \end{array}$$

Since $z_0^{\top }\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]z_0\ge z_1^{\top }\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]z_1$, we have

$$\begin{array}{cc}\hfill & {\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]-{\psi }_0\left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\hfill \\ \hfill & \ge -z_0^{\top }\left(\Gamma \left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]-\Gamma \left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\right)z_0.\hfill \end{array}$$

Finally we use in the last inequality the representation of the matrices in the form

$$\begin{array}{cc}\hfill & P=\sum _{1\le i\le j\le n}{p}_{ij}{\Delta }_{ij},Q_k=\sum _{1\le i\le j\le n}{q}_{ij,k}{\Delta }_{ij},S_m=\sum _{1\le i\le j\le n}{s}_{ij,m}{\Delta }_{ij},\hfill \end{array}$$

and the linearity of $\Gamma$ with respect to $(P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2})$, which allows us to get the presentation of the generalized gradient in the form (11). ☐

When solving (10), we pass from a constrained problem to an unconstrained one. We define the penalty functions

$${\psi }_1\left(B\right)={\lambda }_{max}\left(B\right)-1,{\psi }_2\left(B\right)=-{\lambda }_{min}\left(B\right),B\in {\Omega }_n,$$

and the corresponding Lagrange function

$$\begin{array}{cc}\hfill & \mathcal{L}(P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2},u):={\psi }_0(P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2})\hfill \\ \hfill & +\sum _{\delta =1,2}\left(u_{P,\delta }{\psi }_{\delta }\left(P\right)+\sum _{k=1}^{r_1}{u}_{Q_k,\delta }{\psi }_{\delta }\left(Q_k\right)+\sum _{m=1}^{r_2}{u}_{S_m,\delta }{\psi }_{\delta }\left(S_m\right)\right),\hfill \end{array}$$

(12)

where $u=\{u_{P,1},u_{Q_1,1},\dots ,u_{Q_{r_1},1},u_{S_1,1},\dots ,u_{S_{r_2},1},u_{P,2},u_{Q_1,2},\dots ,u_{Q_{r_1},2},u_{S_1,2},\dots ,u_{S_{r_2},2}\}\in {\mathbb{R}}^{2(1+r_1+r_2)}$ are non-negative Lagrange multipliers.

Theorem 2.

Provided that the condition H1 holds, the function ${\psi }_0\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]$ attains its minimum at the point $\left[P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\overline{\Omega }}_n^1$ if and only if the point

$(P_0,Q_{1,0},\dots ,Q_{r_1,0},S_{1,0},\dots ,S_{r_2,0},u^0)$, where

$$\begin{array}{cc}\hfill u^0=& {(u_{P,1}^0,u_{Q_1,1}^0,\dots ,u_{Q_{r_1},1}^0,u_{S_1,1}^0,\dots ,u_{S_{r_2},1}^0,u_{P,2}^0,u_{Q_1,2}^0,\dots ,u_{Q_{r_1},2}^0,u_{S_1,2}^0,\dots ,u_{S_{r_2},2}^0)}^{\top },\hfill \end{array}$$

is a saddle point of the Lagrange function (12).

Proof. 

The objective function ${\psi }_0(·)$ and the constraint functions ${\psi }_1(·)$, ${\psi }_2(·)$ are convex for the matrices $P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\in {\Omega }_n$. It follows from the convexity of maximum eigenvalue and the concavity of minimum eigenvalue of a symmetric matrix (see Example 3.10 in [18]). When proving the convexity of the function ${\psi }_0$ with respect to $\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\Omega }_n$ we also use the linear dependence of the operator $\Gamma$ on $\left[P,Q_1,\dots ,Q_{r_1},S_1,\dots ,S_{r_2}\right]$.

Due to the Karush–Kuhn–Tucker conditions for convex problems it is left to show that Slater conditions hold [18] (page 244). By virtue the assumption H1 there exists a tuple of matrices $\left[\overline{P},{\overline{Q}}_1,\dots ,{\overline{Q}}_{r_1},{\overline{S}}_1,\dots ,{\overline{S}}_{r_2}\right]\in {\prod }_{i=1}^{1+r_1+r_2}{\Omega }_n\cap \mathrm{dom}\left({\psi }_0\right)$ such that

$${\psi }_{\delta }\left(\overline{P}\right)<0,{\psi }_{\delta }\left({\overline{Q}}_1\right)<0,\dots ,{\psi }_{\delta }\left({\overline{Q}}_{r_1}\right)<0,{\psi }_{\delta }\left({\overline{S}}_1\right)<0,\dots ,{\psi }_{\delta }\left({\overline{S}}_{r_2}\right)<0,\delta =1,2,$$

is satisfied, and the Slater condition is true, which concludes the proof. ☐

4. Conclusions

The work is devoted to modeling and investigation of the architecture design for the delayed recurrent neural network basing on the delayed differential equations. The usage of discrete and distributed delays makes it possible to model the calculation of the next states using internal memory, which corresponds to the artificial recurrent neural network architecture used in the field of deep learning.

The paper proposes a method for constructing an exponential estimate of the solution of a model of a recurrent neural network using the Lyapunov–Krasovskii functional. This estimate is reduced to solving the corresponding linear matrix inequality. This is the most costly operation in terms of the computational complexity, however, this is where we can apply the optimization approach to find the optimal set of matrices from the viewpoint of the exponential estimate.

In contrast to the indirect method of constructing exponential estimates, which was proposed in previous works [12,13], in this study, the method based on the Lyapunov–Krasovskii functional allows the optimization of the estimate within the compact domain of positive definite matrices.

To this end, the concept of a generalized gradient of a convex function on a set of positive definite matrices is introduced. The constructive form of the generalized gradient for the minimal eigenvalue of the matrix $\Gamma$ is presented. The Lagrange function for the unconditional optimization problem is constructed. In this case, the search for the optimal exponential estimate is reduced to finding the saddle point of the Lagrange function.