Random recurrent neural networks with delays

https://doi.org/10.1016/j.jde.2020.06.008Get rights and content

Abstract

An infinite lattice model of a recurrent neural network with random connection strengths between neurons is developed and analyzed. To incorporate the presence of various type of delays in the neural networks, both discrete and distributed time varying delays are considered in the model. For the existence of random pullback attractors and periodic attractors, the nonlinear terms of the resulting system are not expected to be Lipschitz continuous, but only satisfy a weaker continuity assumption along with growth conditions, under which the uniqueness of the underlying Cauchy problem may not hold. Then after extending the concept and theory of monotone multi-valued semiflows to the random context, the structure of random pullback attractors with or without periodicity is investigated. In particular, the existence and stability properties of extremal random complete trajectories are studied.

Introduction

Recurrent neural networks (RNNs) arise in a wide range of applications such as classification, combinatorial optimization, parallel computing, signal processing and pattern recognition (see, e.g. [13], [15], [20], [23], [30], [31], [34]). The simplest form of RNNs can be modeled by a system of ordinary differential equationsdxi(t)dt=xi(t)+j=1Nλijfj(xj(t))+Ji,i=1,,N, where xi(t) is the state variable representing the potential of the ith neuron at time t, fj is the neuron activation function of the jth neuron, λij is the connection strength or the coupling weight between the ith and the jth neuron, and Ji is the external forcing on the ith neuron. The aim of this work is three-fold: (1) to take into account uncertainties in the connection structure among neurons, (2) to incorporate the presence of various types of time delays in RNNs, and (3) to investigate the dynamics of the RNN (1.1) when its size becomes increasingly large, i.e., N.

First notice that the N×N matrix (λij)i,j=1,,N characterizes the connection structure between neurons within the network. Therefore when a neural network is under a random environment that affects the interactions between neurons, λij may no longer be constant for each pair of i and j. To take into account the randomness in RNNs, in this work we formulate the connection weight parameter λij as a random process λij(θtω), where θtω is a canonical representation of the driving noise system. Second, due to the finite switching speed of neurons and amplifiers, time delays commonly occur in neural networks. It is well known that time delays may affect the stability of a dynamical system and eventually lead to complex dynamic behavior, so they have been widely considered in neural networks. In particular, neural networks with discrete time delays have been studied by [21], [23], [26], [27], [31], [36], [42], amongst others. Moreover, since noninstantaneous signal propagation may not be suitably modeled by discrete delay, in this work we also consider continuously distributed delays. Last, we are interested in dynamical behavior of the RNN (1.1) when its size approaches infinity. This is of particular importance while a neural network arises as the discrete counterpart of a continuum model and thus is essentially infinite dimensional. Here we assume that the RNN has infinite number of neurons, but each neuron j interacts with finite number of neurons jN,,j+N in its neighborhood on the network. In addition, we consider a time dependent external forcing Ji(t) instead of constant forcing.

Summarizing the above we reach the following lattice model for random RNNs with discrete and distributed varying delays:dxi(t,ω)dt=xi(t,ω)+j=iNi+Nλij(θtω)Fj(t,xjt(ω))+Ji(t),iZ, where xjt(ω) denotes a segment of xj(t,ω) for each ωΩ, and for each jZ the activation function Fj is composed of an instantaneous activation fj, an activation gj with discrete time dependent delays hˆ(t), and an activation ηj with distributive delays with the bound h˜(t):Fj(t,xjt(ω))=afj(xj(t,ω))+bgj(xj(thˆ(t),ω))+ch˜(t)0ηj(xj(t+s,ω))ds,ωΩ. Throughout this work it is assumed that both hˆ(t) and h˜(t) belong to C(R;[0,h]) for some h>0. The goal of this work is to study detailed long term dynamics of the random RNN lattice system (1.2) under the initial conditionxi(t)=ϕi(tτ),t[τh,τ],iZ.

There exists a rich literature on mathematical studies of neural networks (NNs) with randomness and time delays. For example, exponential stability of stochastic NNs with constant or time-varying delays has been studied in [12], [22], [23], [25], [28], [36], and the pth moment exponential stability of stochastic recurrent NNs with time-varying delays was investigated in [33]. Asymptotic stability of stochastic NNs with discrete and distributed delays has been developed, see, e.g., [2], [3], [35] for NNs with Markovian jumping parameters, [18] for NNs with Brownian motion, [31] for NNs with impulsive effects, and [4], [26] for NNs with infinite delays. Robust analysis for stochastic NNs with time-varying delays can be found in [27], [42]. However, to the best of our knowledge, there is no existing result on random pullback attractors or extremal random complete trajectories for random lattice RNNs such as (1.2).

This work consists of two major parts. The first part is devoted to the existence of random pullback attractors and periodic attractors for the random lattice RNN system with delays (1.2). The second part focuses on the development of new theory on monotone multi-valued non-compact random dynamical systems, that is then utilized to analyze extremal random complete and periodic trajectories for the system (1.2). The manuscript is organized as follows. In Section 2 we prove the existence of solutions to the random lattice system (1.2). In Section 3 we first present some basic concepts and theories on multi-valued non-compact random dynamical systems, and establish the existence of random pullback attractors and periodic attractors for random dynamical system generated by the solutions to the random lattice system (1.2). In Section 4 we first generalize existing results on monotone multi-valued semiflows to the random context, and then use them to further investigate the structure of pullback attractors. In particular, the existence and stability properties of extremal random complete and periodic trajectories in the random pullback attractors are studied. In the end a summary of this work is provided in Section 5.

Section snippets

Basic properties of solutions

In this section we study the existence of solutions to the random RNN lattice system (1.2). To that end we will first formulate the lattice system (1.2) as a functional differential equation on an appropriate Banach space. Consider the separable Hilbert space of square summable bi-infinite real-valued sequences2={x=(xi)iZ:iZxi2<} equipped with the inner product and the normx,y=iZxiyi,x=(iZxi2)12,x=(xi)iZ,y=(yi)iZ.

Denote by Ch=C([h,0];2) the set of continuous functions from [h,

Random pullback attractors and periodic attractors

This section is devoted to the existence of random pullback attractors and periodic attractors for system (1.2). First let us recall some basic concept of multi-valued non compact random dynamical systems and existing results on the existence of random pullback attractors for such systems.

Extremal complete quasi-trajectories

In this section we further investigate the structure of the pullback attractor A for the multi-valued cocycle Φ defined in (3.2). In particular, we will show that Φ possesses extremal complete quasi-trajectories defined as below.

Definition 4.1

([37], [40]) Let D be a collection of some families of nonempty subsets of X and let Φ:R+×Q×Ω×XP(X) be a multi-valued cocycle on X over the parametric dynamical systems (Q,ζ) and (Ω,F,P,θ).

  • (1)

    A mapping ψ:R×Q×ΩX is called a complete trajectory (orbit) of Φ ifψ(t+τ,q,ω)Φ(t

Summary

In this work we studied a lattice dynamical system that models recurrent neural networks with random connection strengths between neurons and time delays. First we reformulated the lattice dynamical system and a delay differential equation on 2, and proved the existence of solutions. We then studied the existence of random pullback attractors and periodic attractors, as well as extremal random complete and periodic trajectories in the random pullback attractors. One technical challenge is that

References (42)

  • J.C. Robinson et al.

    Pullback attractors and extremal complete trajectories for non-autonomous reaction-diffusion problems

    J. Differ. Equ.

    (2007)
  • Y.H. Sun et al.

    Pth moment exponential stability of stochastic recurrent neural networks with time-varying delays

    Nonlinear Anal., Real World Appl.

    (2007)
  • B.X. Wang

    Sufficient and necessary criteria for existence of pullback attractors for non-compact random dynamical systems

    J. Differ. Equ.

    (2012)
  • Y.J. Wang et al.

    Pullback attractors for multi-valued non-compact random dynamical systems generated by reaction-diffusion equations on an unbounded domain

    J. Differ. Equ.

    (2015)
  • J.P. Aubin et al.

    Set-Valued Analysis

    (1990)
  • M.S. Ali et al.

    Stochastic stability of discrete-time uncertain recurrent neural networks with Markovian jumping and time-varying delays

    Math. Comput. Model.

    (2011)
  • M.S. Ali

    Stochastic stability of uncertain recurrent neural networks with Markovian jumping parameters

    Acta Math. Sci. Ser. B (Engl. Ed.)

    (2015)
  • P.W. Bates et al.

    Attractors for lattice dynamical systems

    Int. J. Bifurc. Chaos Appl. Sci. Eng.

    (2001)
  • T. Caraballo et al.

    Asymptotic behaviour of monotone multi-valued dynamical systems

    Dyn. Syst.

    (2005)
  • T. Caraballo et al.

    Non-autonomous and random attractors for delay random semilinear equations without uniqueness

    Discrete Contin. Dyn. Syst.

    (2008)
  • T. Caraballo et al.

    Pullback attractors of stochastic lattice dynamical systems with a multiplicative noise and non-Lipschitz nonlinearities

  • Cited by (24)

    • Hopf bifurcation mechanism analysis in an improved cortex-basal ganglia network with distributed delays: An application to Parkinson's disease

      2023, Chaos, Solitons and Fractals
      Citation Excerpt :

      In recent years, more and more research work focuses on heterogeneous delays in neuronal systems due to they are more feasible in practical applications [22–30]. For example, in order to reflect that the recent state of neurons has more influence on the current state of the network than in the distant past, distributed delay has attracted more and more attentions [31–38]. Distributed delay is continuously distributed over an infinite interval, which can be transformed into discrete delays when the delay kernel is taken as a Dirac delta function; therefore, distributed delay is more general in application than discrete delay [39–42].

    • Stability analysis of hybrid stochastic delayed Cohen-Grossberg neural networks with Lévy noise and Markov switching

      2022, Journal of the Franklin Institute
      Citation Excerpt :

      From then on the CGNNs models have become a hotly debated topic and broadly studied in the literatures [4–7], because it is widely applicable in parallel computing, classification, associative memory, and some optimization problems. There are some special cases of CGNNs, such as recurrent neural networks [8,9], Hopfield neural networks [10,11]. A plenty of results have been obtained around the basic theories and applications of them.

    • Existence, exponential mixing and convergence of periodic measures of fractional stochastic delay reaction-diffusion equations on R<sup>n</sup>

      2022, Journal of Differential Equations
      Citation Excerpt :

      Such systems can be described by differential equations with delay [17–19]. The stability and random attractors of stochastic delay equations have been investigated in [20–24]. The invariant measures of stochastic delay systems have been studied in [25–30] for finite dimensional equations and in [31,32] for infinite dimensional lattice equations.

    • NON-AUTONOMOUS STOCHASTIC LATTICE SYSTEMS WITH MARKOVIAN SWITCHING

      2023, Discrete and Continuous Dynamical Systems- Series A
    View all citing articles on Scopus

    This work was partially supported by NSF of China (Grant No. 41875084, 11571153) and the Fundamental Research Funds for the Central Universities (Grant No. lzujbky-2018-it58, lzujbky-2018-ot03).

    View full text