This paper introduces a novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network—it ensures all layers are connected to one another. This DNN, called Fractional-DNN, can be viewed as a time-discretization of a fractional in time non-linear ordinary differential equation (ODE). The learning problem then is a minimization problem subject to that fractional ODE as constraints. We emphasize that an analogy between the existing DNN and ODEs, with standard time derivative, is well-known by now. The focus of our work is the Fractional-DNN. Using the Lagrangian approach, we provide a derivation of the backward propagation and the design equations. We test our network on several datasets for classification problems. Fractional-DNN offers various advantages over the existing DNN. The key benefits are a significant improvement to the vanishing gradient issue due to the memory effect, and better handling of nonsmooth data due to the network’s ability to approximate non-smooth functions.

本文介绍了一种用于深度神经网络（DNN）的新颖算法框架，该框架以数学上严格的方式允许我们将历史（或内存）整合到网络中-它确保所有层都相互连接。这种DNN（称为分数DNN）可以看作是时间非线性分数阶微分方程（ODE）的时间离散。于是，学习问题是一个最小化问题，该最小化问题受到该分数ODE的约束。我们强调，以标准时间导数在现有DNN和ODE之间进行类比现在是众所周知的。我们的工作重点是分数DNN。使用拉格朗日方法，我们提供了反向传播和设计方程的推导。我们在几个数据集中测试我们的网络是否存在分类问题。相对于现有DNN，分数DNN具有各种优势。关键好处是由于内存效应而对消失梯度问题有了重大改进，并且由于网络具有近似非平滑函数的能力，因此可以更好地处理非平滑数据。