Universal statistics of Fisher information in deep neural networks: mean field approach^*

This study analyzes the FIM of deep networks with random weights and biases, which are widely used settings to analyze the phenomena of DNNs [2–14]. First, we analytically obtain novel statistics of the FIM, namely, the mean (theorem 1), variance (theorem 3), and maximum of eigenvalues (theorem 4). These are universal among a wide class of shallow and deep networks with various activation functions. These quantities can be obtained from simple iterative computations of macroscopic variables. To our surprise, the mean of the eigenvalues asymptotically decreases with an order of O(1/M) in the limit of a large network width M, while the variance takes a value of O(1), and the maximum eigenvalue takes a huge value of O(M) by using the O(⋅) order notation. Since the eigenvalues are non-negative, these results mean that most of the eigenvalues are close to zero, but the edge of the eigenvalue distribution takes a huge value. Because the FIM defines the Riemannian metric of the parameter space, the derived statistics imply that the space is locally flat in most dimensions, but strongly distorted in others. In addition, because the FIM also determines the local shape of a loss landscape, the landscape is also expected to be locally flat while strongly distorted.

Furthermore, to confirm the potential usage of the derived statistics, we show some exercises. One is on the Fisher–Rao norm [22] (theorem 5). This norm was originally proposed to connect the flatness of a parameter space to the capacity measure of generalization ability. We evaluate the Fisher–Rao norm by using an indicator of the small eigenvalues, κ₁ in theorem 1. Another exercise is related to the more practical issue of determining the size of the learning rate necessary for the steepest descent gradient to converge. We demonstrate that an indicator of the huge eigenvalue, κ₂ in theorem 4, enables us to roughly estimate learning rates that make the gradient method converge to global minima (theorem 7). We expect that it will help to alleviate the dependence of learning rates on heuristic settings.

Despite its importance in statistics and machine learning, study on the FIM for neural networks has been limited so far. This is because layer-by-layer nonlinear maps and huge parameter dimensions make it difficult to take analysis any further. Degeneracy of the eigenvalues of the FIM has been found in certain parameter regions [23]. To understand the loss landscape, Pennington and Bahri [5] has utilized random matrix theory and obtained the spectrum of FIM and Hessian under several assumptions, although the analysis is limited to special types of shallow networks. In contrast, this paper is the first attempt to apply the mean field approach, which overcomes the difficulties above and enables us to identify universal properties of the FIM in various types of DNNs.

LeCun et al [24] investigated the Hessian of the loss, which coincides with the FIM at zero training error, and empirically reported that very large eigenvalues exist, i.e. 'big killers', which affects the optimization (discussed in section 4.2). The eigenvalue distribution peaks around zero while its tail is very long; this behavior has been empirically known for decades [25], but its theoretical evidence and evaluation have remained unsolved as far as we know. Therefore, our theory provides novel theoretical evidence that this skewed eigenvalue distribution and its huge maximum appear universally in DNNs.

The theoretical tool we use here is known as the mean field theory of deep networks [3, 4, 10–14] as briefly overviewed in section 2.4. This method has been successful in analyzing neural networks with random weights under a large width limit and in explaining the performance of the models. In particular, it quantitatively coincides with experimental results very well and can predict appropriate initial values of parameters for avoiding the vanishing or explosive gradient problems [4]. This analysis has been extended from fully connected deep networks to residual [11] and convolutional networks [14]. The evaluation of the FIM in this study is also expected to be extended to such cases.

Recently, some advances have been made in the understanding of FIM's eigenvalue statistics. While the FIM's maximum eigenvalue takes a huge value and acts as an outlier in naive settings, we can alleviate it under batch normalization [26]. Karakida et al [27] has analyzed the FIM corresponding to the cross-entropy loss with softmax output, and also made some remarks on the connection between the FIM and neural tangent kernel.

We focus on the FIM of neural network models, which previous works have developed and is commonly used [28–33]. It is defined by

$$\begin{equation}F=\mathrm{E}\left[{\nabla }_{\theta }\enspace \mathrm{log}\enspace p\left(x,y;\theta \right){\nabla }_{\theta }\enspace \mathrm{log}\enspace p{\left(x,y;\theta \right)}^{\mathrm{T}}\right],\end{equation} \tag{ 1 }$$

where the statistical model is given by p(x, y; θ) = p(y|x; θ)p(x). The output model is given by $p\left(y\vert x;\theta \right)=\mathrm{exp}\left(-{\Vert}y-{f}_{\theta }\left(x\right){{\Vert}}^{2}/2\right)/\sqrt{2\pi }$, where f_θ (x) is the network output parameterized by θ and ||⋅|| is the Euclidean norm. The q(x) is an input distribution. The expectation E[⋅] is taken over the input-output pairs (x, y) of the joint distribution p(x, y; θ). This FIM is transformed into $F={\sum }_{k=1}^{C}\mathrm{E}\left[{\nabla }_{\theta }{f}_{\theta ,k}\left(x\right){\nabla }_{\theta }{f}_{\theta ,k}{\left(x\right)}^{\mathrm{T}}\right]$, where f_θ,k is the kth entry of the output (k = 1, ..., C). When T training samples x(t) (t = 1, ..., T) are available, the expectation can be replaced by the empirical mean. This is known as the empirical FIM and often appears in practice [29–33]:

$$\begin{equation}F=\frac{1}{T}\sum _{t=1}^{T}\sum _{k=1}^{C}{\nabla }_{\theta }{f}_{\theta ,k}\left(t\right){\nabla }_{\theta }{f}_{\theta ,k}{\left(t\right)}^{\mathrm{T}}.\end{equation} \tag{ 2 }$$

This study investigates the above empirical FIM for arbitrary T. It converges to the expected FIM as T → ∞. Although the form of the FIM changes a bit in other statistical models (i.e. softmax outputs), these differences are basically limited to the multiplication of activations in the output layer [32]. Our framework can be straightforwardly applied to such cases.

The FIM determines the asymptotic accuracy of the estimated parameters, as is known from a fundamental theorem of statistics, namely, the Cramér–Rao bound [35]. Below, we summarize a more intuitive understanding of the FIM from geometric views.

Information geometric view. Let us define an infinitesimal squared distance dr², which represents the Kullback–Leibler divergence between the statistical model p(x, y; θ) and p(x, y; θ + dθ) against a perturbation dθ. It is given by

$$\begin{equation}\mathrm{d}{r}^{2}{:=}\mathrm{K}\mathrm{L}\left(p\left(x,y;\theta \right){\Vert}p\left(x,y;\theta +\mathrm{d}\theta \right)\right)=\mathrm{d}{\theta }^{\mathrm{T}}F\mathrm{d}\theta .\end{equation} \tag{ 3 }$$

It means that the parameter space of a statistical model forms a Riemannian manifold and the FIM works as its Riemannian metric, as is known in information geometry [35]. This quadratic form is equivalent to the robustness of a deep network: E[||f_θ+dθ (t) − f_θ (t)||²] = dθ^T Fdθ. Insights from information geometry have led to the development of natural gradient algorithms [31–33] and, recently, a capacity measure based on the Fisher–Rao norm [22].

Loss landscape view. The empirical FIM (2) determines the local landscape of the loss function around the global minimum. Suppose we have a squared loss function $E\left(\theta \right)=\left(1/2T\right){\sum }_{t}^{T}{\Vert}y\left(t\right)-{f}_{\theta }\left(t\right){{\Vert}}^{2}$. The FIM is related to the Hessian of the loss function, H := ∇_θ ∇_θ E(θ), in the following way:

$$\begin{equation}H=F-\frac{1}{T}\sum _{t}^{T}\sum _{k}^{C}\left({y}_{k}\left(t\right)-{f}_{\theta ,k}\left(t\right)\right){\nabla }_{\theta }{\nabla }_{\theta }{f}_{\theta ,k}\left(t\right).\end{equation} \tag{ 4 }$$

The Hessian coincides with the FIM when the parameter converges to the global minimum by learning, that is, the true parameter θ* from which the teacher signal y(t) is generated by $y\left(t\right)={f}_{{\theta }^{{\ast}}}\left(t\right)$ or, more generally, with noise (i.e. $y\left(t\right)={f}_{{\theta }^{{\ast}}}\left(t\right)+{\varepsilon }_{t}$, where ɛ_t denotes zero-mean Gaussian noise) [29]. In the literature on deep learning, its eigenvectors whose eigenvalues are close to zero locally compose flat minima, which leads to better generalization empirically [19, 22]. Modifying the loss function with the FIM has also succeeded in overcoming the catastrophic forgetting [36].

Note that the information geometric view tells us more than the loss landscape. While the Hessian (4) assumes the special teacher signal, the FIM works as the Riemannian metric to arbitrary teacher signals.

This study investigates a fully connected feedforward neural network. The network consists of one input layer with M₀ units, L − 1 hidden layers (L ⩾ 2) with M_l units per hidden layer (l = 1, 2, ..., L − 1), and one output layer with M_L units:

$$\begin{equation}{u}_{i}^{l}=\sum _{j=1}^{{M}_{l-1}}{W}_{ij}^{l}{h}_{j}^{l-1}+{b}_{i}^{l},\qquad {h}_{i}^{l}=\phi \left({u}_{i}^{l}\right).\end{equation} \tag{ 5 }$$

This study focuses on the case of linear outputs, that is, ${f}_{\theta ,k}\left(x\right)={h}_{k}^{L}={u}_{k}^{L}$. We assume that the activation function ϕ(x) and its derivative ϕ'(x) := dϕ(x)/dx are square-integrable functions on a Gaussian measure. A wide class of activation functions, including the sigmoid-like and (leaky-) ReLU functions, satisfy these conditions. Different layers may have different activation functions. Regarding the network width, we set M_l = α_l M(l ⩽ L − 1) and consider the limiting case of large M with constant coefficients α_l . This study mainly focuses on the case where the number of output units is given by a constant M_L = C. The higher-dimensional case of C = O(M) is argued in section 4.3.

The FIM (2) of a deep network is computed by the chain rule in a manner similar to that of the backpropagation algorithm:

$$\begin{equation}\frac{\partial {f}_{\theta ,k}}{\partial {W}_{ij}^{l}}={\delta }_{k,i}^{l}\phi \left({u}_{j}^{l-1}\right),\end{equation} \tag{ 6 }$$

$$\begin{equation}{\delta }_{k,i}^{l}={\phi }^{\prime }\left({u}_{i}^{l}\right)\sum _{j}{\delta }_{k,j}^{l+1}{W}_{ji}^{l+1},\qquad {\delta }_{k,k}^{L}={\phi }^{\prime }\left({u}_{k}^{L}\right),\end{equation} \tag{ 7 }$$

where ${\delta }_{k,i}^{l}{:=}\partial {f}_{\theta ,k}/\partial {u}_{i}^{l}$ for k = 1, ..., C. To avoid the complicated notation, we omit the index of the output unit, i.e. ${\delta }_{i}^{l}={\delta }_{k,i}^{l}$, in the following.

The parameter set $\theta =\left\{{W}_{ij}^{l},{b}_{i}^{l}\right\}$ is an ensemble generated by

$$\begin{equation}{W}_{ij}^{l}\sim \mathcal{N}\left(0,{\sigma }_{{w}^{l}}^{2}/{M}_{l-1}\right),\qquad {b}_{i}^{l}\sim \mathcal{N}\left(0,{\sigma }_{{b}^{l}}^{2}\right),\end{equation} \tag{ 8 }$$

and then fixed, where $\mathcal{N}\left(0,{\sigma }^{2}\right)$ denotes a Gaussian distribution with zero mean and variance σ², and we set ${\sigma }_{{w}^{l}}{ >}0$ and ${\sigma }_{{b}^{l}}{ >}0$. To avoid complicated notation, we set them uniformly as ${\sigma }_{{w}^{l}}^{2}={\sigma }_{w}^{2}$ and ${\sigma }_{{b}^{l}}^{2}={\sigma }_{b}^{2}$, but they can easily be generalized. It is essential to normalize the variance of the weights by M in order to normalize the output ${u}_{i}^{l}$ to O(1). This setting is similar to how parameters are initialized in practice [37]. We also assume that the input samples ${h}_{i}^{0}\left(t\right)={x}_{i}\left(t\right)\enspace \enspace \left(t=1,\dots ,T\right)$ are generated in an i.i.d. manner from a standard Gaussian distribution: ${x}_{i}\left(t\right)\sim \mathcal{N}\left(0,1\right)$. We focus here on the Gaussian case for simplicity, although we can easily generalize it to other distributions with finite variances.

Let us remark that the above random connectivity is a common setting widely supposed in theories. Analyzing such a network can be regarded as the typical evaluation [2, 3, 5]. It is also equal to analyzing the network randomly initialized [4, 20]. The random connectivity is often assumed in the analysis of optimization as a true parameter of the networks, that is, the global minimum of the parameters [21, 38].

On neural networks with random connectivity, taking a large width limit, we can analyze the asymptotic behaviors of the networks. Recently, this asymptotic analysis is referred to as the mean field theory of deep networks, and we follow the previously reported notations and terminology [3, 4, 11, 12].

First, let us introduce the following variables for feedforward signal propagations: ${\hat{q}}^{l}{:=}{\sum }_{\;i}{h}_{i}^{l}{\left(t\right)}^{2}/{M}_{l}$ and ${\hat{q}}_{st}^{l}{:=}{\sum }_{\;i}{h}_{i}^{l}\left(s\right){h}_{i}^{l}\left(t\right)/{M}_{l}$. In the context of deep learning, these variables have been utilized to explain the depth to which signals can sufficiently propagate. The variable ${\hat{q}}_{st}^{l}$ is the correlation between the activations for different input samples x(s) and x(t) in the lth layer. Under the large M limit, these variables are given by integration over Gaussian distributions because the pre-activation ${u}_{l}^{i}$ is a weighted sum of independent random parameters and the central limit theorem is applicable [2–4]:

$$\begin{equation}{\hat{q}}^{l+1}=\int Du{\phi }^{2}\left(\sqrt{{q}^{l+1}}u\right),\qquad {q}^{l+1}={\sigma }_{w}^{2}{\hat{q}}^{l}+{\sigma }_{b}^{2},\end{equation} \tag{ 9 }$$

$$\begin{equation}{\hat{q}}_{st}^{l+1}={I}_{\phi }\left[{q}^{l+1},{q}_{st}^{l+1}\right],\qquad {q}_{st}^{l+1}={\sigma }_{w}^{2}{\hat{q}}_{st}^{l}+{\sigma }_{b}^{2},\end{equation} \tag{ 10 }$$

with ${\hat{q}}^{0}=1$ and ${\hat{q}}_{st}^{0}=0$ (l = 0, ..., L − 1). We can generalize the theory to unnormalized data with ${\hat{q}}^{0}\ne 0$ and ${\hat{q}}_{st}^{0}\ne 0$, just by substituting them into the recurrence relations. The notation $Du=\mathrm{d}u\enspace \mathrm{exp}\left(-{u}^{2}/2\right)/\sqrt{2\pi }$ means integration over the standard Gaussian density. Here, the notation I_⋅[⋅, ⋅] represents the following integral: ${I}_{\phi }\left[a,b\right]=\int D{z}_{1}D{z}_{2}\phi \left(\sqrt{a}{z}_{1}\right)\phi \left(\sqrt{a}\left(c{z}_{1}+\sqrt{1-{c}^{2}}{z}_{2}\right)\right)$ with c = b/a. The ${q}_{st}^{l}$ is linked to the compositional kernel and utilized as the kernel of the Gaussian process [39].

Next, let us introduce variables for backpropagated signals: ${\tilde {q}}^{l}{:=}{\sum }_{i}{\;\delta }_{i}^{l}{\left(t\right)}^{2}$ and ${\tilde {q}}_{st}^{l}{:=}{\sum }_{i}{\;\delta }_{i}^{l}\left(s\right){\delta }_{i}^{l}\left(t\right)$. Note that they are defined not by averages but by the sums. They remain O(1) because of C = O(1). ${\tilde {q}}_{st}^{l}$ is the correlation of backpropagated signals. To compute these quantities, the previous studies assumed the following:

Assumption 1 (Schoenholz et al [4]). On the evaluation of the variables ${\tilde {q}}^{l}$ and ${\tilde {q}}_{st}^{l}$, one can use a different set of parameters, θ for the forward chain (5) and θ' for the backpropagated chain (7), instead of using the same parameter set θ in both chains.

This assumption makes the dependence between $\phi \left({u}_{i}^{l}\right)$ (or ${\phi }^{\prime }\left({u}_{i}^{l}\right)$) and ${\delta }_{j}^{l+1}$, which share the same parameter set, very weak, and one can regard it as independent. It enables us to apply the central limit theorem to the backpropagated chain (7). Thus, the previous studies [4, 7, 11, 12] derived the following recurrence relations (l = 0, ..., L − 1):

$$\begin{equation}{\tilde {q}}^{l}={\sigma }_{w}^{2}{\tilde {q}}^{l+1}\int Du{\left[{\phi }^{\prime }\left(\sqrt{{q}^{l}}u\right)\right]}^{2},\end{equation} \tag{ 11 }$$

$$\begin{equation}{\tilde {q}}_{st}^{l}={\sigma }_{w}^{2}{\tilde {q}}_{st}^{l+1}{I}_{{\phi }^{\prime }}\left[{q}^{l},{q}_{st}^{l}\right],\end{equation} \tag{ 12 }$$

with ${\tilde {q}}^{L}={\tilde {q}}_{st}^{L}=1$ because of the linear outputs. The previous works confirmed excellent agreements between the above equations and experiments. In this study, we also adopt the above assumption and use the recurrence relations.

The variables (${\hat{q}}^{l},{\tilde {q}}^{l},{\hat{q}}_{st}^{l},{\tilde {q}}_{st}^{l}$) depend only on the variance parameters ${\sigma }_{w}^{2}$ and ${\sigma }_{b}^{2}$, not on the unit indices. In that sense, they are referred to as macroscopic variables (a.k.a. order parameters in statistical physics). The recurrence relations for the macroscopic variables simply require L iterations of one- and two-dimensional numerical integrals. Moreover, we can obtain their explicit forms for some activation functions (such as the error function, linear, and ReLU; see appendix B).

The FIM is a P × P matrix, where P represents the total number of parameters. First, we compute the arithmetic mean of the FIM's eigenvalues as ${m}_{\lambda }{:=}{\sum }_{i=1}^{P}{\lambda }_{i}/P$. We find a hidden relation between the macroscopic variables and the statistics of FIM:

Theorem 1. Suppose that assumption 1 holds. In the limit of M ≫ 1, the mean of the FIM's eigenvalues is given by

$$\begin{equation}{m}_{\lambda }=C\frac{{\kappa }_{1}}{M},\qquad {\kappa }_{1}{:=}\sum _{l=1}^{L}\enspace \frac{{\alpha }_{l-1}}{\alpha }{\tilde {q}}^{l}{\hat{q}}^{l-1},\end{equation} \tag{ 13 }$$

where $\alpha {:=}{\sum }_{l=1}^{L-1}{\alpha }_{l}{\alpha }_{l-1}$. The macroscopic variables ${\hat{q}}^{l}$ and ${\tilde {q}}^{l}$ can be computed recursively, and notably m_λ is O(1/M).

This is obtained from a relation m_λ = Trace(F)/P (detailed in appendix A.1). The coefficient κ₁ is a constant not depending on M, so it is O(1). It is easily computed by L iterations of the layer-wise recurrence relations (9) and (11).

Because the FIM is a positive semi-definite matrix and its eigenvalues are non-negative, this theorem means that most of the eigenvalues asymptotically approach zero when M is large. Recall that the FIM determines the local geometry of the parameter space. The theorem suggests that the network output remains almost unchanged against a perturbation of the parameters in many dimensions. It also suggests that the shape of the loss landscape is locally flat in most dimensions.

Furthermore, by using Markov's inequality, we can prove that the number of larger eigenvalues is limited, as follows:

Corollary 2. Let us denote the number of eigenvalues satisfying λ ⩾ k by N(λ ⩾ k) and suppose that assumption 1 holds. For a constant k > 0, N(λ ⩾ k) ⩽  min {ακ₁ CM/k, CT} holds in the limit of M ≫ 1.

The proof is shown in appendix A.2. When T is sufficiently small, we have a trivial upper bound N(λ ⩾ k) ⩽ CT and the number of non-zero eigenvalue is limited. The corollary clarifies that even when T becomes large, the number of eigenvalues whose values are O(1) is O(M) at most, and still much smaller than the total number of parameters P.

Next, let us consider the second moment ${s}_{\lambda }{:=}{\sum }_{i=1}^{P}{\lambda }_{i}^{2}/P$. We now demonstrate that s_λ can be computed from the macroscopic variables:

Theorem 3. Suppose that assumption 1 holds. In the limit of M ≫ 1, the second moment of the FIM's eigenvalues is

$$\begin{equation}{s}_{\lambda }=C\alpha \left(\frac{T-1}{T}{\kappa }_{2}^{2}+\frac{1}{T}{\kappa }_{1}^{2}\right),\end{equation} \tag{ 14 }$$

$$\begin{equation}{\kappa }_{2}{:=}\sum _{l=1}^{L}\frac{{\alpha }_{l-1}}{\alpha }{\tilde {q}}_{st}^{l}{\hat{q}}_{st}^{l-1}.\end{equation} \tag{ 15 }$$

The macroscopic variables ${\hat{q}}_{st}^{l}$ and ${\tilde {q}}_{st}^{l}$ can be computed recursively, and s_λ is O(1) ³ .

The proof is shown in appendix A.3.

From theorems 1 and 3, we can conclude that the variance of the eigenvalue distribution, ${s}_{\lambda }-{m}_{\lambda }^{2}$, is O(1). Because the mean m_λ is O(1/M) and most eigenvalues are close to zero, this result means that the edge of the eigenvalue distribution takes a huge value.

As we have seen so far, the mean of the eigenvalues is O(1/M), and the variance is O(1). Therefore, we can expect that at least one of the eigenvalues must be huge. Actually, we can show that the maximum eigenvalue (that is, the spectral norm of the FIM) increases in the order of O(M) as follows.

Theorem 4. Suppose that assumption 1 holds. In the limit of M ≫ 1, the maximum eigenvalue of the FIM is

$$\begin{equation}{\lambda }_{\mathrm{max}}=\alpha \left(\frac{T-1}{T}{\kappa }_{2}+\frac{1}{T}{\kappa }_{1}\right)M.\end{equation} \tag{ 16 }$$

The λ_max is derived from the dual matrix F* (detailed in appendix A.4). If we take the limit T → ∞, we can characterize the quantity κ₂ by the maximum eigenvalue as λ_max = ακ₂ M. Note that λ_max is independent of C. When C = O(M), it may depend on C, as shown in section 3.4.

This theorem suggests that the network output changes dramatically with a perturbation of the parameters in certain dimensions and that the local shape of the loss landscape is strongly distorted in that direction. Here, note that λ_max is proportional to α, which is the summation over L terms. This means that, when the network becomes deeper, the parameter space is more strongly distorted.

We confirmed the agreement between our theory and numerical experiments, as shown in figure 1. Three types of deep networks with parameters generated by random connectivity (8) were investigated: tanh, ReLU, and linear activations (L = 3, α_l = C = 1). The input samples were generated using i.i.d. Gaussian samples, and T = 10². When P > T, we calculated the eigenvalues by using the dual matrix F* (defined in appendix A.3) because F* is much smaller and its eigenvalues are easy to compute. The theoretical values of m_λ , s_λ and λ_max agreed very well with the experimental values in the large M limit. We could predict m_λ even for small M. In addition, in appendix C.1, we also show the results of experiments with fixed M and changing T. The theoretical values coincided with the experimental values very well for any T as the theorems predict.

Zoom In Zoom Out Reset image size

Recently, Liang et al [22] proposed the Fisher–Rao norm for a capacity measure of generalization ability:

$$\begin{equation}{\Vert}\theta {{\Vert}}_{FR}={\theta }^{\mathrm{T}}F\theta ,\end{equation} \tag{ 17 }$$

where θ represents weight parameters. They reported that this norm has several desirable properties to explain the high generalization capability of DNNs. In deep linear networks, its generalization capacity (Rademacher complexity) is upper bounded by the norm. In deep ReLU networks, the Fisher–Rao norm serves as a lower bound of the capacities induced by other norms, such as the path norm [40] and the spectral norm [41]. The Fisher–Rao norm is also motivated by information geometry, and invariant under node-wise linear rescaling in ReLU networks. This is a desirable property to connect capacity measures with flatness induced by the rescaling [42].

Here, to obtain a typical evaluation of the norm, we define the average over possible parameters with fixed variances (${\sigma }_{w}^{2},{\sigma }_{b}^{2}$) by ⟨⋅⟩_θ = ∫∏_i Dθ_i (⋅), which leads to the following theorem:

Theorem 5. Suppose that assumption 1 holds. In the limit of M ≫ 1, the Fisher–Rao norm of DNNs satisfies

$$\begin{equation}{\langle {\Vert}\theta {{\Vert}}_{FR}\rangle }_{\theta }{\leqslant}{\sigma }_{w}^{2}\frac{\alpha }{{\alpha }_{\mathrm{min}}}C{\kappa }_{1},\end{equation} \tag{ 18 }$$

where α_min = min_i α_i . Equality holds in a network with a uniform width M_l = M, and then we have ${\langle {\Vert}\theta {{\Vert}}_{FR}\rangle }_{\theta }={\sigma }_{w}^{2}\left(L-1\right)C{\kappa }_{1}$.

The proof is shown in appendix A.6. Although what we can evaluate is only the average of the norm, it can be quantified by κ₁. This guarantees that the norm is independent of the network width in the limit of M ≫ 1, which was empirically conjectured by [22].

Recently, Smith and Le [43] argued that the Bayesian factor composed of the Hessian of the loss function, whose special case is the FIM, is related to the generalization. Similar analysis to the above theorem may enable us to quantitatively understand the relation between the statistics of the FIM and the indicators to measure the generalization ability.

Consider the steepest gradient descent method in a batch regime. Its update rule is given by

$$\begin{equation}{\theta }_{t+1}{\leftarrow}{\theta }_{t}-\eta \frac{\partial E\left({\theta }_{t}\right)}{\partial \theta }+\mu \left({\theta }_{t}-{\theta }_{t-1}\right),\end{equation} \tag{ 19 }$$

where η is a constant learning rate. We have added a momentum term with a coefficient μ because it is widely used in training deep networks. Assume that the squared loss function E(θ) of equation (4) has a global minimum θ* achieving the zero training error E(θ*) = 0. Then, the FIM's maximum eigenvalue is dominant over the convergence of learning as follows:

Lemma 6. A learning rate satisfying η < 2(1 + μ)/λ_max is necessary for the steepest gradient method to converge to the global minimum θ*.

The proof is given by the expansion around the minimum, i.e. E(θ* + dθ) = dθ^T Fdθ (detailed in appendix A.7). This lemma is a generalization of LeCun et al [24], which proved the case of μ = 0. Let us refer to η_c := 2(1 + μ)/λ_max as the critical learning rate. When η > η_c, the gradient method never converges to the global minimum. The previous work [24] also claimed that η = η_c/2 is the best choice for fastest convergence around the minimum. Although we focus on the batch regime, the eigenvalues also determine the bound of the gradient norms and the convergence of learning in the online regime [44].

Then, combining lemma 6 with theorem 4 leads to the following:

Theorem 7. Suppose that assumption 1 holds. Let a global minimum θ* be generated by equation (8) and satisfying E(θ*) = 0. In the limit of M ≫ 1, the gradient method never converges to θ* when

$$\begin{equation}\eta { >}{\eta }_{c},\quad {\eta }_{c}{:=}\frac{2\left(1+\mu \right)}{\alpha \left(\frac{T-1}{T}{\kappa }_{2}+\frac{1}{T}{\kappa }_{1}\right)M}.\end{equation} \tag{ 20 }$$

Theorem 7 quantitatively reveals that, the wider the network becomes, the smaller the learning rate we need to set. In addition, α is the sum over L constant positive terms, so a deeper network requires a finer setting of the learning rate and it will make the optimization more difficult. In contrast, the expressive power of the network grows exponentially as the number of layers increases [3, 45]. We thus expect there to be a trade-off between trainability and expressive power.

To confirm the effectiveness of theorem 7, we performed several experiments. As shown in figure 2, we exhaustively searched training losses while changing M and η, and found that the theoretical estimation coincides well with the experimental results. We trained deep networks (L = 4, α_l = 1, C = 10) and the loss function was given by the squared error.

Zoom In Zoom Out Reset image size

The left column of figure 2 shows the results of training on artificial data. We generated training samples x(t) in the Gaussian manner (T = 100) and teacher signals y(t) by the teacher network with a true parameter set θ* satisfying equation (8). We used the gradient method (19) with μ = 0.9 and trained the DNNs for 100 steps. The variances $\left({\sigma }_{w}^{2},{\sigma }_{b}^{2}\right)$ of the initialization of the parameters were set to the same as the global minimum. We found that the losses of the experiments were clearly divided into two areas: one where the gradient exploded (gray area) and the other where it was converging (colored area). The red line is η_c theoretically calculated using κ₁ and κ₂ on $\left({\sigma }_{w}^{2},{\sigma }_{b}^{2}\right)$ of the initial parameters. Training on the regions above η_c exploded, just as theorem 7 predicts. The explosive region with η < η_c got smaller in the limit of large M.

We performed similar experiments on benchmark datasets and found that the theory can estimate the appropriate learning rates. The results on MNIST are shown in the right column of figure 2. As shown in appendix C.2, the results of training on CIFAR-10 were almost the same as those of MINIST. We used stochastic gradient descent (SGD) with a mini-batch size of 500 and μ = 0.9, and trained the DNNs for 1 epoch. Each training sample was x(t) normalized to zero mean and variance 1 (T = 50 000). The initial values of $\left({\sigma }_{w}^{2},{\sigma }_{b}^{2}\right)$ were set to the vicinity of the special parameter region, i.e. the critical line of the order-to-chaos transition, which the previous works [3, 4] recommended to use for achieving high expressive power and trainability. Note that the variances $\left({\sigma }_{w}^{2},{\sigma }_{b}^{2}\right)$ may change from the initialization to the global minimum, and the conditions of the global minimum in theorem 7 do not hold in general. Nevertheless, the learning rates estimated by theorem 7 explained the experiments well. Therefore, the ideal conditions supposed in theorem 7 seem to hold effectively. This may be explained by the conjecture that the change from the initialization to the global minima is small in the large limit [46].

Theoretical estimations of learning rates in deep networks have so far been limited; such gradients as AdaGrad and Adam also require heuristically determined hyper-parameters for learning rates. Extending our framework would be beneficial in guessing learning rates to prevent the gradient update from exploding.

This study mainly focuses on the multi-dimensional output of C = O(1). This is because the number of labels is much smaller than the number of hidden units in most practice cases. However, since classification problems with far more labels are sometimes examined in the context of machine learning [47], it would be helpful to remark on the case of C = O(M) here. Denote the mean of the FIM's eigenvalues in the case of C = O(M) as m_λ ' and so on. Straightforwardly, we can derive

$$\begin{equation}{m}_{\lambda }^{\prime }={m}_{\lambda },\quad {s}_{\lambda }{\leqslant}{s}_{\lambda }^{\prime }{\leqslant}C{s}_{\lambda },\end{equation} \tag{ 21 }$$

$$\begin{equation}{\lambda }_{\mathrm{max}}\enspace {\leqslant}{\lambda }_{\mathrm{max}}^{\prime }{\leqslant}\sqrt{\alpha C{s}_{\lambda }}M.\end{equation} \tag{ 22 }$$

The derivation is shown in appendix A.5. The mean of eigenvalues has the same form as equation (13) obtained in the case of C = O(1). The second moment and maximum eigenvalues can be evaluated by the form of inequalities. We found that the mean is of O(1) while the maximum eigenvalue is of O(M) at least and of O(M²) at most. Therefore, the eigenvalue distribution is more widely distributed than the case of C = O(1).

(i) Case of C = 1

To avoid complicating the notation, we first consider the case of the single output (C = 1). The general case is shown after. The network output is denoted by f(t) here. We denote the Fisher information matrix with full components as

$$\begin{equation}F=\sum _{t=1}^{T}\left[\begin{matrix}\hfill {\nabla }_{W}f\left(t\right){\nabla }_{W}f{\left(t\right)}^{\mathrm{T}}\hfill & \hfill {\nabla }_{W}f\left(t\right){\nabla }_{b}f{\left(t\right)}^{\mathrm{T}}\hfill \\ \hfill {\nabla }_{b}f\left(t\right){\nabla }_{W}f{\left(t\right)}^{\mathrm{T}}\hfill & \hfill {\nabla }_{b}f\left(t\right){\nabla }_{b}f{\left(t\right)}^{\mathrm{T}}\hfill \end{matrix}\right]/T,\end{equation} \tag{ A.1 }$$

where we notice that

$$\begin{equation}{\nabla }_{{b}_{i}^{l}}f\left(t\right)={\delta }_{i}^{l}\left(t\right).\end{equation} \tag{ A.2 }$$

In general, the sum over the eigenvalues is given by the matrix trace, m_λ = Trace(F)/P. We denote the average of the eigenvalues of the diagonal block as ${m}_{\lambda }^{\left(W\right)}$ for ∇_W f∇_W f^T, and ${m}_{\lambda }^{\left(b\right)}$ for ∇_b f∇_b f^T. Accordingly, we find

$$\begin{equation}{m}_{\lambda }={m}_{\lambda }^{\left(W\right)}+{m}_{\lambda }^{\left(b\right)}.\end{equation} \tag{ A.3 }$$

The contribution of ${m}_{\lambda }^{\left(b\right)}$ is negligible in the large M limit as follows. The first term is

$$\begin{equation}{m}_{\lambda }^{\left(W\right)}=\sum _{t=1}^{T}\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({\nabla }_{W}f\left(t\right){\nabla }_{W}f{\left(t\right)}^{\mathrm{T}}\right)/\left(TP\right)\end{equation} \tag{ A.4 }$$

$$\begin{equation}=\sum _{t=1}^{T}\sum _{l}\sum _{i,j}{\delta }_{i}^{l}{\left(t\right)}^{2}{h}_{j}^{l-1}{\left(t\right)}^{2}/\left(TP\right).\end{equation} \tag{ A.5 }$$

We can apply the central limit theorem to summations over the units ${\sum }_{i}{\delta }_{i}^{l}{\left(t\right)}^{2}$ and ${\sum }_{j}{h}_{j}^{l-1}{\left(t\right)}^{2}$ independently because they do not share the index of the summation. By taking the limit of M ≫ 1, we obtain ${\sum }_{i}{\delta }_{i}^{l}{\left(t\right)}^{2}{\sum }_{j}{h}_{j}^{l-1}{\left(t\right)}^{2}/{M}_{l-1}={\tilde {q}}^{l}{\hat{q}}^{l-1}$. The variable ${\hat{q}}^{l-1}$ is computed by the recursive relation (9). Under the assumption 1, ${\tilde {q}}^{l}$ is given by the recursive relation (11). Note that this transformation to the macroscopic variables holds regardless of the sample index t. Therefore, we obtain

$$\begin{equation}{m}_{\lambda }^{\left(W\right)}={\kappa }_{1}/M,\qquad {\kappa }_{1}{:=}\sum _{l=1}^{L}\enspace \frac{{\alpha }_{l-1}}{\alpha }{\tilde {q}}^{l}{\hat{q}}^{l-1},\end{equation} \tag{ A.6 }$$

where α_l comes from M_l = α_l M, and α comes from P = αM².

In contrast, the contributions of the bias entries are smaller than those of the weight entries in the limit of M ≫ 1, as is easily confirmed:

$$\begin{equation}{m}_{\lambda }^{\left(b\right)}=\sum _{t}\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({\nabla }_{b}f\left(t\right){\nabla }_{b}f{\left(t\right)}^{\mathrm{T}}\right)/\left(TP\right)\end{equation} \tag{ A.7 }$$

$$\begin{equation}=\sum _{t}\sum _{l}\sum _{i}{\delta }_{i}^{l}{\left(t\right)}^{2}/\left(TP\right)\end{equation} \tag{ A.8 }$$

$$\begin{equation}=\sum _{l}{\tilde {q}}^{l}/\left(\alpha {M}^{2}\right)\quad \left(\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}\enspace M\gg 1\right).\end{equation} \tag{ A.9 }$$

${m}_{\lambda }^{\left(W\right)}$ is O(1/M) while ${m}_{\lambda }^{\left(b\right)}$ is O(1/M²). Hence, the mean ${m}_{\lambda }^{\left(b\right)}$ is negligible and we obtain m_λ = κ₁/M.

(ii) C > 1 of O(1)

We can apply the above computation of C = 1 to each network output ∇f_k (k = 1, ..., C):

$$\begin{equation}\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({\nabla }_{\theta }{f}_{k}{\nabla }_{\theta }{f}_{k}^{\mathrm{T}}/T\right)/P={\kappa }_{1}/M.\end{equation} \tag{ A.10 }$$

Therefore, the mean of the eigenvalues becomes

$$\begin{equation}{m}_{\lambda }=\sum _{k}^{C}\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({\nabla }_{\theta }{f}_{k}{\nabla }_{\theta }{f}_{k}^{\mathrm{T}}/T\right)/P\end{equation} \tag{ A.11 }$$

$$\begin{equation}=C{\kappa }_{1}/M.\end{equation} \tag{ A.12 }$$

□

Because the FIM is a positive semi-definite matrix, its eigenvalues are non-negative. For a constant k > 0, we obtain

$$\begin{equation}{m}_{\lambda }=\frac{1}{P}\left(\sum _{i;{\lambda }_{i}{< }k}{\lambda }_{i}+\sum _{i;{\lambda }_{i}{\geqslant}k}{\lambda }_{i}\right)\end{equation} \tag{ A.13 }$$

$$\begin{equation}{\geqslant}\frac{1}{P}\sum _{i;{\lambda }_{i}{\geqslant}k}{\lambda }_{i}\end{equation} \tag{ A.14 }$$

$$\begin{equation}{\geqslant}\frac{1}{P}N\left(\lambda {\geqslant}k\right)k.\end{equation} \tag{ A.15 }$$

This is known as Markov's inequality. When M ≫ 1, combining this with theorem 1 immediately yields

$$\begin{equation}N\left(\lambda {\geqslant}k\right){\leqslant}\alpha {\kappa }_{1}CM/k.\end{equation} \tag{ A.16 }$$

Because CT is also a trivial upper bound of N(λ ⩾ k), we obtain corollary 2. □

Let us describe the outline of the proof. One can express the FIM as F = (BB^T)/T by definition. Here, let us consider a dual matrix of F, that is, F* := (B^T B)/T. F and F* have the same nonzero eigenvalues. Because the sum of squared eigenvalues is equal to $\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({F}^{{\ast}}{\left({F}^{{\ast}}\right)}^{\mathrm{T}}\right)$, we have ${s}_{\lambda }={\sum }_{s,t}^{T}{\left({F}_{st}^{{\ast}}\right)}^{2}/P$. The non-diagonal entry ${F}_{st}^{{\ast}}\enspace \left(s\ne t\right)$ corresponds to an inner product of the network activities for different inputs x(s) and x(t), that is, κ₂. The diagonal entry ${F}_{ss}^{{\ast}}$ is given by κ₁. Taking the summation of ${\left({F}_{st}^{{\ast}}\right)}^{2}$ over all of s and t, we obtain the theorem. In particular, when T = 1 and C = 1, F* is equal to the squared norm of the derivative ∇_θ f, that is, F* = ||∇_θ f||², and one can easily check ${s}_{\lambda }=\alpha {\kappa }_{1}^{2}$.

The detailed proof is given as follows.

(i) Case of C = 1

Here, let us express the FIM as F = ∇_θ f∇_θ f^T/T, where ∇_θ f is a P × T matrix whose columns are the gradients on each input sample, i.e. ∇_θ f(t) (t = 1, ..., T). We also introduce a dual matrix of F, that is, F*:

$$\begin{equation}{F}^{{\ast}}{:=}{\nabla }_{\theta }{f}^{\mathrm{T}}{\nabla }_{\theta }f/T.\end{equation} \tag{ A.17 }$$

Note that F is a P × P matrix while F* is a T × T matrix. We can easily confirm that these F and F* have the same non-zero eigenvalues.

The squared sum of the eigenvalues is given by ${\sum }_{i}{\lambda }_{i}^{2}=\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({F}^{{\ast}}{\left({F}^{{\ast}}\right)}^{\mathrm{T}}\right)={\sum }_{st}{\left({F}_{st}^{{\ast}}\right)}^{2}$. By using the Frobenius norm ${\Vert}A{{\Vert}}_{F}{:=}\sqrt{{\sum }_{ij}{A}_{ij}^{2}}$, this is ${\sum }_{i}{\lambda }_{i}^{2}={\Vert}{F}^{{\ast}}{{\Vert}}_{F}^{2}$. Similar to m_λ , the bias entries in F* are negligible because the number of the entries is much less than that of weight entries. Therefore, we only need to consider the weight entries. The stth entry of F* is given by

$$\begin{equation}{F}_{st}^{{\ast}}=\sum _{l}\sum _{ij}{\nabla }_{{W}_{ij}^{l}}f\left(s\right){\nabla }_{{W}_{ij}^{l}}f\left(t\right)/T\end{equation} \tag{ A.18 }$$

$$\begin{equation}=\sum _{l}{M}_{l-1}{\tilde {Z}}^{l}\left(s,t\right){\hat{Z}}^{l-1}\left(s,t\right)/T,\end{equation} \tag{ A.19 }$$

where we defined

$$\begin{equation}{\hat{Z}}^{l}\left(s,t\right){:=}\frac{1}{{M}_{l}}\sum _{j}{h}_{j}^{l}\left(s\right){h}_{j}^{l}\left(t\right),\qquad {\tilde {Z}}^{l}\left(s,t\right){:=}\sum _{i}{\delta }_{i}^{l}\left(s\right){\delta }_{i}^{l}\left(t\right).\end{equation} \tag{ A.20 }$$

We can apply the central limit theorem to ${\hat{Z}}^{l-1}\left(s,t\right)$ and ${\tilde {Z}}^{l}\left(s,t\right)$ independently because they do not share the index of the summation. For s ≠ t, we have ${\hat{Z}}^{l}={\hat{q}}_{st}^{l}+\mathcal{N}\left(0,\hat{\gamma }/M\right)$ and ${\tilde {Z}}^{l}={\tilde {q}}_{st}^{l}+\mathcal{N}\left(0,\tilde {\gamma }/M\right)$ in the limit of M ≫ 1, where the macroscopic variables ${\hat{q}}_{st}^{l}$ and ${\tilde {q}}_{st}^{l}$ satisfy the recurrence relations (10) and (12). Note that the recurrence relation (12) requires the assumption 1. $\hat{\gamma }$ and $\tilde {\gamma }$ are constants of O(1). Then, for all s and t(≠s),

$$\begin{equation}{F}_{st}^{{\ast}}=\sum _{l}{M}_{l-1}\left({\tilde {q}}_{st}^{l}+O\left(1/\sqrt{M}\right)\right)\left({\hat{q}}_{st}^{l-1}+O\left(1/\sqrt{M}\right)\right)/T\end{equation} \tag{ A.21 }$$

$$\begin{equation}=\alpha {\kappa }_{2}M/T+O\left(\sqrt{M}\right)/T.\end{equation} \tag{ A.22 }$$

Similarly, for s = t, we have ${\hat{Z}}^{l}={\hat{q}}^{l}+O\left(1/\sqrt{M}\right)$, ${\tilde {Z}}^{l}={\tilde {q}}^{l}+O\left(1/\sqrt{M}\right)$ and then ${F}_{ss}^{{\ast}}=\alpha {\kappa }_{1}M/T+O\left(\sqrt{M}\right)/T$.

Thus, under the limit of M ≫ 1, the dual matrix is asymptotically given by

$$\begin{equation}{F}^{{\ast}}=\alpha MK/T+O\left(\sqrt{M}\right)/T,\qquad K{:=}\left[\begin{matrix}\hfill {\kappa }_{1}\hfill & \hfill {\kappa }_{2}\hfill & \hfill \dots \hfill & \hfill {\kappa }_{2}\hfill \\ \hfill {\kappa }_{2}\hfill & \hfill {\kappa }_{1}\hfill & \hfill \hfill & \hfill {\vdots}\hfill \\ \hfill {\vdots}\hfill & \hfill \hfill & \hfill \ddots \hfill & \hfill {\kappa }_{2}\hfill \\ \hfill {\kappa }_{2}\hfill & \hfill \dots \hfill & \hfill {\kappa }_{2}\hfill & \hfill {\kappa }_{1}\hfill \end{matrix}\right].\end{equation} \tag{ A.23 }$$

Neglecting the lower order term, we obtain

$$\begin{equation}{s}_{\lambda }=\sum _{s,t}^{T}{\left({F}_{st}^{{\ast}}\right)}^{2}/P\end{equation} \tag{ A.24 }$$

$$\begin{equation}=\alpha \left(\frac{T-1}{T}{\kappa }_{2}^{2}+\frac{1}{T}{\kappa }_{1}^{2}\right).\end{equation} \tag{ A.25 }$$

Note that, when ${\hat{q}}_{st}^{l}=0$, κ₂ becomes zero and the lower order term may be non-negligible. In this exceptional case, we have ${s}_{\lambda }=\alpha {\kappa }_{1}^{2}/T+O\left(1/M\right)$, where the second term comes from the $O\left(\sqrt{M}\right)/T$ term of equation (A.23). Therefore, the lower order evaluation depends on the T/M ratio, although it is outside the scope of this study. Intuitively, the origin of ${\hat{q}}_{st}^{l}\ne 0$ is related to the offset of firing activities ${h}_{i}^{l}$. The condition of ${\hat{q}}_{st}^{l}\ne 0$ is satisfied when the bias terms exist or when the activation ϕ(⋅) is not an odd function. In such cases, the firing activities have the offset $\mathrm{E}\left[{h}_{i}^{l}\left(t\right)\right]\ne 0$. Therefore, for any input samples s and t (s ≠ t), we have ${\sum }_{i}{h}_{i}^{l}\left(s\right){h}_{i}^{l}\left(t\right)/{M}_{l}={\hat{q}}_{st}^{l}\ne 0$ and then κ₂ ≠ 0 makes s_λ of O(1).

(ii) C > 1 of O(1)

Here, we introduce the following dual matrix F*:

$$\begin{equation}{F}^{{\ast}}{:=}{B}^{\mathrm{T}}B/T,\end{equation} \tag{ A.26 }$$

$$\begin{equation}B{:=}\left[{\nabla }_{\theta }{f}_{1}\enspace \enspace {\nabla }_{\theta }{f}_{2}\enspace \enspace \dots \enspace \enspace {\nabla }_{\theta }{f}_{C}\right],\end{equation} \tag{ A.27 }$$

where ∇_θ f_k is a P × T matrix whose columns are the gradients on each input sample, i.e. ∇_θ f_k (t) (t = 1, ..., T), and B is a P × CT matrix. The FIM is represented by F = BB^T/T. F* is a CT × CT matrix and consists of T × T block matrices,

$$\begin{equation}{F}^{{\ast}}\left(k,{k}^{\prime }\right){:=}{\nabla }_{\theta }{f}_{k}^{\mathrm{T}}{\nabla }_{\theta }{f}_{{k}^{\prime }}/T,\end{equation} \tag{ A.28 }$$

for k, k' = 1, ..., C.

The diagonal block F*(k, k) is evaluated in the same way as the case of C = 1. It becomes αMK/T as shown in equation (A.23). The non-diagonal block F*(k, k') has the following stth entries:

$$\begin{equation}{F}^{{\ast}}{\left(k,{k}^{\prime }\right)}_{st}=\sum _{l}\sum _{ij}{\nabla }_{{W}_{ij}^{l}}{f}_{k}^{}\left(s\right){\nabla }_{{W}_{ij}^{l}}{f}_{{k}^{\prime }}\left(t\right)/T\end{equation} \tag{ A.29 }$$

$$\begin{equation}=\sum _{l}{M}_{l-1}\left(\sum _{i}{\delta }_{k,i}^{l}\left(s\right){\delta }_{{k}^{\prime },i}^{l}\left(t\right)\right){\hat{Z}}^{l-1}\left(s,t\right)/T.\end{equation} \tag{ A.30 }$$

Under the limit of M ≫ 1, while ${\tilde {Z}}^{l}\left(s,t\right)$ becomes ${\tilde {q}}_{st}^{l}$ of O(1), $\left({\sum }_{i}{\delta }_{k,i}^{l}\left(s\right){\delta }_{{k}^{\prime },i}^{l}\left(t\right)\right)$ becomes zero and its lower order term of $O\left(1/\sqrt{M}\right)$ appears. This is because the different outputs (k ≠ k') do not share the weights ${W}_{ij}^{L}$. We have ${\sum }_{i}{\delta }_{k,i}^{L}\left(s\right){\delta }_{{k}^{\prime },i}^{L}\left(t\right)=0$ and then obtain ${\sum }_{i}{\delta }_{k,i}^{l}\left(s\right){\delta }_{{k}^{\prime },i}^{l}\left(t\right)=0$ (l = 1, ..., L − 1) through the backpropagated chain (7). Thus, the entries of the non-diagonal blocks (A.28) become of $O\left(\sqrt{M}\right)/T$, and we have

$$\begin{equation}{F}^{{\ast}}\left(k,{k}^{\prime }\right)=\alpha MK/T{\delta }_{k,{k}^{\prime }}+O\left(\sqrt{M}\right)/T,\end{equation} \tag{ A.31 }$$

where δ_k,k' is the Kronecker delta.

After all, we have

$$\begin{equation}{s}_{\lambda }=\sum _{k,{k}^{\prime }}^{C}\sum _{s,t}^{T}{\left({F}^{{\ast}}{\left(k,{k}^{\prime }\right)}_{st}\right)}^{2}/P\end{equation} \tag{ A.32 }$$

$$\begin{equation}=C\alpha \left(\frac{T-1}{T}{\kappa }_{2}^{2}+\frac{1}{T}{\kappa }_{1}^{2}\right)+CO\left(1/\sqrt{M}\right)+C\left(C-1\right)O\left(1/M\right),\end{equation} \tag{ A.33 }$$

where the first term comes from the diagonal blocks of O(M) and the second one is their lower order term. The third term comes from the non-diagonal blocks of $O\left(\sqrt{M}\right)$. As one can see from here, when C = O(M), the third term becomes non-negligible. This case is examined in section 4.3. □

(i) Case of C = 1

Because F and F* have the same non-zero eigenvalues, what we should derive here is the maximum eigenvalue of F*. As shown in equation (A.23), the leading term of F* asymptotically becomes αMK/T in the limit of M ≫ 1. The eigenvalues of αMK/T are explicitly obtained as follows: ${\lambda }_{\mathrm{max}}=\alpha \left(\frac{T-1}{T}{\kappa }_{2}+\frac{1}{T}{\kappa }_{1}\right)M$ for an eigenvector e = (1, ..., 1), and λ_i = α(κ₁ − κ₂)M/T for eigenvectors e₁ − e_i (i = 2, ..., T) where e_i denotes a unit vector whose entries are 1 for the ith entry and 0 otherwise. Thus, we obtain ${\lambda }_{\mathrm{max}}=\alpha \left(\frac{T-1}{T}{\kappa }_{2}+\frac{1}{T}{\kappa }_{1}\right)M$.

(ii) C > 1 of O(1)

Let us denote F* shown in equation (A.31) by ${F}^{{\ast}}{:=}{\bar{F}}^{{\ast}}+R$. ${\bar{F}}^{{\ast}}$ is the leading term of F* and given by a CT × CT block diagonal matrix whose diagonal blocks are given by αMK/T. R denotes the residual term of $O\left(\sqrt{M}\right)/T$. In general, the maximum eigenvalue is denoted by the spectral norm ||⋅||₂, that is, λ_max = ||F*||₂. Using the triangle inequality, we have

$$\begin{equation}{\lambda }_{\mathrm{max}}\enspace {\leqslant}{\Vert}{\bar{F}}^{{\ast}}{{\Vert}}_{2}+{\Vert}R{{\Vert}}_{2},\end{equation} \tag{ A.34 }$$

We can obtain ${\Vert}{\bar{F}}^{{\ast}}{{\Vert}}_{2}=\alpha \left(\frac{T-1}{T}{\kappa }_{2}+\frac{1}{T}{\kappa }_{1}\right)M$ because the maximum eigenvalues of the diagonal blocks are the same as the case of C = 1. Regarding ||R||₂, this is bounded by ${\Vert}R{{\Vert}}_{2}{\leqslant}{\Vert}R{{\Vert}}_{F}=\sqrt{{C}^{2}{\sum }_{st}{\left(O\left(\sqrt{M}\right)/T\right)}^{2}}=O\left(C\sqrt{M}\right)$. Therefore, when C = O(1), we can neglect ||R||₂ of $O\left(\sqrt{M}\right)$ compared to ${\Vert}{\bar{F}}^{{\ast}}{{\Vert}}_{2}$ of O(M).

On the other hand, we can also derive the lower bound of λ_max as follows. In general, we have

$$\begin{equation}{\lambda }_{\mathrm{max}}={\mathrm{max}}_{\mathbf{v};{\Vert}\mathbf{v}{{\Vert}}^{2}=1}{\mathbf{v}}^{\mathrm{T}}{F}^{{\ast}}\mathbf{v}.\end{equation} \tag{ A.35 }$$

Then, we find

$$\begin{equation}{\lambda }_{\mathrm{max}}\enspace {\geqslant}{\mathbf{v}}_{1}^{\mathrm{T}}{F}^{{\ast}}{\mathbf{v}}_{1},\end{equation} \tag{ A.36 }$$

where v₁ is a CT-dimensional vector whose first T entries are $1/\sqrt{T}$ and the others are 0, that is, ${v}_{1}=\left(1,\dots ,1,0,\dots ,0\right)/\sqrt{T}$. We can compute this lower bound by taking the sum over the entries of F*(1, 1), which is equal to equation (A.23):

$$\begin{equation}{\lambda }_{\mathrm{max}}\enspace {\geqslant}\left(\frac{T-1}{T}{\kappa }_{2}+\frac{1}{T}{\kappa }_{1}\right)M.\end{equation} \tag{ A.37 }$$

Finally, we find that the upper bound (A.34) and lower bound (A.37) asymptotically take the same value of O(M), that is, ${\lambda }_{\mathrm{max}}=\left(\frac{T-1}{T}{\kappa }_{2}+\frac{1}{T}{\kappa }_{1}\right)M$.

□

The mean of eigenvalues m_λ ' is derived in the same way as shown in section A.1(ii), that is, m'_λ = Cκ₁/M.

Regarding the second moment s_λ ', the lower order term becomes non-negligible as remarked in equation (A.33). We evaluate this s_λ ' by using inequalities as follows:

$$\begin{equation}{s}_{\lambda }^{\prime }={\Vert}{F}^{{\ast}}{{\Vert}}_{F}^{2}/P\end{equation} \tag{ A.38 }$$

$$\begin{equation}=\left(\sum _{k}^{C}{\Vert}{\nabla }_{\theta }{f}_{k}^{\mathrm{T}}{\nabla }_{\theta }{f}_{k}{{\Vert}}_{F}^{2}+\sum _{k,{k}^{\prime }}^{C}{\Vert}{\nabla }_{\theta }{f}_{k}^{\mathrm{T}}{\nabla }_{\theta }{f}_{{k}^{\prime }}{{\Vert}}_{F}^{2}\right)/P\end{equation} \tag{ A.39 }$$

$$\begin{equation}{\geqslant}\sum _{k}^{C}{\Vert}{\nabla }_{\theta }{f}_{k}^{\mathrm{T}}{\nabla }_{\theta }{f}_{k}{{\Vert}}_{F}^{2}/P.\end{equation} \tag{ A.40 }$$

As shown in section A.3, for any k, we obtain ${\Vert}{\nabla }_{\theta }{f}_{k}^{\mathrm{T}}{\nabla }_{\theta }{f}_{k}{{\Vert}}_{F}^{2}/P=\alpha \left(\frac{T-1}{T}{\kappa }_{2}^{2}+\frac{1}{T}{\kappa }_{1}^{2}\right)$ in the limit of M ≫ 1. Thus, the lower bound becomes the same form as s_λ , that is, ${s}_{\lambda }=C\alpha \left(\frac{T-1}{T}{\kappa }_{2}^{2}+\frac{1}{T}{\kappa }_{1}^{2}\right)$. In contrast, the upper bound is given by

$$\begin{equation}{s}_{\lambda }^{\prime }={\Vert}F{{\Vert}}_{F}^{2}/P\end{equation} \tag{ A.41 }$$

$$\begin{equation}={\Vert}\sum _{k}^{C}{F}_{k}{{\Vert}}_{F}^{2}/P\end{equation} \tag{ A.42 }$$

$$\begin{equation}{\leqslant}{\left(\sum _{k}^{C}{\Vert}{F}_{k}{{\Vert}}_{F}\right)}^{2}/P,\end{equation} \tag{ A.43 }$$

where F_k denotes the FIM of the kth output, i.e. F_k := ∑_t ∇_θ f_k (t)∇_θ f_k (t)^T/T. Therefore, the upper bound is reduced to the summation over s_λ of C = 1. In the limit of M ≫ 1, we obtain ${s}_{\lambda }^{\prime }{\leqslant}{C}^{2}{\Vert}{F}_{k}{{\Vert}}_{F}^{2}/P={C}^{2}\alpha \left(\frac{T-1}{T}{\kappa }_{2}^{2}+\frac{1}{T}{\kappa }_{1}^{2}\right)=C{s}_{\lambda }$.

Next, we show inequalities for λ_max. We have already derived the lower bound (A.37) and this bound holds in the case of C = O(M) as well. In contrast, the upper bound (A.34) may become loose when C is larger than O(1) because of the residual term ||R||₂. Although it is hard to explicitly obtain the value of ||R||₂, the following upper bound holds and is easy to compute by using s_λ of equation (14). Because the FIM is a positive semi-definite matrix, λ_i ⩾ 0 holds by definition. Then, we have ${\lambda }_{\mathrm{max}}\enspace {\leqslant}\sqrt{{\sum }_{i}{\lambda }_{i}^{2}}$. Combining this with ${s}_{\lambda }^{\prime }={\sum }_{i}{\lambda }_{i}^{2}/P$, we have ${\lambda }_{\mathrm{max}}\enspace {\leqslant}\sqrt{\alpha {s}_{\lambda }^{\prime }}M{\leqslant}\sqrt{\alpha C{s}_{\lambda }}M$.

□

The Fisher–Rao norm is written as

$$\begin{equation}{\Vert}\theta {{\Vert}}_{FR}=\sum _{l,ij}\sum _{{l}^{\prime },ab}{F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}{W}_{ij}^{l}{W}_{ab}^{{l}^{\prime }},\end{equation} \tag{ A.44 }$$

where F_{(l,ij),(l',ab)} represents an entry of the FIM, that is, ${\sum }_{k}^{C}{\sum }_{t}{\nabla }_{{W}_{ij}^{l}}{f}_{k}\left(t\right){\nabla }_{{W}_{ab}^{{l}^{\prime }}}{f}_{k}\left(t\right)/T$. Because F_{(l,ij),(l',ab)} includes the random variables ${W}_{ij}^{l}$ and ${W}_{ab}^{{l}^{\prime }}$, we consider the following expansion. Note that ${W}_{ij}^{l}$ and ${W}_{ab}^{{l}^{\prime }}$ are infinitesimals generated by equation (8). Performing a Taylor expansion around ${W}_{ij}^{l}={W}_{ab}^{{l}^{\prime }}=0$, we obtain

$$\begin{align}\hfill {F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}\left(\theta \right)& ={F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}\left({\theta }^{{\ast}}\right)+\frac{\partial {F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}}{\partial {W}_{ij}^{l}}\left({\theta }^{{\ast}}\right){W}_{ij}^{l}+\frac{\partial {F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}}{\partial {W}_{ab}^{{l}^{\prime }}}\left({\theta }^{{\ast}}\right){W}_{ab}^{{l}^{\prime }}\hfill \\ \hfill & \quad +\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{e}\mathrm{r}-\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{e}\mathrm{r}\enspace \enspace \mathrm{t}\mathrm{e}\mathrm{r}\mathrm{m}\mathrm{s},\hfill \end{align} \tag{ A.45 }$$

where θ* is the parameter set $\left\{{W}_{ij}^{l},{b}_{i}^{l}\right\}$ with ${W}_{ij}^{l}={W}_{ab}^{{l}^{\prime }}=0$. By substituting the above expansion into the Fisher-Rao norm and taking the average ⟨⋅⟩_θ , we obtain the following leading term:

$$\begin{equation}{\left\langle {F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}{W}_{ij}^{l}{W}_{ab}^{{l}^{\prime }}\right\rangle }_{\theta }={\left\langle {F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}\left({\theta }^{{\ast}}\right){W}_{ij}^{l}{W}_{ab}^{{l}^{\prime }}\right\rangle }_{\theta }\end{equation} \tag{ A.46 }$$

$$\begin{equation}={\left\langle {F}_{\left(l,ij\right),\left({l}^{\prime },ab\right)}\left({\theta }^{{\ast}}\right)\right\rangle }_{{\theta }^{{\ast}}}{\left\langle {W}_{ij}^{l}{W}_{ab}^{{l}^{\prime }}\right\rangle }_{\left\{{W}_{ij}^{l},{W}_{ab}^{{l}^{\prime }}\right\}}\end{equation} \tag{ A.47 }$$

For, (l, ij) ≠ (l', ab), the last line becomes zero because of ${\langle {W}_{ij}^{l}{W}_{ab}^{{l}^{\prime }}\rangle }_{\left\{{W}_{ij}^{l},{W}_{ab}^{{l}^{\prime }}\right\}}={\langle {W}_{ij}^{l}\rangle }_{{W}_{ij}^{l}}{\langle {W}_{ab}^{{l}^{\prime }}\rangle }_{{W}_{ab}^{{l}^{\prime }}}=0$. For (l, ij) = (l', ab), we have ${\langle {\left({W}_{ij}^{l}\right)}^{2}\rangle }_{\left\{{W}_{ij}^{l}\right\}}={\sigma }_{w}^{2}/{M}_{l-1}$. After all, in the limit of M ≫ 1, we obtain

$$\begin{equation}{\left\langle {\Vert}\theta {{\Vert}}_{FR}\right\rangle }_{\theta }=\sum _{k}^{C}\frac{{\sum }_{t}}{T}\sum _{l}{\left\langle \sum _{i}{\delta }_{k,i}^{l}{\left(t\right)}^{2}\sum _{j}{h}_{j}^{l-1}{\left(t\right)}^{2}\right\rangle }_{{\theta }^{{\ast}}}\frac{{\sigma }_{w}^{2}}{{M}_{l-1}}\end{equation} \tag{ A.48 }$$

$$\begin{equation}=\sum _{k}^{C}\frac{{\sum }_{t}}{T}{\sigma }_{w}^{2}\sum _{l}{\left\langle {\tilde {q}}^{l}\right\rangle }_{\theta }{\left\langle {\hat{q}}^{l-1}\right\rangle }_{\theta }\end{equation} \tag{ A.49 }$$

$$\begin{equation}={\sigma }_{w}^{2}C\sum _{l}{\tilde {q}}^{l}{\hat{q}}^{l-1},\end{equation} \tag{ A.50 }$$

where the derivation of the macroscopic variables is similar to that of m_λ , as shown in section A.1. Since we have ${\kappa }_{1}={\sum }_{l}\frac{{\alpha }_{l-1}}{\alpha }{\tilde {q}}^{l}{\hat{q}}^{l-1}$, it is easy to confirm ${\langle {\Vert}\theta {{\Vert}}_{FR}\rangle }_{\theta }{\leqslant}{\sigma }_{w}^{2}\alpha /{\alpha }_{\mathrm{min}}\enspace C{\kappa }_{1}$. When all α_l take the same value, we have α/α_min = L − 1 and the equality holds. □

Suppose a perturbation around the global minimum: θ_t = θ* + Δ_t . Then, the gradient update becomes

$$\begin{equation}{{\Delta}}_{t+1}{\leftarrow}\left(I-\eta F\right){{\Delta}}_{t}+\mu \left({{\Delta}}_{t}-{{\Delta}}_{t-1}\right),\end{equation} \tag{ A.51 }$$

where we have used E(θ*) = 0 and ∂E(θ*)/∂θ = 0.

Consider a coordinate transformation from Δ_t to ${\bar{{\Delta}}}_{t}$ that diagonalizes F. It does not change the stability of the gradients. Accordingly, we can update the ith component as follows:

$$\begin{equation}{\bar{{\Delta}}}_{t+1,i}{\leftarrow}\left(1-\eta {\lambda }_{i}+\mu \right){\bar{{\Delta}}}_{t,i}-\mu {\bar{{\Delta}}}_{t-1,i}.\end{equation} \tag{ A.52 }$$

Solving its characteristic equation, we obtain the general solution,

$$\begin{equation}{\bar{{\Delta}}}_{t,i}=A{\lambda }_{+}^{t}+B{\lambda }_{-}^{t},\qquad {\lambda }_{{\pm}}=\left(1-\eta {\lambda }_{i}+\mu {\pm}\sqrt{{\left(1-\eta {\lambda }_{i}+\mu \right)}^{2}-4\enspace \mu }\right)/2,\end{equation} \tag{ A.53 }$$

where A and B are constants. This recurrence relation converges if and only if ηλ_i < 2(1 + μ) for all i. Therefore, η < 2(1 + μ)/λ_max is necessary for the steepest gradient to converge to θ*. □

Consider the following error function as an activation function ϕ(x):

$$\begin{equation}\enspace \text{erf}\left(x\right)=\frac{2}{\sqrt{\pi }}{\int }_{0}^{x}\enspace \mathrm{exp}\left(-{t}^{2}\right)\mathrm{d}t.\end{equation} \tag{ B.1 }$$

The error function well approximates the tanh function and has a sigmoid-like shape. For a network with ϕ(x) = erf(x), the recurrence relations for macroscopic variables do not require numerical integrations.

(i) ${\hat{q}}^{l}$ and ${\tilde {q}}^{l}$: note that we can analytically integrate the error functions over a Gaussian distribution:

$$\begin{equation}{\int }_{0}^{\infty }Dx\enspace \text{erf}\left(ax\right)\enspace \text{erf}\left(bx\right)=\frac{1}{\pi }\enspace {\mathrm{tan}}^{-1}\frac{\sqrt{2}ab}{\sqrt{{a}^{2}+{b}^{2}+1/2}}.\end{equation} \tag{ B.2 }$$

Hence, the recurrence relations for the feedforward signals (9) have the following analytical forms:

$$\begin{equation}{\hat{q}}^{l+1}=\frac{2}{\pi }\enspace {\mathrm{tan}}^{-1}\left(\frac{{q}^{l+1}}{\sqrt{{q}^{l+1}+1/4}}\right),\qquad {q}^{l+1}={\sigma }_{w}^{2}{\hat{q}}^{l}+{\sigma }_{b}^{2}.\end{equation} \tag{ B.3 }$$

Because the derivative of the error function is Gaussian, we can also easily integrate ϕ'(x) over the Gaussian distribution and obtain the following analytical representations of the recurrence relations (11):

$$\begin{equation}{\tilde {q}}^{l}=\frac{2{\tilde {q}}^{l+1}{\sigma }_{w}^{2}}{\pi \sqrt{{q}^{l}+1/4}},\quad {\tilde {q}}^{L}=1.\end{equation} \tag{ B.4 }$$

(ii) ${\hat{q}}_{st}^{l}$ and ${\tilde {q}}_{st}^{l}$:

To compute the recurrence relations for the feedforward correlations (10), note that we can generally transform I_ϕ [a, b] into

$$\begin{equation}{I}_{\phi }\left[a,b\right]=\int Dy{\left(\int Dx\phi \left(\sqrt{a-b}x+\sqrt{b}y\right)\right)}^{2}.\end{equation} \tag{ B.5 }$$

For the error function,

$$\begin{equation}\int Dx\phi \left(\sqrt{a-b}x+\sqrt{b}y\right)=\text{erf}\enspace \frac{\sqrt{b}y}{\sqrt{1+2a-2b}},\end{equation} \tag{ B.6 }$$

and we obtain

$$\begin{equation}{I}_{\phi }\left[a,b\right]=\frac{2}{\pi }\enspace {\mathrm{tan}}^{-1}\frac{2b}{\sqrt{{\left(1+2a\right)}^{2}-{\left(2b\right)}^{2}}}.\end{equation} \tag{ B.7 }$$

This is the analytical form of the recurrence relation for ${\hat{q}}_{st}^{l}$.

Finally, because the derivative of the error function is Gaussian, we can also easily obtain

$$\begin{equation}{I}_{{\phi }^{\prime }}\left[a,b\right]=\frac{4}{\pi \sqrt{{\left(1+2a\right)}^{2}-{\left(2b\right)}^{2}}}.\end{equation} \tag{ B.8 }$$

This is the analytical forms of the recurrence relations for ${\tilde {q}}_{st}^{l}$.

We define a ReLU activation as ϕ(x) = 0(x < 0), x(0 ⩽ x). For a network with this ReLU activation function, the recurrence relations for the macroscopic variables require no numerical integrations.

(i) ${\hat{q}}^{l}$ and ${\tilde {q}}^{l}$: we can explicitly perform the integrations in the recurrence relations (9) and (11):

$$\begin{equation}{\hat{q}}^{l+1}={\hat{q}}^{l}{\sigma }_{w}^{2}/2+{\sigma }_{b}^{2}/2,\end{equation} \tag{ B.9 }$$

$$\begin{equation}{\tilde {q}}^{l}={\tilde {q}}^{l+1}{\sigma }_{w}^{2}/2,\enspace \enspace {\tilde {q}}^{L}=1.\end{equation} \tag{ B.10 }$$

(ii) ${\hat{q}}_{st}^{l}$ and ${\tilde {q}}_{st}^{l}$: we can explicitly perform the integrations in the recurrence relations (10) and (12):

$$\begin{equation}{I}_{\phi }\left[a,b\right]=\frac{a}{2\pi }\left(\sqrt{1-{c}^{2}}+c\pi /2+c\enspace {\mathrm{sin}}^{-1}\enspace c\right),\end{equation} \tag{ B.11 }$$

$$\begin{equation}{I}_{{\phi }^{\prime }}\left[a,b\right]=\frac{1}{2\pi }\left(\pi /2+{\mathrm{sin}}^{-1}\right),\end{equation} \tag{ B.12 }$$

where c = b/a.

We define a linear activation as ϕ(x) = x. For a network with this linear activation function, the recurrence relations for the macroscopic variables do not require numerical integrations.

(i) ${\hat{q}}^{l}$ and ${\tilde {q}}^{l}$: we can explicitly perform the integrations in the recurrence relations (9) and (11):

$$\begin{equation}{\hat{q}}^{l}={\hat{q}}^{l-1}{\sigma }_{w}^{2}+{\sigma }_{b}^{2},\end{equation} \tag{ B.13 }$$

$$\begin{equation}{\tilde {q}}^{l}={\tilde {q}}^{l+1}{\sigma }_{w}^{2},\quad {\tilde {q}}^{L}=1.\end{equation} \tag{ B.14 }$$

(ii) ${\hat{q}}_{st}^{l}$ and ${\tilde {q}}_{st}^{l}$: we can explicitly perform the integrations in the recurrence relations (10) and (12):

$$\begin{equation}{\hat{q}}_{st}^{l+1}={\hat{q}}_{st}^{l}{\sigma }_{w}^{2}+{\sigma }_{b}^{2},\end{equation} \tag{ B.15 }$$

$$\begin{equation}{\tilde {q}}_{st}^{l}={\tilde {q}}_{st}^{l+1}{\sigma }_{w}^{2},\quad {\tilde {q}}_{st}^{L}=1.\end{equation} \tag{ B.16 }$$

It is shown in figure C.1.

Zoom In Zoom Out Reset image size

It is shown in figure C.2.

Zoom In Zoom Out Reset image size