Universal Approximations of Invariant Maps by Neural Networks

Abstract

We describe generalizations of the universal approximation theorem for neural networks to maps invariant or equivariant with respect to linear representations of groups. Our goal is to establish network-like computational models that are both invariant/equivariant and provably complete in the sense of their ability to approximate any continuous invariant/equivariant map. Our contribution is three-fold. First, in the general case of compact groups we propose a construction of a complete invariant/equivariant network using an intermediate polynomial layer. We invoke classical theorems of Hilbert and Weyl to justify and simplify this construction; in particular, we describe an explicit complete ansatz for approximation of permutation-invariant maps. Second, we consider groups of translations and prove several versions of the universal approximation theorem for convolutional networks in the limit of continuous signals on euclidean spaces. Finally, we consider 2D signal transformations equivariant with respect to the group SE(2) of rigid euclidean motions. In this case we introduce the “charge–conserving convnet”—a convnet-like computational model based on the decomposition of the feature space into isotypic representations of SO(2). We prove this model to be a universal approximator for continuous SE(2)—equivariant signal transformations.

Proof of Lemma 4.1

The proof is a slight modification of the standard proof of Central Limit Theorem via Fourier transform (the CLT can be directly used to prove the lemma in the case $a=b=0$ when ${\mathcal {L}}_{\lambda }^{(a,b)}$ only includes diffusion factors).

To simplify notation, assume without loss of generality that $d_V=1$ (in the general case the proof is essentially identical). We will use the appropriately discretized version of the Fourier transform (i.e., the Fourier series expansion). Given a discretized signal $\Phi :(\lambda {\mathbb {Z}})^2\rightarrow {\mathbb {C}}$, we define ${\mathcal {F}}_\lambda \Phi$ as a function on $[-\frac{\pi }{\lambda },\frac{\pi }{\lambda }]^2$ by

$$\begin{aligned} {\mathcal {F}}_\lambda \Phi ({\mathbf {p}})=\frac{\lambda ^2}{2\pi }\sum _{{\gamma }\in (\lambda {\mathbb {Z}})^2} \Phi ({\gamma })e^{-i{\mathbf {p}}\cdot {\gamma }}. \end{aligned}$$

Then, ${\mathcal {F}}_\lambda : L^2((\lambda {\mathbb {Z}})^2,{\mathbb {C}})\rightarrow L^2([-\frac{\pi }{\lambda },\frac{\pi }{\lambda }]^2,{\mathbb {C}})$ is a unitary isomorphism, assuming that the scalar product in the input space is defined by $\langle \Phi ,\Psi \rangle =\lambda ^2\sum _{\gamma \in (\lambda {\mathbb {Z}})^2}\overline{\Phi (\gamma )}\Psi (\gamma )$ and in the output space by $\langle \Phi ,\Psi \rangle =\int _{[-\frac{\pi }{\lambda },\frac{\pi }{\lambda }]^2} \overline{\Phi ({\mathbf {p}})}\Psi ({\mathbf {p}}) \mathrm{d}^2{\mathbf {p}}$. Let $P_\lambda$ be the discretization projector (3.6). It is easy to check that ${\mathcal {F}}_\lambda P_\lambda$ strongly converges to the standard Fourier transform as $\lambda \rightarrow 0:$

$$\begin{aligned} \lim _{\lambda \rightarrow 0}{\mathcal {F}}_\lambda P_\lambda \Phi = {\mathcal {F}}_0 \Phi , \quad \Phi \in L^2({\mathbb {R}}^2,{\mathbb {C}}), \end{aligned}$$

where

$$\begin{aligned} {\mathcal {F}}_0 \Phi ({\mathbf {p}})=\frac{1}{2\pi }\int _{{\mathbb {R}}^2} \Phi ({\gamma })e^{-i{\mathbf {p}}\cdot {\gamma }}\mathrm{d}^2{\gamma } \end{aligned}$$

and where we naturally embed $L^2([-\frac{\pi }{\lambda },\frac{\pi }{\lambda }]^2,{\mathbb {C}})\subset L^2({\mathbb {R}}^2,{\mathbb {C}})$. Conversely, let $P_\lambda '$ denote the orthogonal projection onto the subspace $L^2([-\frac{\pi }{\lambda },\frac{\pi }{\lambda }]^2,{\mathbb {C}})$ in $L^2({\mathbb {R}}^2,{\mathbb {C}}):$

$$\begin{aligned} P_\lambda ':\Phi \mapsto \Phi |_{[-\frac{\pi }{\lambda },\frac{\pi }{\lambda }]^2}. \end{aligned}$$

(A.1)

Then

$$\begin{aligned} \lim _{\lambda \rightarrow 0}{\mathcal {F}}_\lambda ^{-1} P'_\lambda \Phi = {\mathcal {F}}_0^{-1} \Phi , \quad \Phi \in L^2({\mathbb {R}}^2,{\mathbb {C}}). \end{aligned}$$

(A.2)

Fourier transform gives us the spectral representation of the discrete differential operators (4.15), (4.16), (4.17) as operators of multiplication by function:

$$\begin{aligned} {\mathcal {F}}_\lambda \partial _{z}^{(\lambda )} \Phi&=\Psi _{\partial _{z}^{(\lambda )}}\cdot {\mathcal {F}}_\lambda \Phi ,\\ {\mathcal {F}}_\lambda \partial _{{\overline{z}}}^{(\lambda )} \Phi&=\Psi _{\partial _{{\overline{z}}}^{(\lambda )}}\cdot {\mathcal {F}}_\lambda \Phi ,\\ {\mathcal {F}}_\lambda \Delta ^{(\lambda )} \Phi&=\Psi _{\Delta ^{(\lambda )}}\cdot {\mathcal {F}}_\lambda \Phi , \end{aligned}$$

where, denoting ${\mathbf {p}}=(p_x,p_y)$,

$$\begin{aligned} \Psi _{\partial _{z}^{(\lambda )}}(p_x,p_y)&=\frac{i}{2\lambda }(\sin \lambda p_x-i\sin \lambda p_y),\\ \Psi _{\partial _{{\overline{z}}}^{(\lambda )}}(p_x,p_y)&=\frac{i}{2\lambda }(\sin \lambda p_x+i\sin \lambda p_y),\\ \Psi _{\Delta ^{(\lambda )}}(p_x,p_y)&=-\frac{4}{\lambda ^2}\Big (\sin ^2\frac{\lambda p_x}{2}+\sin ^2\frac{\lambda p_y}{2}\Big ). \end{aligned}$$

The operator ${\mathcal {L}}_\lambda ^{(a,b)}$ defined in (4.19) can then be written as

$$\begin{aligned} {\mathcal {F}}_\lambda {\mathcal {L}}_\lambda ^{(a,b)} \Phi =\Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}\cdot {\mathcal {F}}_\lambda P_\lambda \Phi , \end{aligned}$$

where the function $\Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}$ is given by

$$\begin{aligned} \Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}=(\Psi _{\partial _{z}^{(\lambda )}})^a (\Psi _{\partial _{{\overline{z}}}^{(\lambda )}})^b\left( 1+\tfrac{\lambda ^2}{8} \Psi _{\Delta ^{(\lambda )}}\right) ^{\lceil 4/\lambda ^2\rceil }. \end{aligned}$$

We can then write ${\mathcal {L}}_\lambda ^{(a,b)}\Phi$ as a convolution of $P_\lambda \Phi$ with the kernel

$$\begin{aligned} \Psi _{a,b}^{(\lambda )}=\frac{1}{2\pi }{\mathcal {F}}_\lambda ^{-1} \Psi _{{\mathcal {L}}_\lambda ^{(a,b)}} \end{aligned}$$

on the grid $(\lambda {\mathbb {Z}})^2:$

$$\begin{aligned} {\mathcal {L}}_\lambda ^{(a,b)}\Phi (\gamma )=\lambda ^2\sum _{\theta \in (\lambda {\mathbb {Z}})^2}P_\lambda \Phi (\gamma -\theta )\Psi _{a,b}^{(\lambda )}(\theta ),\quad \gamma \in (\lambda {\mathbb {Z}})^2. \end{aligned}$$

(A.3)

Now consider the operator ${\mathcal {L}}_0^{(a,b)}$ defined in (4.21). At each ${\mathbf {x}}\in {\mathbb {R}}^2$, the value ${\mathcal {L}}_0^{(a,b)} \Phi ({\mathbf {x}})$ can be written as a scalar product:

$$\begin{aligned} {\mathcal {L}}_0^{(a,b)} \Phi ({\mathbf {x}})=\int _{{\mathbb {R}}^2}\Phi ({\mathbf {x}}-{\mathbf {y}}) \Psi _{a,b}({\mathbf {y}})\mathrm{d}^2{\mathbf {y}}=\langle R_{-{\mathbf {x}}}{\widetilde{\Phi }},\Psi _{a,b}\rangle _{L^2({\mathbb {R}}^2)}, \end{aligned}$$

(A.4)

where ${\widetilde{\Phi }}({\mathbf {x}})={\overline{\Phi }}(-{\mathbf {x}})$, $\Psi _{a,b}$ is defined by (4.20), and $R_{{\mathbf {x}}}$ is our standard representation of the group ${\mathbb {R}}^2$, $R_{{\mathbf {x}}}\Phi ({\mathbf {y}})=\Phi ({\mathbf {y}}-{\mathbf {x}})$. For $\lambda >0$, we can write ${\mathcal {L}}_\lambda ^{(a,b)} \Phi ({\mathbf {x}})$ in a similar form. Indeed, using (A.3) and naturally extending the discretized signal $\Psi _{a,b}^{(\lambda )}$ to the whole ${\mathbb {R}}^2$, we have

$$\begin{aligned} {\mathcal {L}}_\lambda ^{(a,b)}\Phi (\gamma )=\int _{{\mathbb {R}}^2}\Phi (\gamma -{\mathbf {y}})\Psi _{a,b}^{(\lambda )}({\mathbf {y}})\mathrm{d}^2{\mathbf {y}}=\langle R_{-\gamma }{\widetilde{\Phi }},\Psi _{a,b}^{(\lambda )}\rangle _{L^2({\mathbb {R}}^2)}. \end{aligned}$$

Then, for any ${\mathbf {x}}\in {\mathbb {R}}^2$ we can write

$$\begin{aligned} {\mathcal {L}}_\lambda ^{(a,b)}\Phi ({\mathbf {x}})=\langle R_{-{\mathbf {x}}+\delta {\mathbf {x}}}{\widetilde{\Phi }},\Psi _{a,b}^{(\lambda )}\rangle _{L^2({\mathbb {R}}^2)}, \end{aligned}$$

(A.5)

where $-{\mathbf {x}}+\delta {\mathbf {x}}$ is the point of the grid $(\lambda {\mathbb {Z}})^2$ nearest to $-{\mathbf {x}}$.

Now consider the formulas (A.4), (A.5) and observe that, by Cauchy-Schwarz inequality and since R is norm-preserving, to prove statement 1) of the lemma we only need to show that the functions $\Psi _{a,b},\Psi _{a,b}^{(\lambda )}$ have uniformly bounded $L^2$-norms. For $\lambda >0$ we have

$$\begin{aligned} \Vert \Psi _{a,b}^{(\lambda )}\Vert ^2_{L^2({\mathbb {R}}^2)}&=\Big \Vert \frac{1}{2\pi }{\mathcal {F}}_\lambda ^{-1}\Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}\Big \Vert ^2_{L^2({\mathbb {R}}^2)}\nonumber \\&=\frac{1}{4\pi ^2}\Vert \Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}\Vert ^2_{L^2({\mathbb {R}}^2)}\nonumber \\&=\frac{1}{4\pi ^2}\big \Vert (\Psi _{\partial _{z}^{(\lambda )}})^a (\Psi _{\partial _{{\overline{z}}}^{(\lambda )}})^b\left( 1+\tfrac{\lambda ^2}{8}\Psi _{\Delta ^{(\lambda )}}\right) ^{\lceil 4/\lambda ^2\rceil }\big \Vert _{L^2({\mathbb {R}}^2)}^2\nonumber \\&\le \frac{1}{4\pi ^2}\int _{-\pi /\lambda }^{\pi /\lambda }\int _{-\pi /\lambda }^{\pi /\lambda } \big (\tfrac{|p_x|+|p_y|}{2}\big )^{2(a+b)}\nonumber \\&\quad \exp \left( -\lceil 4/\lambda ^2\rceil \left( \sin ^2\tfrac{\lambda p_x}{2}+\sin ^2\tfrac{\lambda p_y}{2}\right) \right) \mathrm{d}p_x\mathrm{d}p_y\nonumber \\&\le \frac{1}{4\pi ^2}\int _{-\infty }^{\infty }\int _{-\infty }^{\infty } \big (\tfrac{|p_x|+|p_y|}{2}\big )^{2(a+b)} \exp (-\tfrac{4}{\pi ^2}(p_x^2+p_y^2))\mathrm{d}p_x\mathrm{d}p_y\\&<\infty ,\nonumber \end{aligned}$$

(A.6)

where we used the inequalities

$$\begin{aligned}&|\sin t| \le |t|,\\&|1+t|\le e^t,\quad t>-1,\\&|\sin t|\ge \tfrac{2|t|}{\pi },\quad t\in [-\tfrac{\pi }{2},\tfrac{\pi }{2}]. \end{aligned}$$

Expression (A.6) provides a finite bound, uniform in $\lambda$, for the squared norms $\Vert \Psi _{a,b}^{(\lambda )}\Vert ^2$. This bound also holds for $\Vert \Psi _{a,b}\Vert ^2$.

Next, observe that to establish the strong convergence in statement 2) of the lemma, it suffices to show that

$$\begin{aligned} \lim _{\lambda \rightarrow 0}\Vert \Psi _{a,b}^{(\lambda )}-\Psi _{a,b}\Vert _{L^2({\mathbb {R}}^2)}=0. \end{aligned}$$

(A.7)

Indeed, by (A.4), (A.5), we would then have

$$\begin{aligned} \Vert {\mathcal {L}}_\lambda ^{(a,b)} \Phi -{\mathcal {L}}_0^{(a,b)} \Phi \Vert _\infty&=\sup _{{\mathbf {x}}\in {\mathbb {R}}^2}|\langle R_{-{\mathbf {x}}+\delta {\mathbf {x}}}{\widetilde{\Phi }},\Psi _{a,b}^{(\lambda )}\rangle -\langle R_{-{\mathbf {x}}}{\widetilde{\Phi }},\Psi _{a,b}\rangle |\\&=\sup _{{\mathbf {x}}\in {\mathbb {R}}^2}|\langle R_{-{\mathbf {x}}}(R_{\delta {\mathbf {x}}}-1){\widetilde{\Phi }},\Psi _{a,b}^{(\lambda )}\rangle +\langle R_{-{\mathbf {x}}}{\widetilde{\Phi }},\Psi _{a,b}^{(\lambda )}-\Psi _{a,b}\rangle |\\&\le \sup _{\Vert \delta {\mathbf {x}}\Vert \le \lambda }\Vert R_{\delta {\mathbf {x}}}{\widetilde{\Phi }}-{\widetilde{\Phi }}\Vert _2\sup _{\lambda }\Vert \Psi _{a,b}^{(\lambda )}\Vert _2+\Vert {\widetilde{\Phi }}\Vert _2\Vert \Psi _{a,b}^{(\lambda )}-\Psi _{a,b}\Vert _2\\&{\mathop {\longrightarrow }\limits ^{\lambda \rightarrow 0}}0 \end{aligned}$$

thanks to the unitarity of R, convergence $\lim _{\delta {\mathbf {x}}\rightarrow 0}\Vert R_{\delta {\mathbf {x}}}{\widetilde{\Phi }}-{\widetilde{\Phi }}\Vert _2=0,$ uniform boundedness of $\Vert \Psi _{a,b}^{(\lambda )}\Vert _2$ and convergence (A.7).

To establish (A.7), we write

$$\begin{aligned} \Psi _{a,b}^{(\lambda )}-\Psi _{a,b}=\frac{1}{2\pi }({\mathcal {F}}_\lambda ^{-1} \Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}-{\mathcal {F}}_0^{-1}\Psi _{{\mathcal {L}}_0^{(a,b)}}), \end{aligned}$$

where $\Psi _{{\mathcal {L}}_0^{(a,b)}}=2\pi {\mathcal {F}}_\lambda \Psi _{a,b}.$ By definition (4.20) of $\Psi _{a,b}$ and standard properties of Fourier transform, the explicit form of the function $\Psi _{{\mathcal {L}}_0^{(a,b)}}$ is

$$\begin{aligned} \Psi _{{\mathcal {L}}_0^{(a,b)}}(p_x,p_y)=\big (\tfrac{i(p_x-ip_y)}{2}\big )^a \big (\tfrac{i(p_x+ip_y)}{2}\big )^b \exp \big (-\tfrac{p_x^2+p_y^2}{2}\big ). \end{aligned}$$

Observe that the function $\Psi _{{\mathcal {L}}_0^{(a,b)}}$ is the pointwise limit of the functions $\Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}$ as $\lambda \rightarrow 0$. The functions $|\Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}|^2$ are bounded uniformly in $\lambda$ by the integrable function appearing in the integral (A.6). Therefore we can use the dominated convergence theorem and conclude that

$$\begin{aligned} \lim _{\lambda \rightarrow 0}\big \Vert \Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}-P'_\lambda \Psi _{{\mathcal {L}}_0^{(a,b)}}\big \Vert _2=0, \end{aligned}$$

(A.8)

where $P_\lambda '$ is the cut-off projector (A.1). We then have

$$\begin{aligned} \Vert \Psi _{a,b}^{(\lambda )}-\Psi _{a,b}\Vert _2&=\frac{1}{2\pi } \big \Vert {\mathcal {F}}_\lambda ^{-1}\Psi _{{\mathcal {L}}_\lambda ^{(a,b)}} -{\mathcal {F}}_0^{-1}\Psi _{{\mathcal {L}}_0^{(a,b)}}\big \Vert _2\\&\le \frac{1}{2\pi }\big \Vert {\mathcal {F}}_\lambda ^{-1}(\Psi _{{\mathcal {L}}_\lambda ^{(a,b)}}-P'_\lambda \Psi _{{\mathcal {L}}_0^{(a,b)}})\big \Vert _2+ \frac{1}{2\pi }\big \Vert ({\mathcal {F}}_\lambda ^{-1}P'_\lambda -{\mathcal {F}}_0^{-1}) \Psi _{{\mathcal {L}}_0^{(a,b)}}\big \Vert _2\\&{\mathop {\longrightarrow }\limits ^{\lambda \rightarrow 0}}0 \end{aligned}$$

by (A.8) and (A.2). We have thus proved (A.7).

It remains to show that the convergence ${\mathcal {L}}_\lambda ^{(a,b)} \Phi \rightarrow {\mathcal {L}}_0^{(a,b)} \Phi$ is uniform on compact sets $K\subset V$. This follows by a version of continuity argument. For any $\epsilon >0$, we can choose finitely many $\Phi _n,n=1,\ldots ,N,$ such that for any $\Phi \in K$ there is some $\Phi _n$ for which $\Vert \Phi -\Phi _n\Vert <\epsilon .$ Then $\Vert {\mathcal {L}}_\lambda ^{(a,b)} \Phi - {\mathcal {L}}_0^{(a,b)} \Phi \Vert \le \Vert {\mathcal {L}}_\lambda ^{(a,b)} \Phi _n- {\mathcal {L}}_0^{(a,b)} \Phi _n\Vert +2\sup _{\lambda \ge 0} \Vert {\mathcal {L}}_\lambda ^{(a,b)}\Vert \epsilon$. Since $\sup _{\lambda \ge 0} \Vert {\mathcal {L}}_\lambda ^{(a,b)}\Vert <\infty$ by statement 1) of the lemma, the desired uniform convergence for $\Phi \in K$ follows from the convergence for $\Phi _n,n=1,\ldots ,N$.