Unregularized Online Algorithms with Varying Gaussians

Abstract

Gaussians are a family of Mercer kernels which are widely used in machine learning and statistics. The variance of a Gaussian kernel reflexes the specific structure of the reproducing kernel Hilbert spaces (RKHS) induced by the Gaussian or other important features of learning problems such as the frequency of function components. As the variance of the Gaussian decreases, the learning performance and approximation ability will improve. This paper introduces the unregularized online algorithm with decreasing Gaussians where no regularization term is imposed and the samples are presented in sequence. With the appropriate step sizes, concrete learning rates are derived under the smoothness assumptions on the target function, for which are used to bound the approximation error. Additionally, a new type of the geometric noise condition is proposed to estimate the approximation error instead of any smoothness assumption. It is more general than the work in Steinwart et al. (Ann Stat 35(2):575–607, 2007), for which is only suitable for the hinge loss. An essential estimate is to bound the difference of the approximation functions generated by varying Gaussian RKHS. Fourier transform plays a crucial role in our analysis.

Appendices

Appendixes

Some Properties of RKHS with Gaussian Kernels

This section presents known results about the Gaussian kernel $k_\sigma$ and its associated RKHS $\mathcal H_\sigma$, that are useful for the proofs of our results.

Lemma A.1

[16] For any $0<\sigma <\tau ,$

$$\begin{aligned} \mathcal{H}_\tau \subset \mathcal{H}_\sigma . \end{aligned}$$

Moreover, for any $f\in \mathcal{H}_\tau ,$

$$\begin{aligned} \Vert f\Vert _{\sigma }\le \left( \frac{\tau }{\sigma }\right) ^{\frac{n}{2}}\Vert f\Vert _{\tau }. \end{aligned}$$

Lemma A.2

[12] Let $\mathcal{H}_{\sigma }({\mathbb {R}}^n)$ be the RKHS with the Gaussian kernel $K_\sigma (\cdot ,\cdot )$ on ${\mathbb {R}}^n\times {\mathbb {R}}^n.$ Then its associated norm is given for any $f\in \mathcal{H}_{\sigma }({\mathbb {R}}^n)$ by

$$\begin{aligned} \Vert f\Vert ^2_{\mathcal{H}_{\sigma }({\mathbb {R}}^n)}=\frac{1}{(2\pi )^{\frac{3n}{2}}\sigma ^{n}}\int _{{\mathbb {R}}^n}| {{\hat{f}}}(w) |^2e^{\frac{\sigma ^2|w|^2}{2}}\mathrm{d}w \end{aligned}$$

where ${{\hat{f}}}(w)$ denotes the Fourier transform of the function f.

Lemma A.3

[1] Let $\mathcal{X}$ be a non-empty subset of ${\mathbb {R}}^n.$ Then

$$\begin{aligned} \mathcal{H}_\sigma :=\left\{ f={{\tilde{f}}}|_\mathcal{X},{\tilde{f}} \in \mathcal{H}_{\sigma }({\mathbb {R}}^n)\right\} \ and \ \Vert f\Vert _\sigma =\inf \left\{ \Vert {{\tilde{f}}}\Vert _{\mathcal{H}_{\sigma }({\mathbb {R}}^n)},{{\tilde{f}}}|_\mathcal{X}=f\right\} . \end{aligned}$$

Elementary Inequalities

Lemma B.1

For $T\ge 3,$ the elementary inequalities hold as follows.

$$\begin{aligned} \sum _{j=1}^T j^{-\theta ^*}\le {\left\{ \begin{array}{ll}\frac{T^{1-\theta ^*}}{1-\theta ^*}, &{}\quad \text {if } 0<\theta ^*<1;\\ 2\log T,&{} \quad \text {if } \theta ^*=1;\\ \frac{\theta ^*}{\theta ^*-1},&{} \quad \text {if } \theta ^*>1. \end{array}\right. } \end{aligned}$$

Lemma B.2

For $T\ge 3,$

$$\begin{aligned} \sum _{k=1}^{T-2}\frac{1}{k(k+1)}\sum _{t=T-k}^T (t-1)^{-1}\le 16T^{-1}(\log T). \end{aligned}$$

Proof

When $1\le k\le \frac{T}{2},$ then $\sum _{t=T-k}^T (t-1)^{-1}\le (T-k-1)^{-1}(k+1)\le 6T^{-1}(k+1),$ and when $\frac{T}{2}< k\le T-2,$ using Lemma B.1, then $\sum _{t=T-k}^T (t-1)^{-1}\le 2\log T.$ Combing the two cases yields that

$$\begin{aligned}&\sum _{k=1}^{T-2}\frac{1}{k(k+1)}\sum _{t=T-k}^T (t-2)^{-1}\le 6T^{-1}\sum _{k=1}^{\frac{T}{2}}\frac{1}{k}+2\log T\sum _{k=\frac{T}{2}+1}^{T-1}\frac{1}{k(k+1)}\\&\quad \le 6T^{-1}\left( 2\log \frac{T}{2}\right) +4(\log T)T^{-1}=16T^{-1}(\log T). \end{aligned}$$

$\square$

Lemma B.3

We have for $T\ge 3,$

$$\begin{aligned} \sum _{k=1}^{T-2}\frac{(T-k-1)^{-1}}{k}\le 16 T^{-1}(\log T). \end{aligned}$$

Proof

When $1\le k\le \frac{T}{2},$ $(T-k-1)^{-1}\le 6 T^{-1}$ and when $\frac{T}{2}<k<T,$ $1\le T-k-1< \frac{T}{2}-1.$ Thus, using Lemma B.1, we have that

$$\begin{aligned}&\sum _{k=1}^{T-2}\frac{(T-k-1)^{-1}}{k}\le 6T^{-1}\sum _{k=1}^{\frac{T}{2}}\frac{1}{k}\\&\qquad +\sum _{k=\frac{T}{2}+1}^{T-2}\frac{(T-k-1)^{-1}}{k}\le 12 T^{-1}(\log T)+2T^{-1}\sum _{k=\frac{T}{2}+1}^{T-2}(T-k-1)^{-1}\\&\quad \le 12T^{-1}(\log T)+2T^{-1}\left( \sum _{i=1}^{\frac{T}{2}-2} i^{-1}\right) \le 16T^{-1}(\log T). \end{aligned}$$

$\square$

Lemma B.4

For any $q^*>1,$ we have for $T\ge 3,$

$$\begin{aligned} \sum _{k=1}^{T-2}\frac{1}{k(k+1)}\sum _{t=T-k}^T (t-1)^{-q^*}\le c_{q^*} T^{-1} \end{aligned}$$

where $c_{q^*}$ is a constant independent of T.

Proof

Using Lemma B.1, its proof is similar to that of Lemma B.2. Here we omit it. $\square$

Useful Lemmas and Proofs in Sect. 4

This lemma is key in estimating the approximation ability of $\mathcal{H}_\sigma .$

Lemma C.1

Let $\mathcal{X}$ be a closed unit ball in ${\mathbb {R}}^n$ and $\mathcal{{{\tilde{X}}}}=3\mathcal{X}.$ Assume that $f: \mathcal{X}\rightarrow {\mathbb {R}}$ be a measurable function. Define an extension ${{\tilde{f}}} :\mathcal{{{\tilde{X}}}}\rightarrow {\mathbb {R}}$ of the function f by

$$\begin{aligned} {{\tilde{f}}} (x)= {\left\{ \begin{array}{ll}f(x), &{}\text {if}\quad x\in \mathcal{X};\\ f\left( \frac{x}{|x|}\right) ,&{}\text {if}\quad x\in \mathcal{{{\tilde{X}}}}/\mathcal{X}. \end{array}\right. } \end{aligned}$$

Then for any $x\in \mathcal{X}$ and $\varepsilon >0,$ we have that

$$\begin{aligned} \inf _{u\in \mathcal{X}}\{|x-u|, s.t.|f(x)-f(u)|\ge \varepsilon \}=\inf _{v\in \mathcal{{{\tilde{X}}}}}\{|x-v|, s.t.|{\tilde{f}}(x)-{\tilde{f}}(v)|\ge \varepsilon \}. \end{aligned}$$

Proof

When $v\in \mathcal{{{\tilde{X}}}}/\mathcal{X},$ by simple calculations, we have that $\left| x-\frac{v}{|v|}\right| ^2< |x-v|^2$ for $x\in \mathcal{X}$ and $v\in \mathcal{{{\tilde{X}}}}/\mathcal{X}.$

For any $v\in \mathcal{{{\tilde{X}}}}/\mathcal{X},$ since ${{\tilde{f}}}(v)=f\left( \frac{v}{|v|}\right) ,$ then $|{\tilde{f}}(x)-{\tilde{f}}(v)|=\left| f(x)-f\left( \frac{v}{|v|}\right) \right|$ and

$$\begin{aligned}&\inf _{v\in \mathcal{{{\tilde{X}}}}/\mathcal{X}}\{|x-v|, s.t.|{\tilde{f}}(x)-{\tilde{f}}(v)|\ge \varepsilon \}=\inf _{v\in \mathcal{{{\tilde{X}}}}/\mathcal{X}}\left\{ |x-v|, s.t.\left| f(x)-f\left( \frac{v}{|v|}\right) \right| \ge \varepsilon \right\} \\&\quad \ge \inf _{v\in \mathcal{{{\tilde{X}}}}/\mathcal{X}}\left\{ \left| x-\frac{v}{|v|}\right| , s.t.\left| f(x)-f\left( \frac{v}{|v|}\right) \right| \ge \varepsilon \right\} \ge \inf _{u\in \mathcal{X}}\{|x-u|, s.t.|f(x)-f(u)|\ge \varepsilon \}. \end{aligned}$$

Notice the fact that for any $x\in \mathcal{X},$

$$\begin{aligned} \inf _{v\in \mathcal{{{\tilde{X}}}}}\{|x-v|, s.t.|{\tilde{f}}(x)-{\tilde{f}}(v)|\ge \varepsilon \}= \inf _{v\in \mathcal{{{\tilde{X}}}}/\mathcal{X}}\{|x-v|, s.t.|{\tilde{f}}(x)-{\tilde{f}}(v)|\ge \varepsilon \} \end{aligned}$$

or

$$\begin{aligned} \inf _{v\in \mathcal{{{\tilde{X}}}}}\{|x-v|, s.t.|{\tilde{f}}(x)-{\tilde{f}}(v)|\ge \varepsilon \}=\inf _{u\in \mathcal{X}}\{|x-u|, s.t.|f(x)-f(u)|\ge \varepsilon \}. \end{aligned}$$

Based on the estimates as above, the proof is completed. $\square$

Next, we will bound the error caused by the varying Gaussians.

Lemma C.2

Define ${{\tilde{f}}}_{\sigma _t}$ by (45) and $f_{\sigma _t}:={{\tilde{f}}}_{\sigma _t}|_\mathcal{X}$. If the variances $\{\sigma _t,t\in {\mathbb {N}}\}$ with $0<\beta <1,$ then

$$\begin{aligned} \Vert f_{\sigma }\Vert _{\sigma }&\le \frac{1}{(\sqrt{2\pi })^{n/2}}\Vert {\tilde{f}}^\phi _\rho \Vert _{L^2({\mathbb {R}}^n)}t^{n\beta /2}, \end{aligned}$$

(50)

and

$$\begin{aligned} \Vert f_{\sigma _{t}}-f_{\sigma _{t-1}}\Vert _{\sigma _t}\le \frac{2c_\beta +c_\beta ^2}{\sqrt{2}(\sqrt{2\pi })^{n/2}}\Vert {\tilde{f}}^\phi _\rho \Vert _{L^2({\mathbb {R}}^n)}t^{-1+\frac{n\beta }{2}} \end{aligned}$$

(51)

where $c_\beta$ is given in the proof of Lemma 1.

Proof

Notice that the Fourier transform $\hat{{\tilde{f}}}_{\sigma }(w)=\hat{{\tilde{K}}}_\sigma (w)\hat{{\tilde{f}}}^\phi _\rho (w)=\exp \left\{ -\frac{\sigma ^2|w|^2}{2}\right\} \hat{{\tilde{f}}}^\phi _\rho (w).$ With Lemmas A.2 and A.3, by the similar procedure in the proofs of Lemma 1, the conclusions (50) and (51) can be obtained. $\square$

With the help of the above lemmas, we can prove our convergence rate in Sect. 4.

Proof of Theorem 5

We shall prove the conclusions above by Theorem 1. By (51) and (46), we have

$$\begin{aligned} \mathcal{A}_t\le \mathcal{A}_1t^{-\frac{\zeta \beta }{1+\zeta }}\ and\ \mathcal{B}_t\le \mathcal{B}_1 t^{-1+\frac{n\beta }{2}} \end{aligned}$$

with $\mathcal{A}_1=C_{n,\zeta ,q}$ and $\mathcal{B}_1=\frac{2c_\beta +c_\beta ^2}{\sqrt{2}(\sqrt{2\pi })^{n/2}}\Vert {\tilde{f}}^\phi _\rho \Vert _{L^2({\mathbb {R}}^n)}.$ Taking $\tau =\frac{1+\epsilon }{1-\frac{n\beta }{2}},$ by (50) and (31), we have that

$$\begin{aligned} {\mathrm{I}\mathrm{E}}_{z_1,\ldots ,z_t}\left[ \Vert f_{t+1}\Vert ^2_{\sigma _t}\right] \le C^{**} t^{1-\min \{\theta +\frac{\zeta \beta }{1+\zeta },1-n\beta -\epsilon \}} \end{aligned}$$

where $C^{**}=4C_{\tau ,q,\eta }\left( 1+\frac{2\eta \mathcal{A}_1+\mathcal{B}_1^\tau +\mathcal{B}_1^{2-\tau }}{\max \{1-\theta -\zeta \beta (1+\zeta )^{-1},n\beta +\epsilon \}} +\frac{q^*}{q^*-1}\frac{\Vert {\tilde{f}}^\phi _\rho \Vert ^2_{L^2({\mathbb {R}}^n)}}{(\sqrt{2\pi })^{n}} \right) +\frac{2\Vert {\tilde{f}}^\phi _\rho \Vert ^2_{L^2({\mathbb {R}}^n)}}{(\sqrt{2\pi })^{n}}.$ Putting the estimates above into (7), following the same proof procedure of Theorem 2, we can get the desired conclusion with

$$\begin{aligned} C_3'= \frac{2{\tilde{C}}\max \Bigg \{\frac{\Vert {\tilde{f}}^\phi _\rho \Vert ^2_{L^2({\mathbb {R}}^n)}}{(\sqrt{2\pi })^{n}},C^{**},\eta \mathcal{A}_1,\mathcal{B}_1^{2-\tau } \Bigg \}}{\max \{1-\theta -\zeta \beta (1+\zeta )^{-1},n\beta +\epsilon \}}. \end{aligned}$$

$\square$