Greedy approximation by arbitrary sets

Let $H$ denote a real Hilbert space with scalar product ${(\,\cdot\,,\cdot\,)}$ and norm ${\|\,\cdot\,\|}$. Let $S(H)$ be the unit sphere of $H$. For every subset $D\subset S(H)$, referred to as a dictionary, there is a greedy approximation algorithm. With every element $x=x_0\in H$ it associates a sequence

$$\begin{equation} x_{n+1}=x_n-(x_n,g_{n+1})g_{n+1}, \qquad n=0,1,\dots, \end{equation} \tag{ 1 }$$

where $g_{n+1}\in D$ is such that

$$\begin{equation*} |(x_n,g_{n+1})|=\max\{|(x_n,g)|\colon g\in D\} \end{equation*}$$

(the existence of $\max\{|(x,g)|\colon g\in D\}$ for every $x\in H$ is an additional condition on $D$; in the case when the maximum is attained at several elements of $D$, any one of them can be selected as $g_{n+1}$).

More precisely, this algorithm is called the pure greedy algorithm, in contrast to other approximation algorithms whose names contain the word 'greedy'; see [1].

It is known that the pure greedy algorithm converges for every complete dictionary $D$ (that is, when the linear combinations of elements of $D$ are dense in $H$). This means that for every initial element $x=x_0$ one has $x_n\to 0$ as $n\to\infty$ and $x$ is the sum of the series $\sum_{n=0}^\infty (x_n,g_{n+1})g_{n+1}$ (see [2] or [1], Ch. 2).

Note that the term $(x_n,g_{n+1})g_{n+1}$ in (1) is the nearest point to $x_n$ in the set

$$\begin{equation*} \Lambda (D)=\{\lambda g\colon \lambda\in {\mathbb R},\, g\in D\}. \end{equation*}$$

This observation leads to the definition of a greedy algorithm of approximation by an arbitrary subset of $H$ as introduced below.

Given any subset $M\subset H$ and any element $x\in H$, we write $\rho(x,M)=\inf\{\|x- y\|\colon y\in M\}$ for the distance from $x$ to $M$ and define the metric projection $P_M(x)=\{y\in M\colon \|x-y\|=\rho(x,M)\}$.

Let $M$ be proximal, that is, let $P_M(x)\ne \varnothing$ for every $x\in H$ (in particular, $M$ is closed).

We define the greedy approximation algorithm with respect to such a proximal set $M$ similarly to (1), that is, for any element $x=x_0\in H$ we consider the sequence

$$\begin{equation} x_{n+1}=x_n-y_{n+1}, \qquad \text{where} \quad y_{n+1}\in P_M(x_n), \quad n=0,1,\dots \end{equation} \tag{ 2 }$$

(when $P_M(x_n)$ is not a singleton, we can choose any one of its elements as $y_{n+1}$). If $M=\Lambda(D)$ for some dictionary $D$, then the sequence (2) clearly coincides with (1). It is also clear that algorithm (2) works in an arbitrary Banach space $X$ and coincides with the well-known $X$-greedy algorithm ([1], Ch. 6) in the case when $M=\Lambda(D)$, where $D$ is a dictionary in $X$.

We say that algorithm (2) converges if $x_n\to 0$ as $n\to\infty$. In this case the initial element $x$ can be expressed as $\sum_{n=1}^\infty y_n$ in terms of elements from $M$.

A systematic study of such nonlinear approximations by sums of elements of a given set (without coefficients) has recently been started within the theory of density of semigroups in a Banach space [3] (see also §6 below). Particular cases of such approximations were studied much earlier. For example, approximations by simple partial fractions (logarithmic derivatives of polynomials) first appeared in a paper by Korevaar [4], and have a natural electrostatic interpretation (see [5]–[8] and the bibliography therein). Algorithm (2) and other algorithms of greedy approximation by an arbitrary set $M$ considered in this paper provide a natural way of constructing such approximations by sums of elements of $M$. In addition, these algorithms extend the geometric part of the classical greedy approximation theory, and it appears to the author that they may even open a new chapter of general geometric approximation theory.

We first investigate conditions on $M$ which are necessary or sufficient for the convergence of algorithm (2), that is, for

$$\begin{equation} x_n\to 0\quad (n\to\infty) \end{equation} \tag{ 3 }$$

for every $x\in H$.

The condition $0\in M$ is clearly necessary for (3): otherwise some ball $B_r(0)$ will be disjoint from $M$ (recall that $M$ is closed), and even if $x_n$ lies in the twice smaller ball $B_{r/2}(0)$, the next element $x_{n+1}=x_n-y_{n+1}$ will not belong to $B_{r/2}(0)$ since the norm of $y_{n+1}\in P_M(x_n)$ is not smaller than $r$.

Furthermore, the convergence of the greedy algorithm forces $M$ to have the property

$$\begin{equation} \forall\, x\ne 0 \quad \rho(x,M)<\|x\|. \end{equation} \tag{ 4 }$$

Indeed, if $\rho(x,M)=\|x\|$ for some non-zero $x$ ("$>$" is impossible since $0\in M$), then $0\in P_M(x)$. Selecting the zero element as $y_{n+1}$ at each step of the greedy algorithm for $x_0=x$, we obtain $x_n=x$ ($n=1,2,\dots$), and there is no convergence.

A set $M$ satisfying (4) is said to be norm-reducing. Condition (4), which clearly makes sense in any Banach space, is equivalent to the presence of points of $M$ in every open ball $B(x,\|x\|)$, whose sphere passes through $0$. In this geometric sense, the property of being norm-reducing in a Hilbert space strengthens the property of being all-round. (A subset of a Banach space $X$ is said to be all-round [3] if it has a non-empty intersection with the open half-space $\{x\colon f(x)>0\}$ for any non-zero functional $f$ in the dual space $X^*$.) A norm-reducing subset of a non-reflexive Banach space does not have to be all-round; see §6 below.

When $M=\Lambda(D)$ for a dictionary $D$ in a Hilbert space, the property of being norm-reducing is equivalent to the property of being all-round, which is, in its turn, equivalent to $D$ being complete.

The purpose of this paper is to show that the norm-reducing condition (4) is generally insufficient for the convergence (3) of greedy approximations (Remark 1), but provides this convergence in a Hilbert space under certain additional assumptions on $M$ (Theorem 1). Moreover, we propose a modified semi-greedy algorithm of approximation by elements of an arbitrary set $M$, which converges for every initial element in the case when $M$ is a norm-reducing symmetric subset of a Hilbert space (Theorem 3). We also propose recursive greedy and semi-greedy algorithms, for which these convergence results hold without the symmetry condition on $M$ (Theorem 4). As a consequence, we prove the density of the additive semigroup generated by a norm-reducing set $M$ in a Hilbert space (Theorem 5). This result does not hold in an arbitrary non-reflexive space (Remark 3).

We construct this example in the Hilbert space $l_2$ of sequences $x=(t_1,t_2,\dots)$ with the norm

$$\begin{equation*} \|x\|=\biggl(\sum_{k=1}^\infty t_k^2\biggr)^{1/2}. \end{equation*}$$

Let $e_n=(0,\dots ,0,1,0,\dots)$ be the basis vector of this space with 1 at the $n$th position, and let

$$\begin{equation*} S_n=\{(t_1,\dots,t_n,0,0,\dots)\colon t_1^2+\dots+t_n^2=1\}. \end{equation*}$$

We take a convex sequence of numbers $\lambda_n$ such that

$$\begin{equation*} \lambda_1=1, \qquad \lambda_n \downarrow \frac{1}{2}, \qquad \varepsilon_n:=\frac{\lambda_n-\lambda_{n+1}}{2}\downarrow 0, \end{equation*}$$

and put $z_n=\lambda_ne_n-\lambda_{n+1}e_{n+1}$.

Remark 1.  The set

$$\begin{equation*} M=\biggl(\bigcup_{n=1}^\infty \varepsilon_nS_n\biggr)\cup\{\pm z_n\}_{n=1}^\infty\cup\{0\} \end{equation*}$$

is symmetric and proximal in $l_2$. It satisfies condition (4), but the algorithm (2) of greedy approximation by elements of $M$ diverges for the initial element $x=e_1$.

Indeed, for every non-zero $x$ we have $\rho(x,\varepsilon_nS_n)<\|x\|$ for all sufficiently large $n$, whence $\rho(x,M)<\|x\|$ and $M$ is norm-reducing. Moreover,

$$\begin{equation*} \begin{gathered} \, \rho(x,\varepsilon_nS_n)\to \|x\| \quad (n\to \infty), \\ \|x-(\pm z_n)\|\to \sqrt{\|x\|^2+\frac12}>\|x\| \quad (n\to \infty), \end{gathered} \end{equation*}$$

and it follows from the compactness of the sets $\varepsilon_nS_n$ that $P_M(x)\ne \varnothing$. Since $P_M(0)=\{0\}$, we can see that $M$ is proximal.

For $x_0=e_1$, we have

$$\begin{equation*} \begin{gathered} \, \|x_0-z_1\|=\|\lambda_2e_2\|=\lambda_2<1, \\ \|x_0-(-z_1)\|>2,\qquad \|x_0-(\pm z_n)\|>1 \quad \text{for} \quad n=2,3,\dots, \\ \rho(x_0,\varepsilon_nS_n)=1-\varepsilon_n\ge 1-\varepsilon_1=1-\frac{\lambda_1-\lambda_2}{2}>\lambda_2, \end{gathered} \end{equation*}$$

whence $P_M(x_0)=\{z_1\}$, and the greedy algorithm (2) gives $x_1=x_0-z_1=\lambda_2e_2$ at the first step.

We now prove by induction that the algorithm produces $x_{k}=x_{k-1}-z_k=\lambda_{k+1}e_{k+1}$ at the $k$th step. Indeed, for $x_{k-1}=\lambda_ke_k$ we have

$$\begin{equation*} \begin{gathered} \, \|x_{k-1}-z_k\|=\|\lambda_{k+1}e_{k+1}\|=\lambda_{k+1}, \\ \|x_{k-1}-(-z_k)\|>2\lambda_{k}>\lambda_{k+1},\qquad \|x_{k-1}-(\pm z_{k-1})\|\ge \lambda_{k-1}>\lambda_{k+1}, \\ \|x_{k-1}-z_n\|>\lambda_k>\lambda_{k+1} \quad \text{for} \quad n\ne k, k-1, \\ \rho(x_{k-1},\varepsilon_nS_n)>\lambda_k>\lambda_{k+1}\quad \text{for} \quad n<k, \\ \rho(x_{k-1},\varepsilon_nS_n)=\lambda_k-\varepsilon_n\ge \lambda_k-\varepsilon_k=\lambda_k-\frac{\lambda_k-\lambda_{k+1}}{2}>\lambda_{k+1} \quad \text{for} \quad n\ge k. \end{gathered} \end{equation*}$$

Therefore $P_M(x_{k-1})=\{z_k\}$, and $x_{k}=x_{k-1}-z_k=\lambda_{k+1}e_{k+1}$.

Since the $\lambda_k$ do not tend to zero, it turns out that the greedy algorithm diverges.

The condition that $M$ should be a proximal set, which is needed for the greedy algorithm (2), is a fairly strong one. Therefore, it is reasonable to relax the requirements for the sequence of approximants $y_n$ so that the algorithm would also work for non-proximal (in particular, non-closed) sets.

Given a norm-reducing set $M$ in a Banach space $X$ such that $0\in M$, we define the weak greedy algorithm by associating the following system with every non-zero element $x=x_0\in X$:

$$\begin{equation} \begin{gathered} \, x_{n+1}=x_n-y_{n+1}, \nonumber \\ \text{where} \quad y_{n+1}\in M, \quad \|x_n-y_{n+1}\|\le \frac{\rho(x_n,M)+\|x_n\|}{2}, \qquad n=0,1,\dots \end{gathered} \end{equation} \tag{ 5 }$$

(we can choose any element $y_{n+1}$ satisfying these conditions).

In the case of a Hilbert space and $M=\Lambda(D)$, where $D$ is a dictionary, this algorithm roughly corresponds to the weak greedy algorithm in [1], Ch. 2 (for the correspondence to be exact one should take the quadratic mean instead of the arithmetic mean in (5); however, the quadratic mean would look unnatural for an arbitrary Banach space). The arithmetic mean in (5) can of course be replaced by an arbitrary convex combination of $\rho(x_n,M)$ and $\|x_n\|$ with fixed coefficients playing the role of weakness parameters.

Theorem 1.  Let $M\ni 0$ be a symmetric all-round set in a Hilbert space $H$ such that $M/2\subset M$ (that is, for every $y\in M$ the element $y/2$ is also in $M$). Then the weak greedy algorithm (5) converges for each $x\in H$.

Here we write 'all-round' instead of 'norm-reducing' because it is the same under the condition $M/2\subset M\subset H$. Instead of $M/2\subset M$, one can write $M/q\subset M$ for any fixed $q>1$. Remark 1 shows that the hypothesis $M/2\subset M$ in Theorem 1 cannot be omitted. The condition $0\in M$ is not burdensome. It guarantees that the algorithm does not stop.

Many algebraic (but not geometric) aspects of the following proof go back to [2].

Proof.  1) We take an arbitrary initial element $x=x_0$. For every $n=0,1,2,\dots$, we have

$$\begin{equation*} \|x_{n+1}\|\le\frac{\rho(x_n,M)+\|x_n\|}{2}\le \|x_n\|, \end{equation*}$$

so that the sequence $\|x_n\|$ decreases (non-strictly if $x_n=0$). The sequence

$$\begin{equation*} \varepsilon_n=\|x_n\|-\|x_{n+1}\| \end{equation*}$$

enjoys the following properties:

$$\begin{equation} \varepsilon_n\ge 0, \qquad \sum_{n=1}^\infty \varepsilon_n <\infty. \end{equation} \tag{ 6 }$$

Hence the sequence

$$\begin{equation*} \alpha_n=\|x_n\|-\rho(x_n,M)=2\|x_n\|-(\rho(x_n,M)+\|x_n\|)\le 2\|x_n\|-2\|x_{n+1}\|=2\varepsilon_n \end{equation*}$$

is such that the series $\sum_{n=1}^\infty \alpha_n$ also converges, and we have $\alpha_n n \to 0$ along some subsequence $\Lambda\subset {\mathbb N}$ of the values of $n$.

2) Since $y_{n+1}/2\in M$, we have

$$\begin{equation*} \begin{aligned} \, &\|x_n-y_{n+1}\|^2\le \biggl(\frac{\rho(x_n,M)+\|x_n\|}{2}\biggr)^2\le \biggl(\frac{\|x_n-y_{n+1}/2\|+\|x_n\|}{2}\biggr)^2 \\ &\ \le \frac{\|x_n-y_{n+1}/2\|^2+\|x_n\|^2}{2}=\frac{\|x_n\|^2}{2}+ \frac{1}{8}(2\|x_n-y_{n+1}\|^2+2\|x_n\|^2-\|y_{n+1}\|^2) \\ &\ \Longleftrightarrow \|y_{n+1}\|^2\le 6(\|x_n\|^2-\|x_n-y_{n+1}\|^2)=6(\|x_n\|^2-\|x_{n+1}\|^2) \\ &\ \qquad\qquad =6(\|x_n\|+\|x_{n+1}\|)(\|x_n\|-\|x_{n+1}\|)\le 12\|x\|\varepsilon_n, \end{aligned} \end{equation*}$$

whence, in view of (6), we obtain

$$\begin{equation} \sum_{n=1}^\infty \|y_n\|^2< \infty. \end{equation} \tag{ 7 }$$

3) Consider two numbers $n,m\in \Lambda$ such that $n<m$. We have

$$\begin{equation} \|x_n-x_m\|^2=\|x_n\|^2-\|x_m\|^2-2(x_n-x_m,x_m). \end{equation} \tag{ 8 }$$

Let us rewrite the last term as

$$\begin{equation} (x_n-x_m,x_m)=\biggl(\sum_{j=n+1}^{m}y_j,x_m\biggr)=\sum_{j=n+1}^{m}\|y_j\| (\sigma_j,x_m), \end{equation} \tag{ 9 }$$

where $\sigma_j=y_j/\|y_j\|$. We distinguish three types among all the indices $j\in \{n+ 1,\dots, m\}$.

I. $|(\sigma_j,x_m)|\le \|y_j\|$. For any such $j$, we have

$$\begin{equation} \|y_j\| |(\sigma_j,x_m)|\le \|y_j\|^2. \end{equation} \tag{ 10 }$$

II. $|(\sigma_j,x_m)|> \|y_j\|$ and the line with direction $\sigma_j$ intersects the open ball $B_m=B(x_m,\rho(x_m,M))$; see Fig. 1.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Since $\pm y_j\in M$, both these points are outside the ball $B_m$. Then $y_j$ lies in the closed interval $[-a,a]$ in view of the inequality $\|y_j\|<|(\sigma_j,x_m)|$, and

$$\begin{equation} \|y_j\| |(\sigma_j,x_m)|< \|a\|\|b\|=k^2=\|x_m\|^2-(\|x_m\|-\alpha_m)^2\le 2\|x_m\|\alpha_m\le 2\|x\|\alpha_m \end{equation} \tag{ 11 }$$

(the first equality holds by the high-school theorem on the square of the tangent).

III. $|(\sigma_j,x_m)|> \|y_j\|$ and the line with direction $\sigma_j$ does not intersect the open ball $B_m$; see Fig. 2.

In this case,

$$\begin{equation} |(\sigma_j,x_m)|\le k<\sqrt{2\|x_m\| \alpha_m}\quad \Longrightarrow\quad \|y_j\| |(\sigma_j,x_m)|\le \sqrt{2\|x\|\alpha_m}\, \|y_j\|. \end{equation} \tag{ 12 }$$

4) In accordance with this partition of the values of $j$ into classes I, II, and III, the sum in (8) splits into three sums, for which the estimates (10)–(12) prepared above give

$$\begin{equation*} \begin{aligned} \, \sum_{I}+\sum_{\mathrm{II}}+\sum_{\mathrm{III}} &\le \sum_{\mathrm{I}} \|y_j\|^2+ \sum_{\mathrm{II}} 2\|x\| \alpha_m +\sum_{\mathrm{III}} \sqrt{2\|x\|\alpha_m}\,\|y_j\| \\ &\le \sum_{j=n+1}^\infty \|y_j\|^2 +2\|x\|m\alpha_m + \sqrt{2\|x\|\alpha_m} \sum_{j=n+1}^{m}\|y_j\|. \end{aligned} \end{equation*}$$

It is easy to see that the first two terms in the last expression tend to zero as $n,m\to\infty$, $n,m\in \Lambda$, because of (7) and the choice of $\Lambda$. The third term also tends to zero:

$$\begin{equation*} \begin{aligned} \, &\sqrt{\alpha_m} \sum_{j=n+1}^{m}\|y_j\| = \sqrt{\alpha_m}\, (m-n) \biggl( \frac{\sum_{j=n+1}^{m}\|y_j\|}{m-n}\biggr) \\ &\qquad\le \sqrt{\alpha_m}\, (m-n) \biggl( \frac{\sum_{j=n+1}^{m}\|y_j\|^2}{m-n}\biggr)^{1/2} \le \sqrt{m\alpha_m}\, \biggl( \sum_{j=n+1}^{m}\|y_j\|^2\biggr)^{1/2}\to 0 \end{aligned} \end{equation*}$$

as $n,m\to\infty$, $n,m\in \Lambda$.

5) Thus the quantity (9) tends to zero as $n,m\in \Lambda$ tend to $\infty$. Hence $\{x_n\}_{n\in \Lambda}$ is a Cauchy sequence in view of (8). Let $z\in H$ be its limit. Clearly,

$$\begin{equation} \lim_{n\to \infty} \|x_n\| =\|z\|. \end{equation} \tag{ 13 }$$

If $z\ne 0$, then $\rho(z,M)<(1-\delta)\|z\|$ for some $\delta>0$ because $M$ is norm-reducing (we have already mentioned that this follows since $M$ is all-round and $M/2\subset M$). Therefore,

$$\begin{equation*} \|x_{n+1}\|\le \frac{\rho(x_n,M)+\|x_n\|}{2}<\biggl(1-\frac{\delta}2\biggr)\|x_n\| \end{equation*}$$

for all sufficiently large $n\in \Lambda$. Hence $\|x_{n+1}\|<(1-\delta/2)\|z\|$ for any sufficiently large $n\in \Lambda$. This contradicts (13).

As a result, we have $z=0$ and $x_n\to 0$ ($n\in \Lambda$), hence $x_n\to 0$ ($n\to \infty$) in view of the monotonicity of $\|x_n\|$ (see part 1 of the proof). $\Box$

Corollary 1.  Let $M$ be a symmetric all-round set in a Hilbert space $H$. Then any element $x\in H$ can be represented as a series

$$\begin{equation} x=\sum_{k=1}^\infty \frac{z_k}{2^{n_k}}, \end{equation} \tag{ 14 }$$

where $z_k\in M$, $n_k\in\{0,1,2,\dots\}$.

Proof.  The set

$$\begin{equation*} M'=\{0\}\cup \biggl(\,\bigcup_{n=0}^\infty M/2^n\biggr) \end{equation*}$$

is also symmetric and all-round, and $M'/2\subset M'$. Thus $M'$ satisfies the hypotheses of Theorem 1 and, by this theorem, every element $x\in H$ can be represented as a series $x=\sum_{k=1}^\infty y_k$ with $y_k\in M'$, that is, as a series (14). $\Box$

Theorem 2.  Let $M\ni 0$ be a norm-reducing set in a Hilbert space $H$. For each $x\in H$, the residuals $x_n$ in the weak greedy algorithm (5) converge weakly to zero.

Proof.  We have already seen in part 1 of the proof of Theorem 1 that $\|x_n\|$ is a decreasing sequence. Hence $\lim_{n\to\infty}\|x_n\|=R\le \|x\|$. We argue by contradiction. Suppose that $R>0$. The bounded sequence $\{x_n\}\subset H$ contains a weakly converging subsequence. Suppose that the elements $x_n$ converge weakly to $z\ne 0$ along some sequence $\Lambda$ of the values of $n$. Clearly, $\|z\|\le R$. By hypothesis, there is an element $y\in M$ such that $\|z-y\|<\|z\|-\varepsilon$ for some $\varepsilon>0$.

All of this takes place in a separable part of $H$, which can be identified with the space $l_2$. Consider the coordinate projections

$$\begin{equation*} \begin{aligned} \, P_N\bigl(a=(a_1,a_2,\dots)\bigr)&=(a_1,\dots,a_N,0,0,\dots), \\ R_N\bigl(a=(a_1,a_2,\dots)\bigr)&=(0,\dots,0,a_{N+1}, a_{N+2},\dots) \end{aligned} \end{equation*}$$

available in $l_2$.

There is an $N$ such that

$$\begin{equation} \|P_N(z)\|>\varepsilon, \end{equation} \tag{ 15 }$$

$$\begin{equation} \|P_N(z)-P_N(y)\|<\|P_N(z)\|-\frac{\varepsilon}2, \end{equation} \tag{ 16 }$$

$$\begin{equation} \|R_N(y)\|<\frac{\|P_N(z)\|\varepsilon}{8\|x\|}. \end{equation} \tag{ 17 }$$

Since weak convergence in $l_2$ implies coordinate convergence, we can find a number $n\in \Lambda$ such that, for this $N$,

$$\begin{equation} \|P_N(x_n)-P_N(z)\|<\frac{\varepsilon}{8}, \end{equation} \tag{ 18 }$$

$$\begin{equation} \|x_n\|^2< R^2+\frac{\varepsilon^2}{20}. \end{equation} \tag{ 19 }$$

Applying (16) and (18), we get

$$\begin{equation*} \begin{aligned} \, &\|P_N(x_n)-P_N(y)\|\le \|P_N(x_n)-P_N(z)\|+\|P_N(z)-P_N(y)\| \\ &\qquad\le\frac{\varepsilon}{8}+\|P_N(z)\|-\frac{\varepsilon}{2}\le \frac{\varepsilon}{8}+\frac{\varepsilon}{8}+\|P_N(x_n)\|-\frac{\varepsilon}{2} =\|P_N(x_n)\|-\frac{\varepsilon}{4}. \end{aligned} \end{equation*}$$

Combining this with (17), (18), (15) and (19), we have

$$\begin{equation*} \begin{aligned} \, \|x_n-y\|^2&=\|P_N(x_n)-P_N(y)\|^2+\|R_N(x_n)-R_N(y)\|^2 \\ &\le\biggl(\|P_Nx_n\|-\frac{\varepsilon}{4}\biggr)^2 +\biggl(\|R_N(x_n)\|+\frac{\|P_N(z)\|\varepsilon}{8\|x\|}\biggr)^2 \\ &=\|x_n\|^2+\frac{\varepsilon^2}{16}-\frac{\varepsilon}{2}\|P_N(x_n)\| +\frac{\|P_N(z)\|^2\varepsilon^2}{64\|x\|^2} +\frac{\|R_N(x_n)\|\|P_N(z)\|\varepsilon}{4\|x\|} \\ &\le \|x_n\|^2+\frac{\varepsilon^2}{16} -\frac{\varepsilon}{2}\biggl(\|P_N(z)\|-\frac{\varepsilon}{8}\biggr) +\frac{\|z\|^2}{\|x\|^2}\cdot \frac{\varepsilon^2}{64}+\frac{\|x_n\|}{\|x\|}\cdot \|P_N(z)\|\cdot\frac{\varepsilon}{4} \\ &\le \|x_n\|^2+\frac{\varepsilon^2}{16}+\frac{\varepsilon^2}{16} -\|P_N(z)\|\cdot\frac{\varepsilon}{2}+\frac{\varepsilon^2}{64} +\|P_N(z)\|\cdot\frac{\varepsilon}{4} \\ &=\|x_n\|^2+\frac{9\varepsilon^2}{64}-\|P_N(z)\|\cdot\frac{\varepsilon}{4}\le \|x_n\|^2+\frac{9\varepsilon^2}{64}-\frac{\varepsilon^2}{4} \\ &=\|x_n\|^2-\frac{7\varepsilon^2}{64}< R^2+\frac{\varepsilon^2}{20}-\frac{\varepsilon^2}{10}. \end{aligned} \end{equation*}$$

Consequently, $\rho^2(x_n,M)\le\|x_n-y\|^2<R^2-\varepsilon^2/20$, hence

$$\begin{equation*} \|x_{n+1}\|\le \frac{\rho(x_n,M)+\|x_n\|}{2} \le\biggl(\frac{\rho^2(x_n,M)+\|x_n\|^2}{2}\biggr)^{1/2}<R. \end{equation*}$$

This contradicts the inequality $\|x_k\|\ge R$.

Thus, every weak partial limit of $x_n$ is $0$, so this sequence converges weakly to zero. $\Box$

Remark 2.  For any norm-reducing set $M$ in a finite-dimensional normed space $X$, the weak greedy algorithm (5) converges for every $x\in X$.

Indeed, $\|x_n\|$ is a decreasing sequence bounded by $\|x\|$. Therefore $x_n\to z$ along some subsequence of the values of $n$. Similarly to part 5 of the proof of Theorem 1, we get $z=0$, hence $x_n\to 0$ ($n\to \infty$).

If one is not too "greedy", it is possible to get convergence without the condition $M/2\subset M$ needed in Theorem 1.

Given a norm-reducing set $M\ni 0$ in an arbitrary Banach space $X$, we define the weak semi-greedy algorithm by associating the following sequence with every $x=x_0\in X$:

$$\begin{equation} \begin{gathered} \, x_{n+1}=x_n-y_{n+1}, \qquad n=1,2,\dots, \nonumber \\ \text{where}\quad y_{n+1}\in M,\quad \biggl\|\frac{x_n}2-y_{n+1}\biggr\|\le \frac{\rho(x_n/2,M)+\|x_n/2\|}{2} \end{gathered} \end{equation} \tag{ 20 }$$

(we select any element $y_{n+1}$ that meets these conditions).

If, in addition, $M$ is a proximal set, one can define a pure semi-greedy algorithm with $y_{n+1}\in P_M(x_n/2)$ instead of (20). In the case of $M=\Lambda(D)$, where $D$ is a dictionary in a Hilbert space, this pure semi-greedy algorithm coincides with a particular type of the weak greedy algorithm in [1], Ch. 2, with weakness coefficient $t=1/2$. However, this coincidence is only formal. Essentially, the semi-greedy algorithm differs fundamentally from all known greedy algorithms: we subtract from $x_n$ not the best or, in some sense, almost best approximation of $x_n$, but the best or almost best approximation of $x_n/2$. For arbitrary sets $M$, the semi-greedy algorithm approximation is worse at every step than the algorithms (2) and (5), but it turns out to be more reliable in the sense of convergence.

Theorem 3.  Let $M$ be a symmetric norm-reducing set in a Hilbert space $H$ with $0\,{\in}\,M$. Then the weak semi-greedy algorithm (20) converges for each $x\,{\in}\,H$.

Proof.  We follow the same scheme as in the proof of Theorem 1, but with some significant modifications.

1) Take any non-zero element $x=x_0$. If $x_n\ne 0$, then $\rho(x_n/2,M)\,{<}\,\|x_n\|/2$ and $y_{n+1}$ lies inside the ball $B(x_n/2, \|x_n\|/2)$, which has the closed interval $[0,x_n]$ as a diameter. Consequently, $y_{n+1}\ne x_n$, so that $x_{n+1}\ne 0$ (for all $n$), and

$$\begin{equation*} \|x_{n+1}\|=\|x_n-y_{n+1}\|<\|x_n\|. \end{equation*}$$

Hence the values $\|x_n\|$ decrease.

The sequence $\varepsilon_n=\|x_n\|-\|x_{n+1}\|$ satisfies

$$\begin{equation*} \varepsilon_n>0, \qquad \sum_{n=0}^\infty\varepsilon_n<\infty. \end{equation*}$$

Hence the sequence

$$\begin{equation*} \begin{aligned} \, \alpha_n&:=\biggl\|\frac{x_n}2\biggr\|-\rho\biggl(\frac{x_n}2,M\biggr)\le \biggl\|\frac{x_n}2\biggr\|-\biggl(2\biggl\|\frac{x_n}2-y_{n+1}\biggr\| -\biggl\|\frac{x_n}2 \biggr\|\biggr) \\ &\hphantom{:}=\|x_n\|-2\biggl\|\frac{x_n}2-y_{n+1}\biggr\| \le \|x_n\|-2\biggl(\|x_n-y_{n+1}\|-\biggl\|\frac{x_n}2\biggr\|\biggr) \\ &\hphantom{:}=2\|x_n\|-2\|x_{n+1}\|=2\varepsilon_n \end{aligned} \end{equation*}$$

is such that

$$\begin{equation*} \sum_{n=0}^\infty \alpha_n<\infty. \end{equation*}$$

It follows that $\alpha_n n\to 0$ along some subsequence $\Lambda\subset\mathbb N$ of the values of $n$.

2) Since $y_{n+1}$ is in the ball with diameter $[0,x_n]$, the angle $\angle\, 0y_nx_n$ is greater than $90^\circ$. Therefore,

$$\begin{equation*} \|y_{n+1}\|^2<\|x_n\|^2-\|x_n-y_{n+1}\|^2=2(\|x_n\|+\|x_{n+1}\|)(\|x_n\|-\|x_{n+1}\|) =2\|x\|\varepsilon_n, \end{equation*}$$

so that

$$\begin{equation} \sum_{n=1}^\infty \|y_n\|^2<\infty. \end{equation} \tag{ 21 }$$

3) We take two numbers $n,m\in \Lambda$ with $n<m$. As in the proof of Theorem 1, we estimate the scalar product

$$\begin{equation} (x_n-x_m,x_m)=\biggl(\sum_{j=n+1}^{m}y_j,x_m\biggr)=\sum_{j=n+1}^{m}\|y_j\| (\sigma_j,x_m), \end{equation} \tag{ 22 }$$

where $\sigma_j=y_j/\|y_j\|$.

We distinguish three types among all the indices $j\in \{n+1,\dots, m\}$.

I. $|(\sigma_j,x_m/2)|\le \|y_j\|$. For any such $j$, we have

$$\begin{equation} \|y_j\| |(\sigma_j,x_m)|\le 2\|y_j\|^2. \end{equation} \tag{ 23 }$$

II. $|(\sigma_j,x_m/2)|> \|y_j\|$ and the line with direction $\sigma_j$ intersects the ball

$$\begin{equation*} B_m=B\biggl(\frac{x_m}2, \rho\biggl(\frac{x_m}2,M\biggr)\biggr). \end{equation*}$$

Just as in part 3 of the proof of Theorem 1 (one should replace $x_m$ by $x_m/2$ in Fig. 1 and in (11)), in this case we obtain

$$\begin{equation} \|y_j\| |(\sigma_j,x_m)|\le 2\|x\|\alpha_m. \end{equation} \tag{ 24 }$$

III. $|(\sigma_j,x_m/2)|> \|y_j\|$ and the line with direction $\sigma_j$ does not intersect the ball $B_m$. Just as in part 3 of the proof of Theorem 1 (one should replace $x_m$ by $x_m/2$ in Fig. 2 and in (12)), in this case we obtain

$$\begin{equation} \|y_j\| |(\sigma_j,x_m)|\le 2\sqrt{\|x\|\alpha_m}\, \|y_j\|. \end{equation} \tag{ 25 }$$

4. As in part 4 of the proof of Theorem 1, we use (23)–(25) to bound the absolute value of (22) by the sum

$$\begin{equation} 2\sum_{j=n+1}^\infty \|y_j\|^2+2\|x\|m\alpha_m+2\sqrt{\|x\|m\alpha_m} \sum_{j=n+1}^{m}\|y_j\|^2. \end{equation} \tag{ 26 }$$

In view of (22) and the choice of $\Lambda$, this sum tends to zero as $n,m\to\infty$, $n,m\in \Lambda$.

5. By (8), $\{x_n\}_{n\in \Lambda}$ is a Cauchy sequence. Repeating the arguments in part 5 of the proof of Theorem 1, we find that $x_n\to 0$ as $n\to \infty$. $\Box$

Corollary 2.  Let $M$ be a norm-reducing symmetric set in a Hilbert space $H$. Then every element $x\in H$ has a representation as a series

$$\begin{equation} x=\sum_{n=1}^\infty y_n, \end{equation} \tag{ 27 }$$

where $y_n\in M$ with $\sum_{n=1}^\infty \|y_n\|^2<\infty$.

It would be interesting to specify a non-trivial class of sets in this corollary for which the series (27) is absolutely convergent.

Theorem 2 remains valid for the modified algorithm (20) with $\theta x_n$ taken instead of $x_n/2$, where $\theta$ is any fixed value in $(0,1/2]$.

The symmetry condition on $M$ in Theorems 1 and 3 seems to be significant, although the author is not aware of any corresponding examples. In order to obtain convergence without symmetry, we need to change the algorithm in a natural way.

In the case of approximation by an asymmetric norm-reducing set $M$ it is convenient to use the recursive greedy algorithm described below. It is a modification of the algorithm invented by Livshitz [9] in the case when $M=\Lambda(D)$, where $D$ is a dictionary, to speed up the convergence rate compared to algorithm (1). In [10] Livshitz also applied this algorithm to an approximation by the set $\Lambda_+(D)=\{\lambda g\colon \lambda\ge 0,\, g\in D\}$, where $D$ is, generally speaking, an asymmetric dictionary in a Hilbert space.

Let $M$ be a norm-reducing set in an arbitrary Banach space $X$, and let $0\in M$. We define the recursive weak greedy algorithm (RWGA) and recursive weak semi-greedy algorithm (RWSGA) by associating with every $x=x_0\in X$ a sequence

$$\begin{equation*} x_n=x_{n-1}-y_n=x_0-y_1-\dots-y_n, \qquad n=1,2,\dots, \end{equation*}$$

according to the following rules.

For each $k$, the element $y_k$ either belongs to $M$ or it belongs to $-M$, and in the latter case there is a $j<k$ such that $y_j=-y_k\in M$ and $y_m\ne y_k$ for all $m=j+1,\dots,k-1$. Then $x_n$ (to be defined by induction) can be written as

$$\begin{equation*} x_n=x_0-z_1^{(n)}-\dots-z_{\nu_n}^{(n)}, \end{equation*}$$

where $z_j^{(n)}\in M$. We put $M_n=M\cup \{-z_1^{(n)},\dots,-z_{\nu_n}^{(n)}\}$ and finally define the algorithm step

$$\begin{equation*} x_{n+1}=x_n-y_{n+1}, \qquad n=0,1,2,\dots, \end{equation*}$$

where $y_{n+1}\in M_n$ and

$$\begin{equation*} \begin{alignedat}{2} \|x_n-y_{n+1}\|&\le \frac{\|x_n\|+\rho(x_n,M_n)}{2} &\quad &\text{in the case of\/ RWGA}, \\ \biggl\|\frac{x_n}2-y_{n+1}\biggr\|&\le \frac{\|x_n/2\|+\rho(x_n/2,M_n)}{2} &\quad &\text{in the case of\/ RWSGA} \end{alignedat} \end{equation*}$$

(such an element $y_{n+1}$ exists since $M_n$ is norm-reducing and $0\in M$).

When $M$ is proximal, all the $M_n$ are also proximal. Then one can define the recursive pure greedy and recursive pure semi-greedy algorithms by choosing $y_{n+1}\in P_{M_n}(x_n)$ and $y_{n+1}\in P_{M_n}(x_n/2)$, respectively.

If the RWGA or RWSGA converges, that is, $x_n\to 0$, then $x$ can be represented as a series $\sum_{n=1}^\infty y_n$ of elements in $M\cup (-M)$ all of whose partial sums are sums of elements in $M$.

Theorem 4.  Let $M$ be a norm-reducing set in a Hilbert space $H$, and let $0\in M$. Then the recursive weak semi-greedy algorithm converges for every $x\in H$. If, moreover, $M/2\subset M$, then the recursive weak greedy algorithm also converges for each $x\in H$.

Proof.  This proof also follows the scheme of the proof of Theorem 1.

1) Clearly, the norms $\|x_n\|$ are decreasing, and the values $\varepsilon_n=\|x_n\|-\|x_{n-1}\|$ satisfy $\sum_{n=1}^\infty \varepsilon_n <\infty$.

Similarly to part 1 of the proof of Theorem 1 and of Theorem 3, we prove that $\sum_{n=1}^\infty \!\alpha_n \!< \!\infty$, where

$$\begin{equation*} \begin{alignedat}{2} \alpha_n&=\|x_n\|-\rho(x_n,M_n) &\quad &\text{in the case of\/ RWGA}, \\ \alpha_n&=\biggl\|\frac{x_n}2\biggr\|-\rho\biggl(\frac{x_n}2,M_n\biggr) &\quad &\text{in the case of\/ RWSGA}. \end{alignedat} \end{equation*}$$

We choose a subsequence $\Lambda\subset {\mathbb N}$ of $n$'s such that $\alpha_n n\to 0$.

2) Similarly to part 2 of the proof of Theorem 1 and of Theorem 3, we get

$$\begin{equation} \sum_{n=1}^\infty \|y_n\|^2 <\infty. \end{equation} \tag{ 28 }$$

3) Consider two numbers $n,m\in \Lambda$ such that $n<m$. We now give only an upper bound for $-(x_n-x_m,x_m)$ from (8), rather than a bound for the modulus: if this upper bound tends to zero as $n,m\to \infty$, $n,m\in \Lambda$, it will be sufficient for $\{x_n\}_{n\in \Lambda}$ to be a Cauchy sequence.

We have

$$\begin{equation*} -(x_n-x_m,x_m)=-\sum_{j=n+1}^m(y_j,x_m)=-\sum\nolimits'(y_j,x_m)=-\sum\nolimits'\|y_j\|(\sigma_j,x_m), \end{equation*}$$

where $\sigma_j=y_j/\|y_j\|$ as above, while the sum $\sum\nolimits'$ is taken over those $j\in\{n+ 1,\dots,m\}$ for which we have either

(a) $y_j\in M$ and $y_k\ne -y_j$ for all $k=j+1,\dots,m$ (so that $-y_j\in M_m$),

or

(b) $y_j\in (-M)$ and $y_k\ne -y_j$ for all $k=n+1,\dots,j-1$.

The terms corresponding to the remaining indices $j$ split into pairs of opposite values which cancel one another.

Next, we have

$$\begin{equation} -\sum\nolimits'\|y_j\|(\sigma_j,x_m)\le -\sum\nolimits''\|y_j\|(\sigma_j,x_m), \end{equation} \tag{ 29 }$$

where the sum $\sum\nolimits''$ contains only terms for which $(\sigma_j,x_m)<0$.

We split these terms into three types.

I. $-\|y_j\|\le(\sigma_j,x_m)\le 0$ in the case of RWGA or $-\|y_j\|\le(\sigma_j,x_m/2)\le 0$ in that of RWSGA. For such terms, we have

$$\begin{equation} -\|y_j\|(\sigma_j,x_m)\le 2\|y_j\|^2. \end{equation} \tag{ 30 }$$

II. $(\sigma_j,x_m)< -\|y_j\|$ in the case of RWGA or $(\sigma_j,x_m/2)< -\|y_j\|$ in that of RWSGA, and the line with direction $\sigma_j$ intersects the open ball

$$\begin{equation*} \begin{alignedat}{2} B_m&=B(x_m,\rho(x_m,M_m)) &\quad &\text{in the case of\/ RWGA}, \\ B_m&=B\biggl(\frac{x_m}2,\rho\biggl(\frac{x_m}2,M_m\biggr)\biggr) &\quad &\text{in the case of\/ RWSGA}. \end{alignedat} \end{equation*}$$

Since $(\sigma_j,x_m)<0$, the point $y_j$ lies in the lower part of the line in Fig. 1 (in the case of RWSGA, $x_m$ is replaced by $x_m/2$ in this figure).

In case (a) we have $-y_j\in M_m$, therefore $-y_j\notin B_m$, which together with $\|-y_j\|<-(\sigma_j,x_m)=|(\sigma_j,x_m)|$ implies $-y_j\in [0,a]$, and so $\|y_j\|\le \|a\|$.

In case (b) we have $-y_j\in M\subset M_m$, therefore $-y_j\notin B_m$, and we similarly get $-y_j\in [0,a]$ and $\|y_j\|\le \|a\|$.

Consequently, in both cases,

$$\begin{equation} -\|y_j\| (\sigma_j,x_m)\le\|a\|\cdot\|b\|=k^2\le 2\|x\|\alpha_m. \end{equation} \tag{ 31 }$$

For the details of the last step, see part 3 of the proof of Theorem 1.

III. $(\sigma_j,x_m)< -\|y_j\|$ in the case of RWGA or $(\sigma_j,x_m/2)< -\|y_j\|$ in that of RWSGA, and the line with direction $\sigma_j$ does not intersect $B_m$. Similarly to part 3 of the proof of Theorem 1, in accordance with Fig. 2 (in the case of RWSGA, $x_m$ is replaced by $x_m/2$ in this figure) we obtain

$$\begin{equation} -\|y_j\| (\sigma_j,x_m)\le 2\|y_j\| \sqrt{\|x\|\alpha_m} \end{equation} \tag{ 32 }$$

(in the case of RWGA, the estimate is even better: $2$ appears under the square root sign).

4. Similarly to part 4 of the proof of Theorem 1, we use (30)–(32) to bound the sum in (29) by the expression (26). In view of (28) and the choice of $\Lambda$, this expression tends to zero as $n,m\to 0$, $n,m \in \Lambda$.

5. By (8), $\{x_n\}_{n\in\Lambda}$ turns out to be a Cauchy sequence. Repeating the arguments in part 5 of the proof of Theorem 1 and using the inequalities $\rho(z,M_n)\le \rho(z,M)< \|z\|$, which hold for all non-zero $z\in H$, we obtain $x_n\to 0$ as $n\to \infty$. $\Box$

The main problem of this theory [3] is to find conditions on a subset $M$ of a Banach space $X$ which are necessary or sufficient for the set

$$\begin{equation*} R(M)=\bigcup_{n=1}^\infty \underbrace{M+\dots +M}_{n} \end{equation*}$$

(the additive semigroup generated by $M$) to be dense in $X$ (that is, every element of $X$ can be approximated with arbitrary accuracy by finite sums of elements of $M$). A necessary condition for $R(M)$ to be dense in $X$ is that $M$ should be all-round; see [3]. There are examples of all-round sets $M$ in a Hilbert space $H$ such that the closure $\overline{R(M)}$ is not even an additive subgroup of $H$; see Example 1 in [3].

Condition (4) that $M$ should be norm-reducing is clearly not necessary for $R(M)$ to be dense in $H$, but it turns out to be sufficient for this to be so.

Theorem 5.  For any norm-reducing set $M$ in a Hilbert space $H$, the set $R(M)$ is dense in $H$.

Proof.  We complement $M$ by the zero element, take an arbitrary $x\in H$ and run the recursive weak semi-greedy algorithm for $x$. It converges by Theorem 4. Hence $x$ can be represented as a series $\sum_{n=1}^\infty y_n$ of elements of $M\cup(-M)$ all of whose partial sums $s_k=\sum_{n=1}^ky_n$ are sums of elements of $M$ by the definition of the recursive algorithm. Thus $R(M)\ni s_k\to x$ and, therefore, $\overline{R(M)}=H$. $\Box$

It should be possible to generalise Theorem 5 for some class of Banach spaces. Nonetheless the theorem does not hold for an arbitrary Banach space.

Remark 3.  Each non-reflexive space $X$ contains a norm-reducing symmetric set $M$ such that $\overline{R(M)}\ne X$.

Indeed, by James' theorem ([11], Ch. 1), there is a functional $f\in X^*$ not attaining its norm: $|f(x)|\ne \|f\|\cdot \|x\|$ for any non-zero $x\in X$. The kernel $\ker f$ has the property that the metric projection $P_{\ker f}(x)$ is empty for every $x\in\ker f$. Indeed, if $y\in P_{\ker f}(x)$, then for $q=x-y\ne 0$ we have $P_{\ker f}(q)\ni 0$ and

$$\begin{equation*} \|q\|=\|q-0\|=\rho(q,\ker f)=\frac{|f(q)|}{\|f\|} \end{equation*}$$

(for the last equality, see, for example, [12], Ch. 1), contradicting the fact that $f$ does not attain its norm. It turns out that for any $x\notin \ker f$ there is no nearest element in $\ker f$, hence there is a $y\in \ker f$ such that $\|x-y\|<\|x-0\|=\|x\|$. Thus $\ker f$ is the desired norm-reducing set.

We outline several questions that naturally arise in connection with the results presented here.

1) Is it possible to omit the symmetry condition on $M$ in Theorems 1 and 3? In particular, does the greedy algorithm (2) or the weak greedy algorithm (5) converge in the case when $M=\Lambda_+(D)=\{\lambda g\colon\lambda\ge 0, g\in D\}$, where $D$ is (generally speaking) an asymmetric all-round dictionary in a Hilbert space? Reference [10] suggests that its author was also interested in the last question.

2) Is it possible to extend Theorems 1–5 to some class of Banach spaces (say, uniformly convex and uniformly smooth spaces)? It is especially interesting whether or not Theorems 2 and 5 hold for an arbitrary reflexive space. For non-reflexive spaces, they do not hold in view of the example in Remark 3.

3) Is it possible to estimate the rate of convergence of greedy algorithms under the hypotheses of Theorems 1, 3 and 4 for some classes of sets $M$ and initial elements $x$? For classical greedy algorithms, there are many estimates of this kind [1]. For example, if $M=\Lambda(D)$, where $D$ is a complete dictionary in a Hilbert space, then for each initial element $x$ which is a finite linear combination of elements of $D$, the norms $\|x_n\|$ of the residuals in the pure greedy algorithm (1) are bounded above by $O(n^{-\gamma})$ with $\gamma=0.182\dots$ (DeVore, Temlyakov, Konyagin and Silnichenko; see [1], Ch. 2). It seems quite reasonable to compare the greedy approximations $\| x_n\|$ in the algorithms of approximation by elements of $M$ considered above with the corresponding $n$-term approximations

$$\begin{equation*} \sigma_n(x)=\rho(x,\underbrace{M+\dots+M}_{n}). \end{equation*}$$

Clearly, $\|x_n\|\ge \sigma_n(x)$, as for the classical approximation with respect to a dictionary.

The author is deeply grateful to V. N. Temlyakov for valuable comments and advice.