Minimizers of Sparsity Regularized Huber Loss Function

Abstract

We investigate the structure of the local and global minimizers of the Huber loss function regularized with a sparsity inducing L0 norm term. We characterize local minimizers and establish conditions that are necessary and sufficient for a local minimizer to be strict. A necessary condition is established for global minimizers, as well as non-emptiness of the set of global minimizers. The sparsity of minimizers is also studied by giving bounds on a regularization parameter controlling sparsity. Results are illustrated in numerical examples.

Appendix

Definition 8.1

Let $\Psi :{{\mathbb {R}}}^N\rightarrow {{\mathbb {R}}}$ be a differentiable function over ${{\mathbb {R}}}^N$ and its gradient has a Lipschitz constant $L_\Psi >0$:

$$\begin{aligned} \left\Vert \triangledown \Psi (x)-\triangledown \Psi (y)\right\Vert _2\le L_{\Psi }\left\Vert x-y\right\Vert _2 \text { for all } x,y\in {{\mathbb {R}}}^N. \end{aligned}$$

Proposition 8.1

$\Psi$ has a Lipschitz constant $\dfrac{\left\Vert A\right\Vert ^2_2}{\gamma }$, where $\left\Vert A\right\Vert _2=\sup _{\left\Vert x\right\Vert _2=1}\dfrac{\left\Vert Ax\right\Vert _2}{\left\Vert x\right\Vert _2}.$

Proof

Let $x,y\in {{\mathbb {R}}}^N$,

$$\begin{aligned} \left\Vert \triangledown \Psi (x)-\triangledown \Psi (y)\right\Vert _2&=\left\Vert A^{ T}[\hbox {clip}(Ax-d)-\hbox {clip}(Ay-d)]\right\Vert _2\\&\le \left\Vert A\right\Vert _2\left\Vert \hbox {clip}(Ax-d)-\hbox {clip}(Ay-d)\right\Vert _2\\&=\left\Vert A\right\Vert _2\dfrac{\left\Vert \hbox {clip}(Ax-d)-\hbox {clip}(Ay-d)\right\Vert _2}{\left\Vert A(x-y)\right\Vert _2}\left\Vert A(x-y)\right\Vert _2\\&\le \left\Vert A\right\Vert _2^2\dfrac{\left\Vert \hbox {clip}(Ax-d)-\hbox {clip}(Ay-d)\right\Vert _2}{\left\Vert (Ax-d)-(Ay-d)\right\Vert _2}\left\Vert x-y\right\Vert _2\\&\le \dfrac{\left\Vert A\right\Vert _2^2}{\gamma }\left\Vert x-y\right\Vert _2. \end{aligned}$$

Last inequality comes from continuity of the gradient. $\square$

Remark 8.1

One should use the Frobenius norm for easy computation, which causes the Lipschitz constant to be $\dfrac{\left\Vert A\right\Vert ^2_F}{\gamma }$. This choice is safe since $\left\Vert A\right\Vert _2\le \left\Vert A\right\Vert _F$.

Proposition 8.2

${\mathbb {B}}_\gamma$ defined in Theorem 5.2 is a dense subset of ${{\mathbb {R}}}^N$.

Proof

Let $c:{{\mathbb {R}}}^N\rightarrow {{\mathbb {R}}}^M$ be a linear continuous operator defined as $c(x)=Ax-d$. Since A is full rank with $M<N$, c is a surjection. Let $T=c({\mathbb {B}}_\gamma )$ where $c({\mathbb {B}}_\gamma )$ is the image of set ${\mathbb {B}}_\gamma$. Hence, $T=\{v\in {{\mathbb {R}}}^M:\forall i\in {{\mathbb {I}}}_M, \left|v[i]\right|\ne \gamma \}$. Let ${{\mathcal {O}}}$ be an arbitrary non-empty open set in ${{\mathbb {R}}}^M$. Let ${\bar{v}}\in {{\mathcal {O}}}$ and define an index set as follows: ${\bar{v}}_\gamma =\{i\in {{\mathbb {I}}}_M:\left|{\bar{v}}[i]\right|=\gamma \}.$ Now, there is some positive radius $r_{{\bar{v}}}$ such that $B_\infty ({\bar{v}},r_{{\bar{v}}})$ stays in ${{\mathcal {O}}}$. If we define

$$\begin{aligned} r^*=\min \Bigg \{\min _{i\notin {\bar{v}}_\gamma }\{\left|{\bar{v}}[i]-\gamma \right|\},\min _{i\notin {\bar{v}}_\gamma }\{\left|{\bar{v}}[i]+\gamma \right|\},r_{{\bar{v}}},\gamma \Bigg \}, \end{aligned}$$

we have $r^*>0$ and $B_\infty ({\bar{v}},r_{{\bar{v}}})$ stays in ${{\mathcal {O}}}$. Then, for any $0<\delta <r^*$, $v^*={\bar{v}}+\delta \mathbb {1}_M$ belongs to ${{\mathcal {O}}}$ and T at the same time. Hence, T is a dense set. T is the image of a continuous surjection; therefore, ${\mathbb {B}}_\gamma$ is a dense set too. $\square$

Remark 8.2

Previous result shows that one can construct a sequence of vectors from ${\mathbb {B}}_\gamma$ converging to any other vector in ${{\mathbb {R}}}^N$. This may be useful for deriving algorithms using second-order methods since the second derivative exists only for vectors in ${\mathbb {B}}_\gamma$.

For all numerical experiments reported in this paper, we use the following equivalent formulation for (HR)

Proposition 8.3

(Equivalent Characterization for (HR), [35]) Any optimal solution to the quadratic program (QHR2) is a minimizer of $\Psi$, and conversely.

This alternative eliminates the need to work with piecewise functions and provides an easier computation tool. We used the following algorithm for the numerical examples reported in the paper: