Asymptotic linear expansion of regularized M-estimators

Abstract

Parametric high-dimensional regression requires regularization terms to get interpretable models. The respective estimators correspond to regularized M-functionals which are naturally highly nonlinear. Their Gâteaux derivative, i.e., their influence curve linearizes the asymptotic bias of the estimator, but only up to a remainder term which is not guaranteed to tend (sufficiently fast) to zero uniformly on suitable tangent sets without profound arguments. We fill this gap by studying, in a unified framework, under which conditions the M-functionals corresponding to convex penalties as regularization are compactly differentiable, so that the estimators admit an asymptotically linear expansion. This key ingredient allows influence curves to reasonably enter model diagnosis and enable a fast, valid update formula, just requiring an evaluation of the corresponding influence curve at new data points. Moreover, this paves the way for optimally-robust estimators, bounding the influence curves in a suitable way.

Miscellaneous

The $L_2-$differentiability originally comes from LeCam (1970).

Definition 5

Let $\mathcal {P}:=\{P_{\theta } \ | \ \theta \in \varTheta \}$ be a family of probability measures on some measurable space $(\varOmega , \mathcal {A})$ and let $\varTheta$ be a subset of $\mathbb {R}^p$. Then, $\mathcal {P}$ is $L_2-$differentiable at $\theta _0$ if there exists $\varLambda _{\theta _0} \in L_2^p(P_{\theta _0})$ such that

$$\begin{aligned} \displaystyle \left| \left| \sqrt{dP_{\theta _0+h}}-\sqrt{dP_{\theta _0}}\left( 1+ \frac{1}{2} h^T \varLambda _{\theta _0} \right) \right| \right| _{L_2}=o(||h||) \end{aligned}$$

for $||h|| \rightarrow 0$. In this case, the function $\varLambda _{\theta _0}$ is the $L_2-$derivative and $I_{\theta _0}:=\mathbb {E}_{\theta _0}[\varLambda _{\theta _0}\varLambda _{\theta _0}^T]$ is the Fisher information of $\mathcal {P}$ at $\theta _0$.

Note that the $L_2-$differentiability is a special case of the wider concept of $L_r-$differentiability (cf. Rieder and Ruckdeschel 2001). The $L_2-$differentiability holds for many distribution families, including normal location and scale families, Poisson families, gamma families, and even for ARMA, ARCH and GPD families (Rieder et al. 2008, Pupashenko et al. 2015). A standard example of a distribution family that is not $L_2-$differentiable is the model $\mathcal {P}:=\{U([0,\theta ]) \ | \ \theta \in \varTheta \}$.

The following definition of partial influence curves and the corresponding asymptotically linear expansion in terms of such partial influence curves is borrowed from (Rieder 1994, Def. 4.2.10) and Rieder et al. (2008).

Definition 6

Let $(\varOmega ^n, \mathcal {A}^n)$ be a measurable space and let $S_n: (\varOmega ^n, \mathcal {A}^n) \rightarrow (\mathbb {R}^q, \mathbb {B}^q)$ be an estimator for the transformed quantity of interest $\tau (\theta )$. Assume that $\tau : \varTheta \rightarrow \mathbb {R}^q$ is differentiable at $\theta _0 \in \varTheta$ where $\varTheta \subset \mathbb {R}^p$ and $q \le p$. Denote the Jacobian by $\partial _{\theta _0} \tau =:D_{\theta _0} \in \mathbb {R}^{q \times p}$. Then, the set of partial influence curves is defined by

$$\begin{aligned} \displaystyle \varPsi _2^D(\theta _0):=\{\eta _{\theta _0} \in L_2^q(P_{\theta _0}) \ | \ \mathbb {E}_{\theta _0}[\eta _{\theta _0}]=0, \ \mathbb {E}_{\theta _0}[\eta _{\theta _0} \varLambda _{\theta _0}^T]=D_{\theta _0}\} . \end{aligned}$$

The sequence $(S_n)_n$ is asymptotically linear at $P_{\theta _0}$ if there exists a partial influence curve $\eta _{\theta _0} \in \varPsi _2^D(\theta _0)$ such that the expansion

$$\begin{aligned} \displaystyle S_n=\tau (\theta _0)+\frac{1}{n} \sum _{i=1}^n \eta _{\theta _0}(x_i)+o_{P_{\theta _0}^n}(n^{-1/2}) \end{aligned}$$

is valid.

For the following lemma, we refer to Evgrafov and Patriksson (2003) and Levitin and Tichatschke (1998).

Lemma 4

Let $f: \mathcal {X} \times \mathcal {Y} \times \varTheta \rightarrow \mathbb {R}$ be continuous, where $\mathcal {X} \subset \mathbb {R}^n$, $\mathcal {Y} \subset \mathbb {R}^m$, $\varTheta \subset \mathbb {R}^k$. Define $\varXi (x,y):=\mathrm{argmin}_{\theta }(f(x,y,\theta ))$. If f is coercive w.r.t. $\theta$, i.e., the sets $\{\theta \in \varTheta \ | \ f(x,y,\theta ) \le c\}$ are bounded for all $c \in \mathbb {R}$ for every $x \in \mathcal {X}$, $y \in \mathcal {Y}$, then $\min _{\theta }(f(x,y,\theta ))>-\infty$ and $\varXi (x,y)$ is nonempty and compact for any x, y.