Nonparametric estimation for big-but-biased data

Abstract

Nonparametric estimation for a large-sized sample subject to sampling bias is studied in this paper. The general parameter considered is the mean of a transformation of the random variable of interest. When ignoring the biasing weight function, a small-sized simple random sample of the real population is assumed to be additionally observed. A new nonparametric estimator that incorporates kernel density estimation is proposed. Asymptotic properties for this estimator are obtained under suitable limit conditions on the small and the large sample sizes and standard and non-standard asymptotic conditions on the two bandwidths. Explicit formulas are shown for the particular case of mean estimation. Simulation results show that the new mean estimator outperforms two classical ones for suitable choices of the two smoothing parameters involved. The influence of two smoothing parameters on the performance of the final estimator is also studied, exhibiting a striking limit behavior of their optimal values. The new method is applied to a real data set from the Telco Company Vodafone ES, where a bootstrap algorithm is used to select the smoothing parameter.

Sketch of the proof

Only the sketches of the proof of Theorems 1 and 2 are included here.

1.1 Sketch of the proof of Theorem 1

Let us first state an auxiliary lemma, whose proof can be found in the supplementary materials.

Lemma 2

The difference ${\hat{\mu }}_v - \mu _v$ can be expressed as follows:

$$\begin{aligned} {\hat{\mu }}_v - \mu _v={\widehat{A}} +{\widehat{A}}\left( 1-{\widehat{B}}\right) +\dfrac{{\widehat{A}}\left( 1-{\widehat{B}}\right) ^2}{{\widehat{B}}}\simeq {\widehat{A}}, \end{aligned}$$

(16)

where

$$\begin{aligned} {\widehat{A}}=\dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{{\hat{f}}_h(Y_i)}{{\hat{g}}_b(Y_i)}(v(Y_i)-\mu _v)} \end{aligned}$$

(17)

and

$$\begin{aligned} {\widehat{B}}=\dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{{\hat{f}}_h(Y_i)}{{\hat{g}}_b(Y_i)}}. \end{aligned}$$

(18)

The term in (17) can be splitted in different terms, ${\widehat{A}} ={\widehat{A}}_1 +{\widehat{A}}_2 - {\widehat{A}}_3 - {\widehat{A}}_4 + {\widehat{A}}_5,$ where

$$\begin{aligned} {\widehat{A}}_1:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{f(Y_i)}{g(Y_i)}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_2:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{{\hat{f}}_h(Y_i)-f(Y_i)}{g(Y_i)}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_3:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{f(Y_i) ( {\hat{g}}_b(Y_i)-g(Y_i) )}{g(Y_i)^2}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_4:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{({\hat{f}}_h(Y_i)-f(Y_i)) ( {\hat{g}}_b(Y_i)-g(Y_i) )}{g(Y_i)^2}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_5:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{{\hat{f}}_h(Y_i)}{{\hat{g}}_b(Y_i)}\left( \dfrac{{\hat{g}}_b(Y_i)-g(Y_i)}{g(Y_i)}\right) ^2(v(Y_i)-\mu _v)}. \end{aligned}$$

Since the terms ${\widehat{A}}_4$ and ${\widehat{A}}_5$ have some factors of quadratic nature inside the sum (i.e., $({\hat{f}}_h(Y_i)-f(Y_i)) ( {\hat{g}}_b(Y_i)-g(Y_i) )$ and $( {\hat{g}}_b(Y_i)-g(Y_i) )^2$) they are negligible with respect to other terms. Consequently, ${\widehat{A}} \simeq {\widehat{A}}_1 +{\widehat{A}}_2 - {\widehat{A}}_3.$

Lemma 3

The expectation and variance of ${\widehat{A}}$ can be approximated by

$$\begin{aligned} E\left( {\widehat{A}}\right)\simeq & {} E\left( {\widehat{A}}_1\right) +E\left( {\widehat{A}}_2\right) - E\left( {\widehat{A}}_3\right) , \end{aligned}$$

(19)

$$\begin{aligned} Var\left( {\widehat{A}}\right)\simeq & {} Var\left( {\widehat{A}}_1\right) +Var\left( {\widehat{A}}_2\right) + Var\left( {\widehat{A}}_3\right) \nonumber \\+ & {} 2 Cov\left( {\widehat{A}}_1, {\widehat{A}}_2\right) -2Cov\left( {\widehat{A}}_1, {\widehat{A}}_3\right) - 2Cov\left( {\widehat{A}}_2, {\widehat{A}}_3\right) . \end{aligned}$$

(20)

The proof of Theorem 1 proceeds by analyzing the expectations and variances involved. It is available in the supplementary materials.

1.2 Sketch of the proof of Theorem 2

Using Lemma 2, in this case ${\widehat{A}}$ can be expressed as: ${\widehat{A}} ={\widehat{A}}_1^* +{\widehat{A}}_2^* - {\widehat{A}}_3^* - {\widehat{A}}_4^* + {\widehat{A}}_5^*,$ where

$$\begin{aligned} {\widehat{A}}_1^*:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{(K_h*f)(Y_i)}{(K_b*g)(Y_i)}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_2^*:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{{\hat{f}}_h(Y_i)-(K_h*f)(Y_i)}{(K_b*g)(Y_i)}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_3^*:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{(K_h*f)(Y_i) ( {\hat{g}}_b(Y_i)-(K_b*g)(Y_i) )}{(K_b*g)(Y_i)^2}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_4^*:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{({\hat{f}}_h(Y_i)-(K_h*f)(Y_i)) ( {\hat{g}}_b(Y_i)-(K_b*g)(Y_i) )}{(K_b*g)(Y_i)^2}(v(Y_i)-\mu _v)}, \\ {\widehat{A}}_5^*:= & {} \dfrac{1}{N}\displaystyle \sum _{i=1}^{N}{\dfrac{{\hat{f}}_h(Y_i)}{{\hat{g}}_b(Y_i)}\left( \dfrac{{\hat{g}}_b(Y_i)-(K_b*g)(Y_i)}{(K_b*g)(Y_i)}\right) ^2(v(Y_i)-\mu _v)}, \end{aligned}$$

being ${\widehat{A}}_4^*$ and ${\widehat{A}}_5^*$ negligible terms. Thus we will consider ${\widehat{A}}^* :={\widehat{A}}_1^* +{\widehat{A}}_2^* - {\widehat{A}}_3^*.$

The proof of Theorem 2 follows parallel lines to that of Theorem 1. It is available in the supplementary materials.