Split sample empirical likelihood

Abstract

Empirical likelihood offers a nonparametric approach to estimation and inference, which replaces the probability density-based likelihood function with a function defined by estimating equations. While this eliminates the need for a parametric specification, the restriction of numerical optimization greatly decreases the applicability of empirical likelihood for large data problems. A solution to this problem is the split sample empirical likelihood; this variant utilizes a divide and conquer approach, allowing for parallel computation of the empirical likelihood function. The results show the asymptotic distribution of the estimators and test statistics derived from the split sample empirical likelihood are the same seen in standard empirical likelihood yet have significantly decreased computational times.

Introduction

Empirical likelihood (Owen, 1988, Owen, 1990) is a data driven likelihood that does not require specification of the data generating mechanism (specifically the probability function). Under mild regularity conditions empirical likelihoods inherit the asymptotic properties of the Fisher likelihood (Wilks, 1938, Qin and Lawless, 1994), and with very few exceptions (for instance Lazar and Mykland, 1999) permit a Bartlett correction (DiCiccio et al., 1991). Furthermore, empirical likelihood can be extended to quantiles, Huber’s location M estimate (Huber, 1964), and any functional that has a Fréchet derivative. The general form and first order asymptotics of the empirical likelihood are explored by Qin and Lawless (1994).

Empirical likelihood methods have been extended to many problems such as bivariate means (Owen, 1990); constrained empirical likelihood which includes the creation of a conditional empirical likelihood, Euclidean likelihood which allows the confidence region to extend beyond the convex hull and triangular array empirical likelihood which relaxes the assumption of the data being identically distributed (Owen, 1991); regression models, correlation models, ANOVA and variance modeling (Owen, 1991, Owen, 2001); the entire class of projection pursuit models (Owen, 1992, Kolaczyk, 1994); time series modeling (Owen, 2001, Kitamura, 1997); incorporating information from multiple moments for a parameter, two sample problems with a common mean, probability measure, incomplete information problems (Qin and Lawless, 1994); ratios of parameters and logistic regression (Qin and Lawless, 1995); partially linear models (Shi and Lau, 2000); missing response problems (Qin and Zhang, 2007); finite population inference (Chen and Qin, 1993); and Bayesian settings (Grendar and Judge, 2010, Lazar, 2003).

Many modern applications involve questions of interest that lead to complex models and to atypical probabilistic distributions. In this context, the ability to relax parametric assumptions makes empirical likelihood a very promising tool. However modern applications can involve a very large number of observations and variables, leading to serious computational obstacles for empirical likelihood. The empirical likelihood cannot typically be written in a closed form; as a consequence the solution requires numerical optimization, resulting in non-trivial and time consuming computations. Even if the time consideration is ignored, there is an additional limitation: as sample sizes (and number of parameters) increase the optimization routines can quickly exceed system memory, making the parameter estimates (let alone any inferential bounds) impossible to compute.

If the computational and memory limitations of empirical likelihood could be solved, this would reintroduce a very flexible and robust method as a potential solution to many modern statistical and scientific questions of interest. One approach to large sample problems is the notion of divide and conquer, which has been used, for instance, in regression (Chen and Xie, 2014, Song and Liang, 2015) and Monte Carlo simulations (Lindsten et al., 2017). Following this idea, we introduce a variation of empirical likelihood which maintains standard properties and can use parallel computation to dramatically reduce computation time. We call this construct split sample empirical likelihood (SSEL).

The construction of the empirical likelihood (which we will refer to as the full-sample empirical likelihood), like all likelihoods, starts with a sample from some distribution. Specifically let $x_1,\dots ,x_n$ be $d$-variate independent identically distributed observations from some cumulative distribution $F$. The empirical likelihood function is

$$L\left(F\right)=\prod _{i=1}^{n}dF\left(x_i\right)=\prod _{i=1}^{n}Pr(X=x_i)=\prod _{i=1}^n{u}_i.$$

The empirical likelihood function is maximized by the empirical distribution function

$$L\left(F_n\right)=\prod _{i=1}^n{n}^{-1},$$

and the empirical likelihood ratio function $R\left(F\right)=L\left(F\right)∕L\left(F_n\right)$ can be written as

$$R\left(F\right)=\prod _{i=1}^{n}n{u}_i.$$

Suppose now we are interested in the estimation of a $p\times 1$ parameter $\theta$. We add additional constraints in the form of $r\ge p$ unbiased estimating equations $g(x,\theta )$, i.e. 

$$Eg(X,{\theta }_0)=0,$$

which along with constraints on $u_i$ give the profile empirical likelihood ratio function

$$R_E\left(\theta \right)=\underset{u}{sup}\left(\prod _{i=1}^{n}n{u}_i\mid u_i\ge 0,\sum _{i=1}^n{u}_i=1,\sum _{i=1}^n{u}_{i}g(x_i,\theta )=0\right).$$

Provided that $\theta$ is inside the convex hull of the set $g(x_1,\theta ),\dots ,g(x_n,\theta )$ a unique value of Eq. (1) exists (Owen, 1988). $R_E\left(\theta \right)$ is undefined for all $\theta$ not inside the convex hull. The convex hull of the point cloud $S$ is the set

$$\text{Conv}\left(S\right)=\left\{\sum _{i=1}^{\left|S\right|}{\alpha }_i{x}_i\left|(\forall i:\ge 0)\wedge \sum _{i=1}^{\left|S\right|}{\alpha }_i=1\right.\right\}.$$

The implications of solutions only existing inside the convex hull will be further explored in Section 2.

The rest of the article is as follows. We formally define split sample empirical likelihood in Section 2, along with necessary conditions and notation. Section 3 includes all relevant lemmas, theorems and corollaries, which show that both the estimators and test statistics have the same asymptotic distributions seen in full-sample empirical likelihood. Section 4 contains a simulation study to demonstrate how to properly construct a split sample empirical likelihood, along with results indicating that SSEL has a significantly decreased computation time. Section 5 discusses some practical issues when using SSEL.