Default priors for the intercept parameter in logistic regressions

Abstract

In logistic regression, separation occurs when a linear combination of predictors perfectly discriminates the binary outcome. Because finite-valued maximum likelihood parameter estimates do not exist under separation, Bayesian regressions with informative shrinkage of the regression coefficients offer a suitable alternative. Classical studies of separation imply that efficiency in estimating regression coefficients may also depend upon the choice of intercept prior, yet relatively little focus has been given on whether and how to shrink the intercept parameter.Alternative prior distributions for the intercept are proposed that downweight implausibly extreme regions of the parameter space, rendering regression estimates that are less sensitive to separation. Through simulation and the analysis of exemplar datasets, differences across priors stratified by established statistics measuring the degree of separation are quantified. Relative to diffuse priors, these proposed priors generally yield more efficient estimation of the regression coefficients themselves when the data are nearly separated. They are equally efficient in non-separated datasets, making them suitable for default use. Modest differences were observed with respect to out-of-sample discrimination.These numerical studies also highlight the interplay between priors for the intercept and the regression coefficients:findings are more sensitive to the choice of intercept prior when using a weakly informative prior on the regression coefficients than an informative shrinkage prior.

Introduction

A default prior in principle falls between nearly flat/improper priors, e.g. Jeffrey’s prior (Jeffreys, 1946), and informative shrinkage/variable selection priors, e.g. the Bayesian Lasso (Park and Casella, 2008). Such a prior mildly shrinks toward some null value non- or weakly identified parameters and leaves alone those well supported by the likelihood (Gelman et al., 2008, Greenland and Mansournia, 2015, Rainey, 2016). However, it does not borrow strength via shared hyperpriors as do informative shrinkage priors.

Default priors have been developed for binary data regression models, e.g. logistic regression, because of the possibility of so-called ‘separation’, or the existence of a linear combination of predictors that can perfectly discriminate the outcomes in the data (Albert and Anderson, 1984, Santner and Duffy, 1986). Separation can be ‘complete’ or ‘quasi-complete’, with both leading to non-finite maximum likelihood estimates (MLEs, Albert and Anderson, 1984, Santner and Duffy, 1986). A truly large association between a predictor and the outcome can cause separation, illustrating that this phenomenon is not always undesirable. Other causes include sparsity, high correlation between the predictors, or the inclusion of many binary predictors (Heinze and Schemper, 2002, Greenland and Mansournia, 2015). Regardless of cause, the lack of identification may warrant mild regularization via priors. For the intercept parameter, a very diffuse or improper prior is the usual choice; in this paper, we contend that its unique role and meaning warrant a more informative choice.

A number of authors have proposed default prior specifications for regression coefficients (Clogg et al., 1991, Bedrick et al., 1996, Zorn, 2005, Gelman et al., 2008, Hanson et al., 2014, Greenland and Mansournia, 2015). The scale-family of $g$-priors, or reference informative priors, is one early example of a default prior for regression coefficients (Zellner, 1983). The degree of shrinkage – and therefore the extent to which that family of priors may be viewed as ‘default’ – depends on the choice of scale parameter $g$, which is shared by all regression coefficients. If fixed at some diffuse or prespecified value, this would satisfy our definition of a default prior (Hanson et al., 2014). If instead $g$ is adaptively tuned via a hyperprior, as in Marin and Robert (2007), this would not, as information is shared across parameters.

Christmann and Rousseeuw (2001) propose quantitative measures of separation: $n_{\text{comp}}\ge 0$ is the size of the smallest subset of observations that, if removed from the data, would completely separate the complementary subset. Separation is equivalent to $n_{\text{comp}}=0$; the resulting lack of numerical convergence allows for relatively easy detection. In contrast, near-separation, i.e. a small but positive $n_{\text{comp}}$, yields finite MLEs but manifests symptoms of separation including instability and efficiency loss. For this reason, mild regularization from default priors can be just as useful when there are a few dozen predictors as when there are hundreds or more, particularly when the number of observations is of a similar order, i.e. $p\approx n$.

The point of departure in this work is a focus on the intercept parameter. In contrast to regression coefficients, there is generally not an intuitive null direction toward which shrinkage should be, and borrowing strength for the intercept is not possible (e.g. Section 3.4, Hastie et al., 2009). The usual recommendation is that a prior should be flat or effectively so (Greenland and Mansournia, 2015, Zorn, 2005, Gelman et al., 2008). We demonstrate that straightforward efficiency gains are possible by assuming that exceptionally large values of the intercept are either implausible or unverifiable in the data. Thus, down-weighting these regions frequently improves efficiency.

This paper makes several contributions. First, we use complete separation to establish a rationale that mild shrinkage of the intercept can improve estimation of the regression coefficients. Following Ghosh et al. (2017), we consider a stronger type of separation that we call ‘pivotal separation’. Second, we propose to adapt the exponential-power scale-family of distributions (Box, 1953, West, 1987, Box and Tiao, 1992) for default use as a prior on the intercept in binary data regression models and develop an algorithm to determine a suitable scale for this prior. Finally, our work highlights the correspondence between choice of prior on the intercept and that of the regression coefficients.

In Section 2, we motivate the interplay between intercept and regression coefficients under the premise of separation. Sections 3 Possible choices of default prior, 4 Prior on describe our choices of the different class priors for the intercept and the regression coefficients, respectively. Findings from a comprehensive simulation study are documented in Section 5, which provides a comparative appraisal of the Bayesian estimators under varied scenarios including sparsity and $p\approx n$. The demonstration of our methodology on ten datasets in Section 6 illustrates the heterogeneity in the degree of separation in real data and, more importantly, highlights the stabilizing properties of our proposed priors in estimation of the regression coefficients. Section 7 interprets our results and discusses some counterarguments against using priors on the intercept parameter.