Iterative restricted OK estimator in generalized linear models and the selection of tuning parameters via MSE and genetic algorithm

Abstract

This article introduces an iterative restricted OK estimator in generalized linear models to address the dilemma of multicollinearity by imposing exact linear restrictions on the parameters. It is a versatile estimator, which contains maximum likelihood (ML), restricted ML, Liu, restricted Liu, ridge and restricted ridge estimators in generalized linear models. To figure out the performance of restricted OK estimator over its counterparts, various comparisons are given where the performance evaluation criterion is the scalar mean square error (SMSE). Thus, illustrations and simulation studies for Gamma and Poisson responses are conducted apart from theoretical comparisons to see the performance of the estimators in terms of estimated and predicted MSE. Besides, the optimization techniques are applied to find the values of tuning parameters by minimizing SMSE and by using genetic algorithm.

Appendices

Appendix A

Differentiating $l(\theta ,\beta ,k,d;y,r)$ with respect to $\beta _{j}$ and using the chain rule, we get

$$\begin{aligned} \frac{\partial }{\partial \beta _{j}}l(\theta ,\beta ,k,d;y,r)= & {} \sum \limits _{i=1}^{n}\frac{y_{i}-\mu _{i}}{a(\phi )}\frac{1}{b^{^{\prime \prime }}(\theta _{i})}\frac{1}{g^{^{\prime }}(\mu _{i})}x_{ij}-k\sum \limits _{j=1}^{p}(\beta _{j}-d{\hat{\beta }}_{MLj}) \nonumber \\&+\sum \limits _{i=1}^{t}\lambda _{i}R_{ij}(r_{i}-R_{ij}\beta _{j}) \nonumber \\= & {} \sum \limits _{i=1}^{n}\frac{y_{i}-\mu _{i}}{a(\phi )}\frac{1}{b^{^{\prime \prime }}(\theta _{i})}\frac{g^{^{\prime }}(\mu _{i})}{(g^{^{\prime }}(\mu _{i}))^{2}}x_{ij}-k\sum \limits _{j=1}^{p}(\beta _{j}-d{\hat{\beta }}_{MLj}) \nonumber \\&+\sum \limits _{i=1}^{t}\lambda _{i}R_{ij}(r_{i}-R_{ij}\beta _{j}) \nonumber \\= & {} \sum \limits _{i=1}^{n}\frac{y_{i}-\mu _{i}}{w_{ii}}g^{^{\prime }}(\mu _{i})x_{ij}-k\sum \limits _{j=1}^{p}\beta _{j}+kd\sum \limits _{j=1}^{p}\hat{ \beta }_{MLj} \nonumber \\&+\sum \limits _{i=1}^{t}\lambda _{i}R_{ij}(r_{i}-R_{ij}\beta _{j}) \end{aligned}$$

(11)

where $w_{ii}=var(y_{i})(g^{^{\prime }}(\mu _{i}))^{2}$, since $var(y_{i})=a(\phi )b^{^{\prime \prime }}(\theta _{i})$. In matrix form Eq. (11) can be written as

$$\begin{aligned} S(\beta ,d,k)=X^{T}WD(y-\mu )-k\beta +kd{\hat{\beta }}_{ML}+R^{T}\varLambda (r-R\beta ) \end{aligned}$$

where $W=Diag(w_{ii}^{-1})$ and $D=Diag(g^{^{\prime }}(\mu _{i}))$ and $\varLambda =Diag(\lambda _{1},\cdots ,\lambda _{t})$.

Now taking derivative of Eq. (11) with respect to $\beta _{k}$, we have

$$\begin{aligned} \frac{\partial ^{2}}{\partial \beta _{j}\partial \beta _{k}}l(\theta ,\beta ,k,d;y,r)= & {} \sum \limits _{i=1}^{n}(y_{i}-\mu _{i})\frac{\partial }{\partial \beta _{k}}\frac{1}{w_{ii}}g^{^{\prime }}(\mu _{i})x_{ij}+\sum \limits _{i=1}^{n}\frac{g^{^{\prime }}(\mu _{i})x_{ij}}{w_{ii} }(\frac{-1}{g^{^{\prime }}(\mu _{i})}x_{ik}) \nonumber \\&-k\delta _{jk}-\sum \limits _{i=1}^{t}\lambda _{i}R_{ij}R_{ik} \end{aligned}$$

(12)

where $\delta _{jk}=1$, if $j=k$, and 0 otherwise.

Minus times the expected value of Eq. (12) gives us:

$$\begin{aligned} Q_{jk}(\beta ,d,k)= & {} -E\left[ \frac{\partial ^{2}}{\partial \beta _{j}\partial \beta _{k}}l(\theta ,\beta ,k,d;y,r)\right] \\= & {} \sum \limits _{i=1}^{n}\frac{x_{ij}x_{ik}}{w_{ii}}+k\delta _{jk}+\sum \limits _{i=1}^{t}\lambda _{i}R_{ij}R_{ik}. \end{aligned}$$

In matrix notation, we can write it as

$$\begin{aligned} Q(\beta ,d,k)=(X^{T}WX+kI_{p})+R^{T}\varLambda R\text {.} \end{aligned}$$

By using the method of Fisher’s scoring, we have

$$\begin{aligned} {\hat{\beta }}_{kd-R}^{\left( m+1\right) }={\hat{\beta }}_{kd-R}^{(m)}+\{\left[ Q(\beta ,d,k)\right] ^{-1}S(\beta ,d,k)\}_{\beta ={\hat{\beta }}_{kd-R}^{(m)}} \text {.} \end{aligned}$$

Premultiplying both sides by $Q(\beta ,d,k)$, we get

$$\begin{aligned} \left[ Q(\beta ,d,k)\right] _{\beta ={\hat{\beta }}_{kd-R}^{(m)}}{\hat{\beta }} _{kd-R}^{\left( m+1\right) }=\left[ Q(\beta ,d,k)\right] _{\beta ={\hat{\beta }} _{kd-R}^{(m)}}{\hat{\beta }}_{kd-R}^{(m)}+\left[ S(\beta ,d,k)\right] _{\beta = {\hat{\beta }}_{kd-R}^{(m)}}. \end{aligned}$$

By substituting the values, we have

$$\begin{aligned} \left[ (X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})+R^{T}\varLambda R\right] {\hat{\beta }} _{kd-R}^{\left( m+1\right) }= & {} \{[(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})+R^{T} \varLambda R]{\hat{\beta }}_{kd-R}^{(m)} \\&+X^{T}{\hat{W}}_{kd-R}^{(m)}D(y-{\hat{\mu }}_{kd-R}^{(m)})\\&-k{\hat{\beta }} _{kd-R}^{(m)}+kd{\hat{\beta }}_{ML}+R^{T}\varLambda (r-R{\hat{\beta }}_{kd-R}^{(m)})\} \end{aligned}$$

where m denotes the iteration step and ${\hat{W}}_{kd-R}^{(m)}$ is a weight matrix evaluated at ${\hat{\beta }}_{kd-R}^{(m)}$. Then we obtain

$$\begin{aligned} {\hat{\beta }}_{kd-r}^{\left( m+1\right) }= & {} \left[ (X^{T}{\hat{W}} _{kd-r}^{(m)}X+kI_{p})+R^{T}\varLambda R\right] ^{-1}\{(X^{T}{\hat{W}} _{kd-r}^{(m)}X+kI_{p}){\hat{\beta }}_{kd-r}^{(m)} \nonumber \\&+X^{T}{\hat{W}}_{kd-r}^{(m)}D(y-{\hat{\mu }}_{kd-r}^{(m)})-k{\hat{\beta }} _{kd-r}^{(m)}+kd{\hat{\beta }}_{ML}+R^{T}\varLambda r\}. \end{aligned}$$

(13)

By using inverse formula, we have

$$\begin{aligned}&\left[ (X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})+R^{T}\varLambda R\right] ^{-1}\\&\quad =(X^{T} {\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}-(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1} \\&\qquad \times R^{T}\{\varLambda ^{-1}+R(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T} \}^{-1}R(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}. \end{aligned}$$

Then Eq. (13) be converted to

$$\begin{aligned} {\hat{\beta }}_{kd-R}^{\left( m+1\right) }= & {} (X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}\{X^{T}{\hat{W}}_{kd-R}^{(m)}X{\hat{\beta }} _{kd-R}^{(m)}+X^{T}{\hat{W}}_{kd-R}^{(m)}D(y-{\hat{\mu }}_{kd-R}^{(m)})+kd\hat{ \beta }_{ML}\} \nonumber \\&-(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\{\varLambda ^{-1}+R(X^{T}\hat{W }_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\}^{-1} \nonumber \\&\times R(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}\{X^{T}{\hat{W}} _{kd-R}^{(m)}X{\hat{\beta }}_{kd-R}^{(m)}+X^{T}{\hat{W}}_{kd-R}^{(m)}D(y-{\hat{\mu }} _{kd-R}^{(m)}) \nonumber \\&+kd{\hat{\beta }}_{ML}\}+\{(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}-(X^{T} {\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T} \nonumber \\&\times [\varLambda ^{-1}+R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}]^{-1}R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}\}R^{T}\varLambda r\text {.} \end{aligned}$$

(14)

By considering:

$$\begin{aligned}&\{(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}-(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T} \\&\quad \times [\varLambda ^{-1}+R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}]^{-1}R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}\}R^{T}\varLambda \\&\quad =\{(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}-(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T} \\&\quad \times [\varLambda ^{-1}+R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}]^{-1}R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\}\varLambda \\&\quad =(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\{I_{p}-[\varLambda ^{-1}+R(X^{T} {\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}]^{-1} \\&\quad \times R(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\}\varLambda \\&\quad =(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\{\varLambda ^{-1}+R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\}^{-1} \end{aligned}$$

Eq. (14) becomes

$$\begin{aligned} {\hat{\beta }}_{kd-R}^{\left( m+1\right) }= & {} (X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}\{X^{T}{\hat{W}}_{kd-R}^{(m)}X{\hat{\beta }} _{kd-R}^{(m)}+X^{T}{\hat{W}}_{kd-R}^{(m)}D(y-{\hat{\mu }}_{kd-R}^{(m)})+kd\hat{ \beta }_{ML}\} \\&-(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\{\varLambda ^{-1}+R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\}^{-1} \\&\times R(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}\{X^{T}{\hat{W}}_{kd-R}^{(m)}X {\hat{\beta }}_{kd-R}^{(m)} \\&+X^{T}{\hat{W}}_{kd-R}^{(m)}D(y-{\hat{\mu }}_{kd-R}^{(m)})+kd{\hat{\beta }} _{ML}\}+(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1} \\&\times R^{T}\{\varLambda ^{-1}+R(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T} \}^{-1}r. \end{aligned}$$

On further simplification, we get

$$\begin{aligned} {\hat{\beta }}_{kd-R}^{\left( m+1\right) }= & {} (X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}[X^{T}{\hat{W}}_{kd-R}^{(m)}\{X{\hat{\beta }} _{kd-R}^{(m)}+D(y-{\hat{\mu }}_{kd-R}^{(m)})\}+kd{\hat{\beta }}_{ML}] \\&-(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\{\varLambda ^{-1}+R(X^{T}{\hat{W}} _{kd-R}^{(m)}X+kI_{p})^{-1}R^{T}\}^{-1} \\&\times \{R(X^{T}{\hat{W}}_{kd-R}^{(m)}X+kI_{p})^{-1}[X^{T}{\hat{W}} _{kd-R}^{(m)}\{X{\hat{\beta }}_{kd-R}^{(m)}+D(y-{\hat{\mu }}_{kd-R}^{(m)})\} \\&+kd{\hat{\beta }}_{ML}]-r\}. \end{aligned}$$

Appendix B

Table 6 EMSE and PMSE values of the FOA estimators when (k, d) values are obtained via GA by minimizing MSEP for Poisson response variable

Table 7 EMSE and PMSE values of the iterative estimators when (k, d) values are obtained via GA by minimizing MSEP for Poisson response variable

Table 8 EMSE and PMSE values of the FOA estimators when (k, d) values are obtained via GA by minimizing MAE for Poisson response variable

Table 9 EMSE and PMSE values of the iterative estimators when (k, d) values are obtained via GA by minimizing MAE for Poisson response variable

Table 10 EMSE and PMSE values of the FOA estimators when (k, d) values are obtained by MSE Poisson response variable

Table 11 EMSE and PMSE values of the iterative estimators when (k, d) values are obtained by MSE for Poisson response variable

Table 12 EMSE and PMSE values of the FOA estimators when (k, d) values are obtained via GA by minimizing MSEP for Gamma response variable

Table 13 EMSE and PMSE values of the iterative estimators when (k, d) values are obtained via GA by minimizing MSEP for gamma response variable

Table 14 EMSE and PMSE values of the FOA estimators when (k, d) values are obtained via GA by minimizing MAE for gamma response variable

Table 15 EMSE and PMSE values of the iterative estimators when (k, d) values are obtained via GA by minimizing MAE for gamma response variable

Table 16 EMSE and PMSE values of the FOA estimators when (k, d) values areobtained by MSE gamma response variable

Table 17 EMSE and PMSE values of the iterative estimators when (k, d) values are obtained by MSE for gamma response variable