PCA-based drift and shift quantification framework for multidimensional data

Abstract

Concept drift is a serious problem confronting machine learning systems in a dynamic and ever-changing world. In order to manage concept drift it may be useful to first quantify it by measuring the distance between distributions that generate data before and after a drift. There is a paucity of methods to do so in the case of multidimensional numeric data. This paper provides an in-depth analysis of the PCA-based change detection approach, identifies shortcomings of existing methods and shows how this approach can be used to measure a drift, not merely detect it.

Appendices

Appendix

Experiments

Some of the experiments conducted involved generating data that have known theoretical, or “true”, Hellinger distance. We describe here the process used to generate these data. Datasets were generated from the multivariate normal distribution. Generation of random samples was done through an “inverted” PCA approach:

First generate independent univariate normal variable and then use a rotation to introduce dependency between them. HD could be attributed to difference in either mean, variance or correlation. Data were generated for each value of HD between 0 and 1 with step 0.01 and for various sample sizes between 100 and 10,000.

1.1 Difference is due to difference in mean

The samples are drawn from distributions that have the same rotation and equal covariance matrices $(V_1=V_2=V)$, but different means $(M_1\ne M_2)$. To generate distributions that differ in mean while retaining identical covariance, we use the following equality.

$$\begin{aligned} H^2= & {} 1-\frac{\root 4 \of {\det {V_1}}\root 4 \of {\det {V_2}}}{\sqrt{\det {\frac{V_1+V_2}{2}}}}\exp \Bigg [-\frac{1}{8}(\mu _1-\mu _2)'\Big (\frac{V_1+V_2}{2}\Big )^{-1}(\mu _1-\mu _2)\Bigg ]\nonumber \\= & {} 1-\exp \Bigg [-\frac{1}{8}(\mu _1-\mu _2)'V^{-1}(\mu _1-\mu _2)\Bigg ] \end{aligned}$$

(13)

Let $\varDelta =(\mu _1-\mu _2)'V^{-1}(\mu _1-\mu _2)$. If V is diagonal, then $\varDelta =\sum _{i=1}^{n}\frac{\mu ^2_i}{\sigma ^2_i}$. We split into n randomly selected addends that sum to $\varDelta$ and assign them to correspondent PCA components. The procedure to generate samples is described by Algorithm 7

1.2 Difference is due to different variance, with the same mean and rotation(eigenvectors)

We use the following equality to generate distributions that differ in variance without any change in mean or rotation.

$$\begin{aligned} V= & {} P'\varLambda P\nonumber \\ H^2= & {} 1-\frac{\root 4 \of {\det {V_1}}\root 4 \of {\det {V_2}}}{\sqrt{\det {\frac{V_1+V_2}{2}}}}\exp \Bigg [-\frac{1}{8}(\mu _1-\mu _2)'\Big (\frac{V_1+V_2}{2}\Big )^{-1}(\mu _1-\mu _2)\Bigg ]\nonumber \\= & {} 1-\frac{\root 4 \of {\det {V_1}}\root 4 \of {\det {V_2}}}{\sqrt{\det {\frac{V_1+V_2}{2}}}}=1-\frac{\root 4 \of {\det {P'\varLambda _1 P}}\root 4 \of {\det {P'\varLambda _2 P}}}{\sqrt{\det {\frac{P'\varLambda _1 P+P'\varLambda _2 P}{2}}}}=1-\frac{\root 4 \of {\prod (\varLambda _{1i}\varLambda _{2i})}}{\sqrt{\prod (\frac{\varLambda _{1i}+\varLambda _{2i}}{2})}}\nonumber \\ \end{aligned}$$

(14)

If we set

$$\begin{aligned} \varLambda _2=(1+\alpha )*\varLambda _1, \mathrm{then} \quad H^2=1-\frac{(1+\alpha )^{\frac{n}{4}}}{(1+\frac{\alpha }{2})^{\frac{n}{2}}} \end{aligned}$$

Then it follows that

$$\begin{aligned} \alpha = \frac{-b+\sqrt{b^2-4b}}{2}, \mathrm{where} \quad b=(1-H^2)^{\frac{4}{n}} \end{aligned}$$

We then use Algorithm 8 to generate the two samples.

1.3 HD is due to different correlations with the same mean and variance

We use the following equality to generate distributions that differ in correlation matrices without any change in mean or variance. Covariance matrix $V=DRD$, where D is a diagonal matrix of standard deviations and R is a corresponding correlation matrix.

$$\begin{aligned} H^2= & {} 1-\frac{\root 4 \of {\det {V_1}}\root 4 \of {\det {V_2}}}{\sqrt{\det {\frac{V_1+V_2}{2}}}}\exp \Bigg [-\frac{1}{8}(\mu _1-\mu _2)'\Big (\frac{V_1+V_2}{2}\Big )^{-1}(\mu _1-\mu _2)\Bigg ]\\= & {} 1-\frac{\root 4 \of {\det {DR_1D}}\root 4 \of {\det {DR_2D}}}{\sqrt{\det {D\frac{R_1+R_2}{2}D}}}=1-\frac{\root 4 \of {\det {R_1}}\root 4 \of {\det {R_2}}}{\sqrt{\det {\frac{R_1+R_2}{2}}}} \end{aligned}$$

Numerical approximation was used to generate a correlation matrix that yields the desired HD. In the first experiment $R_1$ was set to identity matrix. In the second experiment $R_1$ was set to the matrix where all off-diagonal elements equal − 0.1. Diagonal elements for the second matrix $R_2$ were set to one and off-diagonal to $\alpha$, where $\alpha$ was numerically approximated for each value of HD.