A von Mises–Fisher mixture model for clustering numerical and categorical variables

Abstract

This work presents a mixture model allowing to cluster variables of different types. All variables being measured on the same n statistical units, we first represent every variable with a unit-norm operator in ${\mathbb {R}}^{n\times n}$ endowed with an appropriate inner product. We propose a von Mises–Fisher mixture model on the unit-sphere containing these operators. The parameters of the mixture model are estimated with an EM algorithm, combined with a K-means procedure to obtain a good starting point. The method is tested on simulated data and eventually applied to wine data.

Appendix: proofs

a) $\Phi ^2(X,Y)=\textit{tr}(\Pi _X \Pi _Y)$.

Proof

Let X and Y be two categorical variables with q and r levels respectively, and let $\mathbf{X }$ and $\mathbf{Y }$ denote their respective matrices of q (resp. r) uncentred indicator variables, and X and Y their respective matrices of $q-1$ (resp. $r-1$) centred indicator variables. We have:

$$\begin{aligned} \langle X\rangle =\langle \mathbf{X } \rangle \cap \langle \mathbb {1} \rangle ^{\perp } ; \langle Y \rangle =\langle \mathbf{Y } \rangle \cap \langle \mathbb {1} \rangle ^{\perp } \end{aligned}$$

And so:

$$\begin{aligned} \Pi _X=\Pi _{\mathbf{X }}-\Pi _{\mathbb {1}}; \Pi _Y=\Pi _{\mathbf{Y }}-\Pi _{\mathbb {1}} \end{aligned}$$

(5)

Besides, it can easily be shown that:

$$\begin{aligned} \textit{tr}(\Pi _{\mathbf{X }} \Pi _{\mathbf{Y }})=\textit{tr}\left( \textit{MM}'\right) , \end{aligned}$$

(6)

where

$$\begin{aligned} M=(\mathbf{X }'W \mathbf{X })^{-1/2} \mathbf{X }'W \mathbf{Y} (\mathbf{Y }'W\mathbf{Y })^{-1/2} \end{aligned}$$

We have:

$$\begin{aligned} M=\left( m_{\textit{ij}}\right) _{i,j} \text { with } m_{\textit{ij}}=\frac{\pi _{\textit{ij}}}{\sqrt{\pi _{\textit{i.}}\pi _{.j}}} \end{aligned}$$

(7)

From (5), (6) and (7), we get:

$$\begin{aligned} \textit{tr}(\Pi _X \Pi _Y)=\displaystyle {\sum _{\begin{array}{l} i=1,\ldots ,q \\ j=1,\ldots ,r \end{array}}} \frac{(\pi _{\textit{ij}}-\pi _{\textit{i.}}\pi _{.j})^2}{\pi _{\textit{i.}}\pi _{.i}}=\Phi ^2(X,Y) \end{aligned}$$

b) $\forall x: \text {arg }\underset{y\in S}{\text {min }}\Vert x-y\Vert ^2=\frac{x}{\Vert x\Vert }$, where S is the unit sphere. $\square$

Proof

Let $\forall y,y^0=\frac{y}{\Vert y\Vert }$. Then

$$\begin{aligned} \forall x,\Vert x-y^0\Vert ^2=\Vert x\Vert ^2+\Vert y^0\Vert ^2-2\langle x|y^0\rangle \end{aligned}$$

So:

$$\begin{aligned} \text {arg }\underset{y\in S}{\text {min}}\Vert x-y\Vert ^2= & {} \text {arg }\underset{y\in S}{\text {max}}\langle x|y\rangle \\= & {} \text {arg }\underset{y\in S}{\text {max}}(\Vert x\Vert ^2\cos ^2(x,y)) \\= & {} \text {arg }\underset{y\in S}{\text {min}}(\Vert x\Vert ^2\sin ^2(x,y)) \\= & {} {\hat{y}}^0, \forall {\hat{y}}=\text {arg } \underset{y}{\text {min}} \big (\Vert x\Vert ^2\sin ^2(x,y)\big ) \end{aligned}$$

Taking for instance ${\hat{y}}=x$ gives the result.

c) Rank-r average of normed projectors.

Let $\Pi _U$ be the projector on a space spanned by H W-orthonormal vectors $u_1 , \ldots , u_r \in {\mathbb {R}}^n$ . Let U denote the matrix $[u_1 , \ldots , u_r]$ . The rank-r average of a set of p normed projectors ${\tilde{O}}_j$ is defined as the normed projector $\tilde{{\bar{O}}}^r = \frac{\Pi _U}{\sqrt{r}}$ which verifies :

$$\begin{aligned} {\tilde{\bar{O}}}^r = \mathop {\hbox {arg min}}\limits _U \sum _{j} \llbracket \tilde{O}_U - {\tilde{O}}_j\rrbracket ^2. \end{aligned}$$

Now, $\forall j, {\tilde{O}}_j = X_j M_j X_j'W$ , where:

• $X_j$ is the variable associated with the projector and coded as mentioned in Sect. 2.1, and

• $M_j = \frac{(X_j'WX_j)^{-1}}{\sqrt{\dim (X_j)}}$

We denote $X = [X_1 , \ldots , X_p]$ and $M = diag(M_j ; j=1,\ldots ,p)$ (block-diagonal matrix with blocks $M_j$).

Since $\forall j, \llbracket \tilde{O}_U - {\tilde{O}}_j \rrbracket ^2 = 2(1 - [\tilde{O}_U | {\tilde{O}}_j ] )$, we have:

$$\begin{aligned} \tilde{{\bar{O}}}^r= & {} \mathop {\hbox {arg max}}\limits _U \sum _{j} [\tilde{O}_U | {\tilde{O}}_j ] \\= & {} \mathop {\hbox {arg max}}\limits _U \sum _{j} [\Pi _U | {\tilde{O}}_j ] \\= & {} \mathop {\hbox {arg max}}\limits _U [\Pi _U | \sum _{j} {\tilde{O}}_j ] \\= & {} \mathop {\hbox {arg max}}\limits _U tr(UU'W \sum _{j} {\tilde{O}}_j ) \\= & {} \mathop {\hbox {arg max}}\limits _U tr(U'W \sum _{j} {\tilde{O}}_j U) \\= & {} \mathop {\hbox {arg max}}\limits _U ( \sum _{h=1,\ldots ,r} u_h'W \sum _{j} X_j M_j X_j'W u_h )\\= & {} \mathop {\hbox {arg max}}\limits _U ( \sum _{h=1,\ldots ,r} u_h'W XMX'W u_h )\\ \end{aligned}$$

Vectors $u_1 , \ldots , u_r$ being W-orthonormal, this maximization program is exactly that of the (dual) PCA of array X with metric M and weights W. The solution vectors $u_1 , \ldots , u_r$ are hence the r first PCs of (X, M, W), and $\tilde{{\bar{O}}}^H$ is the normed projector on the space they span. $\square$