Manifold Approximation by Moving Least-Squares Projection (MMLS)

Abstract

In order to avoid the curse of dimensionality, frequently encountered in big data analysis, there has been vast development in the field of linear and nonlinear dimension reduction techniques in recent years. These techniques (sometimes referred to as manifold learning) assume that the scattered input data is lying on a lower-dimensional manifold; thus the high dimensionality problem can be overcome by learning the lower dimensionality behavior. However, in real-life applications, data is often very noisy. In this work, we propose a method to approximate $\mathcal {M}$ a d-dimensional $C^{m+1}$ smooth submanifold of $\mathbb {R}^n$ ($d \ll n$) based upon noisy scattered data points (i.e., a data cloud). We assume that the data points are located “near” the lower-dimensional manifold and suggest a nonlinear moving least-squares projection on an approximating d-dimensional manifold. Under some mild assumptions, the resulting approximant is shown to be infinitely smooth and of high approximation order (i.e., $\mathcal {O}(h^{m+1})$, where h is the fill distance and m is the degree of the local polynomial approximation). The method presented here assumes no analytic knowledge of the approximated manifold and the approximation algorithm is linear in the large dimension n. Furthermore, the approximating manifold can serve as a framework to perform operations directly on the high-dimensional data in a computationally efficient manner. This way, the preparatory step of dimension reduction, which induces distortions to the data, can be avoided altogether.

Appendix A - Geometrically Weighted PCA

We wish to present here in our language the concept of geometrically weighted PCA (presented a bit differently in [16]), as this concept plays an important role in some of the lemmas proved in Sect. 4 and even in the algorithm itself.

Given a set of I vectors $x_1 ,\ldots , x_I$ in $\mathbb {R}^n$, we look for a Rank(d) projection $P \in \mathbb {R}^{n \times n}$ that minimizes

$$\begin{aligned} \sum \limits _{i=1}^I || Px_i - x_i ||_2^2. \end{aligned}$$

If we denote by A the matrix whose i’th column is $x_i$, then this is equivalent to minimizing

$$\begin{aligned} || PA - A ||_F^2 ,\end{aligned}$$

as the best possible Rank(d) approximation to the matrix A is the SVD Rank(d) truncation denoted by $A_d$, we have:

$$\begin{aligned}&PA = P U \Sigma V^T = A_d = U \Sigma _d V^T\\&P = U \Sigma _d V^T V \Sigma ^{-1} U^T\\&P = U \Sigma _d \Sigma ^{-1} U^T\\&P = U I_d U^T\\&P = U_d U_d^T. \end{aligned}$$

And this projection yields:

$$\begin{aligned} Px = U_d U_d^T x = \sum \limits _{i=1}^d \langle x, u_i \rangle \cdot u_i , \end{aligned}$$

(25)

which is the orthogonal projection of x onto $\mathrm{span} \lbrace u_i \rbrace _{i=1}^d$. Here $u_i$ represents the $\hbox {i}{th}$ column of the matrix U.

Remark 5.1

The projection P is identically the projection induced by the PCA algorithm.

1.1 The Weighted Projection

In this case, given a set of n vectors $x_1 ,\ldots , x_I$ in $\mathbb {R}^n$, we look for a Rank(d) projection $P \in \mathbb {R}^{n \times n}$ that minimizes

$$\begin{aligned}&\sum \limits _{i=1}^I || Px_i - x_i ||_2^2 ~ \theta (||x_i - q||_2) = \sum \limits _{i=1}^I || Px_i - x_i ||_2^2 ~ w_i\\&\quad = \sum \limits _{i=1}^I || \sqrt{w_i} Px_i - \sqrt{w_i} x_i ||_2^2\\&\quad = \sum \limits _{i=1}^I || P \sqrt{w_i} x_i - \sqrt{w_i} x_i ||_2^2\\&\quad = \sum \limits _{i=1}^I || P y_i - y_i ||_2^2. \end{aligned}$$

So if we define the matrix $\tilde{A}$ such that the i’th column of $\tilde{A}$ is the vector $y_i = \sqrt{w_i}x_i$, then we get the projection

$$\begin{aligned} P = \tilde{U}_d \tilde{U}_d^T , \end{aligned}$$

(26)

where $\tilde{U}_d$ is the matrix containing the first d principal components of the matrix $\tilde{A}$.