Efficient locality-sensitive hashing over high-dimensional streaming data

Abstract

Approximate nearest neighbor (ANN) search in high-dimensional spaces is fundamental in many applications. Locality-sensitive hashing (LSH) is a well-known methodology to solve the ANN problem. Existing LSH-based ANN solutions typically employ a large number of individual indexes optimized for searching efficiency. Updating such indexes might be impractical when processing high-dimensional streaming data. In this paper, we present a novel disk-based LSH index that offers efficient support for both searches and updates. The contributions of our work are threefold. First, we use the write-friendly LSM-trees to store the LSH projections to facilitate efficient updates. Second, we develop a novel estimation scheme to estimate the number of required LSH functions, with which the disk storage and access costs are effectively reduced. Third, we exploit both the collision number and the projection distance to improve the efficiency of candidate selection, improving the search performance with theoretical guarantees on the result quality. Experiments on four real-world datasets show that our proposal outperforms the state-of-the-art schemes.

Appendix 1: Sketch of Proof of Lemma 1

In this section, we provide a brief Proof of Lemma  1 in Sect. 4.3. Recall that, for arbitrary $\varvec{o}_1,\varvec{o}_2,\varvec{q}\in \mathbb {R}^d$, Lemma  1 asserts the following:

$$\begin{aligned} \Pr \left[ \mathbf{E }_1(s_1,R)\right]&>\Pr \left[ \mathbf{E }_1(s_2,R)\right] , \end{aligned}$$

(4)

$$\begin{aligned} \Pr \left[ \mathbf{E }_2(s_1,R)|\mathbf{E }_1(s_1,R)\right]&>\Pr \left[ \mathbf{E }_2(s_2,R)|\mathbf{E }_1(s_2,R)\right] , \end{aligned}$$

(5)

$$\begin{aligned} \Pr \left[ \mathbf{E }_0(s_1,R)\right]&> \Pr \left[ \mathbf{E }_0(s_2,R)\right] , \end{aligned}$$

(6)

where $s_1={\text{ dist }}(\varvec{o}_1,\varvec{q})$, $s_2={\text{ dist }}(\varvec{o}_2,\varvec{q})$, and $s_1 < s_2$.

To prove Lemma  1, we need to consider the following functions:

$$\begin{aligned} F(k,m,x)&\overset{{\tiny {\text{ def }}}}=\sum _{i=0}^k\left( {\begin{array}{c}m\\ i\end{array}}\right) x^i(1-x)^{m-i},\\ g_i(n,m,x)&\overset{{\tiny {\text{ def }}}}=\frac{\left( {\begin{array}{c}m\\ i\end{array}}\right) x^i(1-x)^{m-i}}{1-F(n-1,m,x)},\\ G(n,n^\prime ,m,x)&\overset{{\tiny {\text{ def }}}}=\sum _{i=n}^{n^\prime }g_i(n,m,x)\\&=\frac{F(n^\prime ,m,x)-F(n-1,m,x)}{1-F(n-1,m,x)}. \end{aligned}$$

The following several technical lemmata are useful.

Lemma 3

F(k, m, x) is monotonically decreasing with respect to x if $0\le x\le 1$ and $k<m$.

Lemma 4

$G(n,n^\prime ,m,x)$ is monotonically decreasing with respect to x if $0\le x\le 1$ and $n\le n^\prime <m$.

Lemmata  3 and  4 state some monotonicity properties of the binomial distribution, of which the proofs are pure mathematical and are essentially unsophisticated. Therefore, we omit their proofs to avoid further distraction from our main focus.

We are now ready to present the proof of Lemma  1.

Proof

(Sketch of Proof of Lemma  1) Recall that we have established

$$\begin{aligned} \Pr \left[ \mathbf{E }_1(s,R)\right]&=\sum \limits _{i=L}^{|\mathcal {H}|} \left( {\begin{array}{c}|\mathcal {H}|\\ i\end{array}}\right) p(s,R)^i(1-p(s,R))^{|\mathcal {H}|-i},\\&=1-F\left( L-1,|\mathcal {H}|,p(s,R)\right) , \end{aligned}$$

where $p(s,R)=2\varPhi \left( \frac{wR}{2s}\right) -1$ (i.e., Eq.  3). Since p(s, R) is monotonically decreasing with respect to s, it is clear that, according to Lemma  3, $\Pr [\mathbf{E }_1(s,R)]$ is monotonically decreasing with respect to s, which proves Inequality  4.

To prove Inequality  5, note that

$$\begin{aligned} \Pr&\left[ \mathbf{E }_2(s,R)|\mathbf{E }_1(s,R)\right] \\&=\sum _{i=L}^m \frac{\Pr [{\text{ Col }}(\varvec{o},R)=i]}{\Pr \left[ \mathbf{E }_1(s,R)\right] }\Pr \left[ \mathbf{E }_2(s,R)|{\text{ Col }}(\varvec{o},R)=i\right] \\&=\sum _{i=L}^m g_i(L,|\mathcal {H}|,p(s,R))\Pr \left[ \mathbf{E }_2(s,R)|{\text{ Col }}(\varvec{o},R)=i\right] . \end{aligned}$$

Consider $g_i(L,|\mathcal {H}|,p(s,R))$ as a distribution over the number of collisions $i=L,L+1,\ldots ,m$. The monotonicity of

$$\begin{aligned} G(L,L',|\mathcal {H}|,p(s,R))=\sum _{i=L}^{L^\prime }g_i(L,|\mathcal {H}|,p(s,R)), \end{aligned}$$

as stated in Lemma  4, implies that $g_i(L,|\mathcal {H}|,p(s,R))$ is more skewed toward small i’s as s increases. In addition, it can be proved that the probability $\Pr \left[ \mathbf{E }_2(s,R)|{\text{ Col }}(\varvec{o},R)=i\right]$, viewed as a function of i and s, is monotonically decreasing with respect to s and monotonically increasing with respect to i. Combining the above results, we obtain

$$\begin{aligned} \Pr&\left[ \mathbf{E }_2(s_1,R)|\mathbf{E }_1(s_1,R)\right] \\&=\sum _{i=L}^{m}g_i(L,|\mathcal {H}|,p(s_1,R))\Pr \left[ \mathbf{E }_2(s_1,R)|{\text{ Col }}(\varvec{o},R)=i\right] \\&>\sum _{i=L}^{m}g_i(L,|\mathcal {H}|,p(s_2,R))\Pr \left[ \mathbf{E }_2(s_1,R)|{\text{ Col }}(\varvec{o},R)=i\right] \\&>\sum _{i=L}^{m}g_i(L,|\mathcal {H}|,p(s_2,R))\Pr \left[ \mathbf{E }_2(s_2,R)|{\text{ Col }}(\varvec{o},R)=i\right] \\&=\Pr \left[ \mathbf{E }_2(s_2,R)|\mathbf{E }_1(s_2,R)\right] . \end{aligned}$$

Finally, Inequality  6 is a direct corollary of Inequalities  4 and  5 since $\mathbf{E }_0(s,R)=\mathbf{E }_1(s,R)\cap \mathbf{E }_2(s,R)$ by definition. $\square$