Skyline queries over incomplete data streams

Abstract

Nowadays, efficient and effective processing over massive stream data has attracted much attention from the database community, which are useful in many real applications such as sensor data monitoring, network intrusion detection, and so on. In practice, due to the malfunction of sensing devices or imperfect data collection techniques, real-world stream data may often contain missing or incomplete data attributes. In this paper, we will formalize and tackle a novel and important problem, named skyline query over incomplete data stream (Sky-iDS), which retrieves skyline objects (in the presence of missing attributes) with high confidences from incomplete data stream. In order to tackle the Sky-iDS problem, we will design efficient approaches to impute missing attributes of objects from incomplete data stream via differential dependency (DD) rules. We will propose effective pruning strategies to reduce the search space of the Sky-iDS problem, devise cost-model-based index structures to facilitate the data imputation and skyline computation at the same time, and integrate our proposed techniques into an efficient Sky-iDS query answering algorithm. Extensive experiments have been conducted to confirm the efficiency and effectiveness of our Sky-iDS processing approach over both real and synthetic data sets.

Appendices

Appendix

Proofs of Lemmas for pruning strategies

1.1 Proof of Lemma 1

Proof

As shown in Fig. 3a, since $o'^p.min$ is the minimum corner of the imputed object $o'^p$, it holds that imputed samples of $o'^p$ is dominating $o'^p.min$, that is, $o'^p \preccurlyeq o'^p.min$. Similarly, we also have $o_i^p.max \preccurlyeq o_i^p$. Due to lemma assumption that $o'^p.min$$\prec o_i^p.max$, by dominance transition, we can derive $o'^p \preccurlyeq o'^p.min \prec o_i^p.max \preccurlyeq o_i^p$. Thus, we have $Pr\{o'^p \prec o_i^p\} = 1$ (or $Pr\{o'^p \prec o_{il}\} = 1$ for any instance $o_{il} \in o_i^p$). According to Eq. (4), it holds that $P_{Sky\text {-}iDS}(o_i^p) = 0$. Moreover, since $o'.exp \ge o_i.exp$ holds (i.e., object $o'$ expires after $o_i^p$ from lemma assumption), it indicates that $o_i^p$ can never be the skyline due to the existence of object $o'^p$. Hence, object $o_i^p\in iDS$ can be safely pruned, which completes the proof. $\square$

1.2 Proof of Lemma 2

Proof

From Eq. (4), we can derive a probability upper bound as follows.

$$\begin{aligned} P_{Sky\text {-}iDS}(o_i^p)\le & {} \sum _{\forall o_{il} \in o_i^p} o_{il}.p \cdot (1- Pr\{o'^p \prec o_{il}\}) \nonumber \\= & {} 1- \sum _{\forall o_{il} \in o_i^p} o_{il}.p \cdot Pr\{o'^p \prec o_{il}\}. \end{aligned}$$

(6)

Since $o_i^p.max \preccurlyeq o_{il}$ ($o_{il}\in o_i^p$) and $Pr\{o'^p \prec o_i^p.max\} \ge 1-\alpha$ hold, we have $Pr\{o'^p \prec o_{il}\} \ge Pr\{o'^p \prec o_i^p.max\} \ge 1-\alpha$. By substituting this probability into Eq. (6), we can obtain: $P_{Sky\text {-}iDS}(o_i^p) \le 1- \sum _{\forall o_{il} \in o_i^p} o_{il}.p \cdot (1-\alpha ) = \alpha$. Moreover, since $o'.exp \ge o_i.exp$ holds, $o_i^p$ always has the skyline probability less than $\alpha$ during its lifetime, due to the existence of object $o'$. Thus, object $o_i$ can be safely pruned. $\square$

1.3 Proof of Lemma 3

Proof

Similar to the proof of Lemma 2, since $o'^p \preccurlyeq o'^p.min$ and $Pr\{o'^p.min \prec o_i^p\} \ge 1-\alpha$ hold, we have $Pr\{o'^p \prec o_i^p\} \ge Pr\{o'^p.min \prec o_i^p\} \ge 1-\alpha$. By substituting this probability into Eq. (6), we can obtain: $P_{Sky\text {-}iDS}(o_i^p) \le 1- Pr\{o'^p \prec o_i^p\} = \alpha$. Thus, since object $o_i$ expires before object $o'$ (i.e., $o'.exp \ge o_i.exp$), object $o_i$ always has the skyline probability lower than $\alpha$ during its lifetime. Hence, object $o_i$ can be safely pruned. $\square$

Proofs of properties for skyline tree ST

1.1 Proof of Property 1 of ST

Proof

We can prove this property by showing that no such an imputed object $o_i^p$ exists, where $o_i^p$ is a valid object not within skyline tree ST but is actually a skyline or may become a skyline later.

First, assume that the object $o_i^p$ is a current skyline. According to Definition 6, we can obtain $P_{Sky\text {-}iDS}(o_i^p)>\alpha$. By substituting this probability into Eq. (6), we have $\sum _{\forall o_{il} \in o_i^p} o_{il}.p \cdot Pr\{n^p \prec o_{il}\} < 1-\alpha$, that is, $Pr\{n^p \prec o_i^p\}<1-\alpha$. Thus, no object $t^p$ in ST dominates $o_i^p$ with probability not smaller than $(1-\alpha )$, and then object $o_i^p$ should be on the first layer of ST.

Second, assume that the object $o_i^p$ is dominated by some objects $n^p \in ST$, and may become the skyline after these objects $n^p$ expire (i.e., $n^p.exp < o_i^p.exp$). In this case, object $n^p$ should be the child of one of these objects $n^p$, since $Pr\{n^p \prec o_i^p\} \ge 1-\alpha$ and $n^p.exp < o_i^p.exp$. Therefore, the ST index contains all the objects $o_i^p \in pDS$ that have the chance to be skylines before they expire. $\square$

1.2 Proof of Property 2 of ST

Proof

Given an imputed object $o_i^p \in ST$, if it is not on the first layer of ST, $o_i^p$ will be dominated by its non-empty parent node (object) $n^p \in ST$ with probability $Pr\{n^p \prec o_i^p\} \ge 1-\alpha$. By substituting this probability into Eq. (6), we can obtain $P_{Sky\text {-}iDS}(o_i^p)\le 1-\sum _{\forall o_{il} \in o_i^p} o_{il}.p \cdot Pr\{n^p \prec o_{il}\}=1-Pr\{n^p \prec o_{il}\} \le \alpha$, that is, $P_{Sky\text {-}iDS}(o_i^p)\le \alpha$, which violates the Sky-iDS definition in Definition 6. Hence, object $o_i^p$ cannot become a skyline before its parent node expires from stream iDS. $\square$

1.3 Proof of Property 3 of ST

Proof

According to Property 2, we can get objects $n^p$ not on the first layer all have the skyline probabilities not bigger than $\alpha$ ($P_{Sky\text {-}iDS}(n^p) \le \alpha$). So, current skyline objects must be all on the first layer of ST, in other words, the set of objects on the first layer of ST is a superset of Sky-iDS answers. $\square$