Abstract

Monitoring large-scale datastreams with limited resources has become increasingly important for real-time detection of abnormal activities in many applications. Despite the availability of large datasets, the challenges associated with designing an efficient change-detection when clustering or spatial pattern exists are not yet well addressed. In this article, a design-adaptive testing procedure is developed when only a limited number of streaming observations can be accessed at each time. We derive an optimal sampling strategy, the pattern-oriented-sampling, with which the proposed test possesses asymptotically and locally best power under alternatives. Then, a sequential change-detection procedure is proposed by integrating this test with generalized likelihood ratio approach. Benefiting from dynamically estimating the optimal sampling design, the proposed procedure is able to improve the sensitivity in detecting clustered changes compared with existing procedures. Its advantages are demonstrated in numerical simulations and a real data example. Ignoring the neighboring information of spatially structured data will tend to diminish the detection effectiveness of traditional detection procedures. Supplementary materials for this article are available online.

摘要

监控资源有限的大规模数据流对于实时检测许多应用程序中的异常活动变得越来越重要。尽管存在大型数据集，但在存在聚类或空间模式时与设计有效的变化检测相关的挑战尚未得到很好的解决。在本文中，当每次只能访问有限数量的流观测时，开发了一种设计自适应测试程序。我们推导出了一种最优采样策略，即面向模式的采样，所提出的测试在替代方案下具有渐近和局部的最佳功效。然后，通过将该测试与广义似然比方法相结合，提出了一种顺序变化检测程序。受益于动态估计最优抽样设计，与现有程序相比，所提出的程序能够提高检测集群变化的灵敏度。它的优势在数值模拟和真实数据示例中得到了证明。忽略空间结构化数据的相邻信息往往会降低传统检测程序的检测有效性。本文的补充材料可在线获取。