Abstract

It is not unusual for a data analyst to encounter datasets distributed across several computers. This can happen for reasons such as privacy concerns, efficiency of likelihood evaluations, or just the sheer size of the whole dataset. This presents new challenges to statisticians as even computing simple summary statistics such as the median becomes computationally challenging. Furthermore, if other advanced statistical methods are desired, then novel computational strategies are needed. In this article, we propose a new approach for distributed analysis of massive data that is suitable for generalized fiducial inference and is based on a careful implementation of a “divide-and-conquer” strategy combined with importance sampling. The proposed approach requires only small amount of communication between nodes, and is shown to be asymptotically equivalent to using the whole dataset. Unlike most existing methods, the proposed approach produces uncertainty measures (such as confidence intervals) in addition to point estimates for parameters of interest. The proposed approach is also applied to the analysis of a large set of solar images. Supplementary materials for this article are available online.

摘要

数据分析师遇到分布在多台计算机上的数据集并不罕见。发生这种情况的原因可能包括隐私问题、可能性评估的效率或整个数据集的庞大规模。这给统计学家带来了新的挑战，因为即使计算简单的汇总统计数据（例如中位数）在计算上也变得具有挑战性。此外，如果需要其他高级统计方法，则需要新颖的计算策略。在本文中，我们提出了一种适用于广义基准推理的海量数据分布式分析新方法，该方法基于结合重要性采样的“分而治之”策略的仔细实施。所提出的方法只需要节点之间的少量通信，并且显示为渐近等效于使用整个数据集。与大多数现有方法不同，所提出的方法除了对感兴趣的参数进行点估计外，还会产生不确定性度量（例如置信区间）。所提出的方法也适用于分析大量太阳图像。本文的补充材料可在线获取。