当前位置: X-MOL 学术IEEE Signal Proc. Mag. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Straggler-Resistant Distributed Matrix Computation via Coding Theory: Removing a Bottleneck in Large-Scale Data Processing
IEEE Signal Processing Magazine ( IF 14.9 ) Pub Date : 2020-05-01 , DOI: 10.1109/msp.2020.2974149
Aditya Ramamoorthy , Anindya Bijoy Das , Li Tang

The current big data era routinely requires the processing of large-scale data on massive distributed computing clusters. In these applications, data sets are often so large that they cannot be housed in the memory and/or the disk of any one computer. Thus, the data and the processing are typically distributed across multiple nodes. Distributed computation is thus a necessity rather than a luxury. The widespread use of such clusters presents several opportunities and advantages over traditional computing paradigms. However, it also presents newer challenges where coding-theoretic ideas have recently had a significant impact. Large-scale clusters (which can be heterogeneous in nature) suffer from the problem of stragglers, which are slow or failed worker nodes in the system. Thus, the overall speed of a computation is typically dominated by the slowest node in the absence of a sophisticated assignment of tasks to the worker nodes.

中文翻译:

基于编码理论的分布式矩阵计算:消除大规模数据处理中的瓶颈

当前的大数据时代通常需要在海量分布式计算集群上处理海量数据。在这些应用程序中,数据集通常非常大,以至于它们无法容纳在任何一台计算机的内存和/或磁盘中。因此,数据和处理通常分布在多个节点上。因此,分布式计算是必需品而不是奢侈品。与传统计算范式相比,此类集群的广泛使用提供了多种机会和优势。然而,它也带来了新的挑战,其中编码理论思想最近产生了重大影响。大规模集群(本质上可能是异构的)存在落后者的问题,即系统中的工作节点缓慢或出现故障。因此,
更新日期:2020-05-01
down
wechat
bug