当前位置: X-MOL 学术Int. J. Parallel. Program › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
CSMqGraph: Coarse-Grained and Multi-external-storage Multi-queue I/O Management for Graph Computing
International Journal of Parallel Programming ( IF 1.5 ) Pub Date : 2019-11-15 , DOI: 10.1007/s10766-019-00651-0
Shuo Chen , Zhan Shi , Dan Feng , Shang Liu , Fang Wang , Lei Yang , Ruili Yu

As graphs continue growing, external storage graph processing systems serve as a promising alternative to distributed in-memory solutions for low cost and high scalability. To obtain high I/O throughput, these systems usually use multiple external storage devices. They adopt the operating system I/O management method based on striped volume, resulting in unsatisfactory performance, such as low sequential bandwidth utilization of each external storage device, limited I/O parallelism and expensive management overhead. In this paper, we analyzed the problems of the operating system I/O management method based on striped volume. Then we designed CSMqGraph, a graph processing system adopts coarse-grained striping method matching sequential large I/O to fully utilize the maximum sequential bandwidth of each external storage device and an I/O management strategy based on multi-external-storage multi-queue making I/O threads dedicated to each external storage device to further improve I/O throughput and fully exploit the parallelism of multiple external storage devices. For different graph algorithms and datasets, our evaluation shows that CSMqGraph consistently outperforms state-of-the-art engines GridGraph by up to 40%, and has better I/O scalability.

中文翻译:

CSMqGraph:用于图形计算的粗粒度和多外部存储多队列 I/O 管理

随着图的不断增长,外部存储图处理系统作为分布式内存解决方案的一种有前途的替代方案,具有低成本和高可扩展性。为了获得高 I/O 吞吐量,这些系统通常使用多个外部存储设备。它们采用基于条带卷的操作系统I/O管理方法,导致性能不理想,例如每个外部存储设备的顺序带宽利用率低、I/O并行度有限以及管理开销昂贵。本文分析了基于条带卷的操作系统I/O管理方法存在的问题。然后我们设计了 CSMqGraph,图处理系统采用粗粒度分条方式匹配顺序大I/O,充分利用每个外部存储设备的最大顺序带宽和基于多外存多队列制作I/O线程的I/O管理策略专用于每个外部存储设备,以进一步提高 I/O 吞吐量并充分利用多个外部存储设备的并行性。对于不同的图算法和数据集,我们的评估表明,CSMqGraph 始终比最先进的引擎 GridGraph 高出 40%,并且具有更好的 I/O 可扩展性。
更新日期:2019-11-15
down
wechat
bug