Data analytics has become an integral part of large-scale scientific computing. Among various data analytics frameworks, MapReduce has gained the most traction. Although some efforts have been made to enable efficient MapReduce for supercomputing systems, they are often limited to fairly homogeneous workloads where equal partitioning of input data across tasks results in essentially equal output or temporary data generated on each task. For workloads that are more skewed, however, current implementations can result in imbalance in memory usage and, consequently, can cause a slowdown in execution time and a loss in data scalability. To tackle this problem, we enhance a previously published memory-conscious MapReduce over MPI framework called Mimir. Our enhancements to Mimir include combiner and dynamic repartition optimizations to minimize and balance memory usage and to achieve close to optimal balance of the memory usage across processes and to reduce the execution time by up to 12 times. Experimental results show that Mimir can scale to at least 3072 processes on the Tianhe-2 supercomputer on skewed datasets.

中文翻译：

---

用于超级计算系统的基于 MPI 的内存高效且抗偏斜的 MapReduce


数据分析已成为大规模科学计算不可或缺的一部分。在各种数据分析框架中，MapReduce 最受欢迎。尽管已经做出了一些努力来为超级计算系统实现高效的 MapReduce，但它们通常仅限于相当同质的工作负载，其中跨任务的输入数据的相等分区会导致每个任务生成的输出或临时数据基本相等。然而，对于更加倾斜的工作负载，当前的实现可能会导致内存使用不平衡，从而导致执行时间减慢和数据可扩展性损失。为了解决这个问题，我们在 MPI 框架上增强了之前发布的内存感知 MapReduce，称为 Mimir。我们对 Mimir 的增强包括组合器和动态重新分区优化，以最大限度地减少和平衡内存使用，实现跨进程的内存使用接近最佳平衡，并将执行时间减少多达 12 倍。实验结果表明，Mimir 在天河二号超级计算机上在倾斜数据集上可以扩展到至少 3072 个进程。