In-memory data processing frameworks (e.g., Spark) make big data analysis greatly simpler and efficient. However, stragglers that take much longer to finish than other tasks significantly degrade performance. There exist multiple factors that cause stragglers, either from the hardware resource layer or application layer, e.g. hardware heterogeneity, interference, data locality and data skew. While state-of-the-art straggler mitigation techniques have presented partial solutions on data skew and data locality, we experimentally demonstrate that the other factors can also result in serious problems.

We present Clio, a cross-layer interference-aware optimization system that can effectively mitigate stragglers for data processing frameworks. Clio supports the scheduling of both map and reduce tasks. It heuristically dispatches intermediate data in proportion to the actual computing ability of each worker node, which is estimated considering various straggler factors, to balance the completion times of tasks in a much finer way. We implement Clio in Apache Spark, and evaluate its performance using both synthetic and real datasets. Experiment results show that, Clio can speed up the execution of applications by up to 67%, compared with the existing algorithms.

内存中数据处理框架（例如Spark）使大数据分析变得更加简单和高效。然而，散兵游勇这需要更长的时间才能完成比其他任务显著降低性能。存在导致硬件资源层或应用程序层散乱的多种因素，例如，硬件异质性，干扰，数据局部性和数据偏斜。虽然最先进的散乱缓解技术已经提出了关于数据偏斜和数据局部性的部分解决方案，但我们通过实验证明了其他因素也可能导致严重的问题。

我们介绍了Clio，这是一个跨层的，可感知干扰的优化系统，可以有效地缓解数据处理框架的混乱情况。Clio支持映射和归约任务的调度。它根据每个工作节点的实际计算能力按比例启发式分派中间数据，并考虑各种散乱因素来估计中间数据，从而以更好的方式平衡任务的完成时间。我们在Apache Spark中实现Clio，并使用合成和真实数据集评估其性能。实验结果表明，与现有算法相比，Clio可以将应用程序的执行速度提高67％。