There are more than 190 configuration parameters affecting the performance of MapReduce jobs on Hadoop. It is time-consuming and tedious for general users who have no deep knowledge of Hadoop configuring to tune the parameters of a MapReduce job for optimal performance. Therefore, a self-tuning system to improve MapReduce performance in an automated and efficient manner in a complicated Hadoop environment is needed. This article explores multiple tuning methods to improve tuning efficiency for MapReduce performance on Hadoop. The proposed Catla system employs succinct templates and proper schemes of MapReduce algorithms, which can be incorporated in facilitating the tuning and optimization of MapReduce performance. A comprehensive evaluation of the Catla system, with the support of multiple tuning approaches, is discussed in this article. Direct search-based and derivative-free optimization-based tuning techniques for improved efficiency and usability are evaluated using a series of tuning experiments. The experimental results reveal that our work can identify optimal Hadoop parameters for deployed MapReduce jobs in a noninvasive, flexible, automated, and comprehensive manner.

中文翻译：

---

在 Hadoop 上使用多种调优方法进行非侵入式 MapReduce 性能调优

Hadoop 上影响 MapReduce 作业性能的配置参数超过 190 个。对于没有深入了解 Hadoop 配置的普通用户来说，调整 MapReduce 作业的参数以获得最佳性能既费时又乏味。因此，需要一个自调优系统，以在复杂的 Hadoop 环境中以自动化和高效的方式提高 MapReduce 的性能。本文探讨了多种调优方法，以提高 Hadoop 上 MapReduce 性能的调优效率。所提出的 Catla 系统采用简洁的模板和 MapReduce 算法的适当方案，可以将其纳入促进 MapReduce 性能的调整和优化。本文讨论了在多种调整方法的支持下对 Catla 系统的综合评估。使用一系列调整实验评估基于直接搜索和无导数优化的调整技术，以提高效率和可用性。实验结果表明，我们的工作可以以非侵入性、灵活、自动化和全面的方式为已部署的 MapReduce 作业确定最佳 Hadoop 参数。