Choosing the right configuration for Spark deployed in the public cloud to ensure the efficient running of periodic jobs is hard, because there can be a huge configuration space to explore which is composed of numerous performance-related parameters in different dimensions (e.g., application-level and cloud-level). Choosing poorly will not only significantly degrade performance but may also lead to greater overhead. However, automatically searching for the optimal configuration of various applications to trade-off performance and cost is challenging. To address this issue, we propose a new optimal configuration search algorithm named AB-MOEA/D by combining multi-objective optimization algorithm and performance prediction model. AB-MOEA/D uses a decomposition-based multi-objective optimization algorithm to find the configuration with the objective of minimizing the execution time and cost, where the performance model constructed on the Adaboost algorithm is used to evaluate the fitness of each candidate configuration. Besides, we also present the configuration automatic tuning system with AB-MOEA/D as the optimization engine. The experimental results on six benchmarks with five data sets show that AB-MOEA/D significantly outperforms the previous work in terms of execution time and cost, with average improvements of approximately 35 and 40 percent.

为部署在公有云中的 Spark 选择合适的配置以确保周期性作业的高效运行是很困难的，因为可能有巨大的配置空间可供探索，这些空间由不同维度（例如应用程序级）的众多性能相关参数组成。和云级）。选择不当不仅会显着降低性能，还可能导致更大的开销。然而，自动搜索各种应用程序的最佳配置以权衡性能和成本是具有挑战性的。为了解决这个问题，我们结合多目标优化算法和性能预测模型，提出了一种新的最优配置搜索算法AB-MOEA/D。AB-MOEA/D 使用基于分解的多目标优化算法寻找配置，以最小化执行时间和成本为目标，其中基于 Adaboost 算法构建的性能模型用于评估每个候选配置的适应度。此外，我们还提出了以AB-MOEA/D为优化引擎的配置自动调优系统。在五个数据集的六个基准测试中的实验结果表明，AB-MOEA/D 在执行时间和成本方面明显优于之前的工作，平均提高了大约 35% 和 40%。我们还展示了以 AB-MOEA/D 作为优化引擎的配置自动调整系统。在五个数据集的六个基准测试中的实验结果表明，AB-MOEA/D 在执行时间和成本方面明显优于之前的工作，平均提高了大约 35% 和 40%。我们还展示了以 AB-MOEA/D 作为优化引擎的配置自动调整系统。在五个数据集的六个基准测试中的实验结果表明，AB-MOEA/D 在执行时间和成本方面明显优于之前的工作，平均提高了大约 35% 和 40%。