Q-Learning and SARSA are two well-known reinforcement learning (RL) algorithms that have shown promising results in several application domains. However, their approach to build solutions is quite different. For example, SARSA tends to be more conservative than Q-Learning while exploring the solution space. Motivated by such differences, in this paper, we conducted an evaluation of both algorithms in the context of online workflow autoscaling in pay-per-use Clouds, where the goal is to learn optimal virtual machine scaling policies to optimize metrics such as execution time and monetary costs. To do so, we based our experiments on a state-of-the-art scaling strategy with encouraging results in learning optimal scaling policies for reducing execution time and monetary cost. We conducted experiments on simulated environments with four widespread benchmark workflows and two types of virtual machines. Results show that SARSA outperforms Q-Learning in almost all cases. For two workflows SARSA obtains significant gains of up to 40.8% in the first 100 and 300 episodes respectively and losses less than 6% in all episodes observed. In one workflow SARSA achieves significant gains up to 13.9% and no significant losses were observed. There was only one workflow with no significant gains and one significant loss (16.2%) in 1 of 50 observations. In summary, we found multiple stages where SARSA achieves significant and remarkable gains, and the rest of the time both algorithms had a similar performance. In general terms, we can observe that SARSA performs better for learning scaling policies in the Cloud considering workflow applications commonly used by the community to benchmark Cloud workflow resource allocation techniques. These represent interesting results to further drive the design and selection of RL-based autoscaling strategies to schedule workflow executions in the Cloud.

中文翻译：

---

用于科学工作流程的基于强化学习的在线云自动缩放：Q-Learning 和 SARSA 的评估

Q-Learning 和 SARSA 是两种著名的强化学习 (RL) 算法，它们在多个应用领域中都显示出了有希望的结果。然而，他们构建解决方案的方法却截然不同。例如，SARSA 在探索解决方案空间时往往比 Q-Learning 更加保守。受这些差异的启发，在本文中，我们在按使用付费云中的在线工作流自动扩展的背景下对这两种算法进行了评估，其目标是学习最佳虚拟机扩展策略以优化执行时间等指标。货币成本。为此，我们的实验基于最先进的扩展策略，在学习最佳扩展策略以减少执行时间和金钱成本方面取得了令人鼓舞的结果。我们在具有四种广泛使用的基准工作流程和两种类型的虚拟机的模拟环境中进行了实验。结果表明，SARSA 在几乎所有情况下都优于 Q-Learning。对于两个工作流程，SARSA 在前 100 个和 300 个事件中分别获得了高达 40.8% 的显着增益，并且在观察到的所有事件中损失小于 6%。在一个工作流程中，SARSA 实现了高达 13.9% 的显着增益，并且没有观察到显着损失。在 50 个观察结果中，只有 1 个工作流程没有显着收益，而有 1 个显着损失 (16.2%)。总之，我们发现 SARSA 取得了显着且显着的收益的多个阶段，并且其余时间两种算法都具有相似的性能。一般来说，我们可以观察到，考虑到社区常用的工作流应用程序来对云工作流资源分配技术进行基准测试，SARSA 在学习云中的扩展策略方面表现更好。这些代表了有趣的结果，可以进一步推动基于强化学习的自动扩展策略的设计和选择，以在云中安排工作流程执行。