In many data-intensive applications, workflow is often used as an important model for organizing data processing tasks and resource provisioning is an important and challenging problem for improving the performance of workflows. Recently, system variations in the cloud and large-scale clusters, such as those in I/O and network performances and failure events, have been observed to greatly affect the performance of workflows. Traditional resource provisioning methods, which overlook these variations, can lead to suboptimal resource provisioning results. In this article, we provide a general solution for workflow performance optimizations considering system variations. Specifically, we model system dynamics as time-dependent random variables and take their probability distributions as optimization input. Despite its effectiveness, this solution involves heavy computation overhead. Thus, we propose three pruning techniques to simplify workflow structure and reduce the probability evaluation overhead. We implement our techniques in a runtime library, which allows users to incorporate efficient probabilistic optimization into existing resource provisioning methods. Experiments show that probabilistic solutions can improve the performance by up to 65 percent compared to state-of-the-art static solutions, and our pruning techniques can greatly reduce the overhead of our probabilistic approach.

中文翻译：

---

驯服数据处理工作流资源优化的系统动力学：概率方法


在许多数据密集型应用中，工作流经常被用作组织数据处理任务的重要模型，而资源配置是提高工作流性能的一个重要且具有挑战性的问题。最近，人们发现云和大规模集群中的系统变化（例如 I/O 和网络性能以及故障事件）极大地影响了工作流的性能。传统的资源供应方法忽视了这些变化，可能导致资源供应结果不理想。在本文中，我们提供了考虑系统变化的工作流性能优化的通用解决方案。具体来说，我们将系统动力学建模为与时间相关的随机变量，并将其概率分布作为优化输入。尽管该解决方案很有效，但它涉及大量的计算开销。因此，我们提出了三种修剪技术来简化工作流程结构并减少概率评估开销。我们在运行时库中实现我们的技术，该库允许用户将有效的概率优化合并到现有的资源配置方法中。实验表明，与最先进的静态解决方案相比，概率解决方案可以将性能提高高达 65%，并且我们的剪枝技术可以大大减少概率方法的开销。