This work presents three new adaptive optimization techniques to maximize the performance of dispel4py workflows. dispel4py is a parallel Python-based stream-oriented dataflow framework that acts as a bridge to existing parallel programming frameworks like MPI or Python multiprocessing. When a user runs a dispel4py workflow, the original framework performs a fixed workload distribution among the processes available for the run. This allocation does not take into account the features of the workflows, which can cause scalability issues, especially for data-intensive scientific workflows. Our aim, therefore, is to improve the performance of dispel4py workflows by testing different workload strategies that automatically adapt to workflows at runtime. For achieving this objective, we have implemented three new techniques, called Naive Assignment, Staging and Dynamic Scheduling. We have evaluated our proposal with several workflows from different domains and across different computing resources. The results show that our proposed techniques have significantly (up to 10X) improved the performance of the original dispel4py framework.

这项工作提出了三种新的自适应优化技术，以最大限度地提高dispel4py工作流程的性能。dispel4py是一个基于并行 Python 的面向流的数据流框架，它充当现有并行编程框架（如 MPI 或 Python 多处理）的桥梁。当用户运行 dispel4py 工作流时，原始框架在可用于运行的进程之间执行固定的工作负载分配。这种分配没有考虑工作流的特性，这可能会导致可扩展性问题，尤其是对于数据密集型科学工作流。因此，我们的目标是提高dispel4py的性能通过测试在运行时自动适应工作流的不同工作负载策略来优化工作流。为了实现这一目标，我们实施了三种新技术，称为Naive Assignment、Staging和Dynamic Sc​​heduling。我们已经使用来自不同领域和不同计算资源的多个工作流评估了我们的提案。结果表明，我们提出的技术显着（高达 10 倍）提高了原始dispel4py框架的性能。