Complex scientific experiments from various domains are typically modeled as workflows and executed on large-scale machines using a Parallel Workflow Management System (WMS). Since such executions usually last for hours or days, some WMSs provide user steering support, i.e., they allow users to run data analyses and, depending on the results, adapt the workflows at runtime. A challenge in the parallel execution control design is to manage workflow data for efficient executions while enabling user steering support. Data access for high scalability is typically transaction-oriented, while for data analysis, it is online analytical-oriented so that managing such hybrid workloads makes the challenge even harder. In this work, we present SchalaDB, an architecture with a set of design principles and techniques based on distributed in-memory data management for efficient workflow execution control and user steering. We propose a distributed data design for scalable workflow task scheduling and high availability driven by a parallel and distributed in-memory DBMS. To evaluate our proposal, we develop d-Chiron, a WMS designed according to SchalaDB's principles. We carry out an extensive experimental evaluation on an HPC cluster with up to 960 computing cores. Among other analyses, we show that even when running data analyses for user steering, SchalaDB's overhead is negligible for workloads composed of hundreds of concurrent tasks on shared data. Our results encourage workflow engine developers to follow a parallel and distributed data-oriented approach not only for scheduling and monitoring but also for user steering.

中文翻译：

---

用于工作流执行的分布式内存数据管理

通常将来自各个领域的复杂科学实验建模为工作流，并使用并行工作流管理系统（WMS）在大型计算机上执行。由于此类执行通常持续数小时或数天，因此某些WMS提供用户指导支持，即，它们允许用户运行数据分析，并根据结果在运行时调整工作流程。并行执行控制设计中的一个挑战是管理工作流数据以实现高效执行，同时启用用户指导支持。具有高可伸缩性的数据访问通常是面向事务的，而对于数据分析，则是面向在线分析的，因此管理此类混合工作负载将使挑战更加艰巨。在这项工作中，我们介绍了SchalaDB，具有基于分布式内存数据管理的一组设计原理和技术的体系结构，以实现有效的工作流执行控制和用户指导。我们提出了一种分布式数据设计，用于由并行和分布式内存DBMS驱动的可伸缩工作流任务调度和高可用性。为了评估我们的建议，我们开发了d-Chiron，这是根据SchalaDB原理设计的WMS。我们对具有多达960个计算核心的HPC集群进行了广泛的实验评估。在其他分析中，我们表明，即使在运行数据分析以进行用户指导时，对于由数百个共享数据上的并行任务组成的工作负载，SchalaDB的开销也可以忽略不计。