Apache Spark is probably the most widely adopted framework for developing big-data batch applications and for executing them on a cluster of (virtual) machines. In general, the more resources (machines) one uses, the faster applications execute, but there is currently no adequate means to determine the proper size of a Spark cluster given time constraints, or to foresee execution times given the number of employed machines. One can only run these applications and use her/his experience to size the cluster and predict expected execution times. Wrong estimation of execution times can lead to costly overruns and overly long executions, thus calling for analytic sizing/prediction techniques that provide precise time guarantees. This paper addresses this problem by proposing a solution based on model-checking. The approach exploits a directed acyclic graph (DAG) to abstract the structure of the execution flows of Spark programs, annotates each node (Spark stage) with execution-related data, and formulates the identification of the global execution time as a reachability problem. To avoid the well-known state space explosion problem, the paper also proposes a technique to reduce the size of generated abstract models. This results in a significant decrease in used memory and/or verification time making our approach feasible for predicting the execution time of Spark applications given the resources available. The benefits of the proposed reduction technique are evaluated by using both timed automata and constraint LTL over clocks logic to formally encode and analyze generated models. The approach is also successfully validated on some realistic case studies. Since the optimization is not Spark-specific, we claim that it can be applied to a wide range of applications whose underlying model can be abstracted as a DAG.

中文翻译：

---

使用形式验证评估 Spark 应用程序的执行时间

Apache Spark 可能是用于开发大数据批处理应用程序并在（虚拟）机器集群上执行它们的最广泛采用的框架。一般来说，一个人使用的资源（机器）越多，应用程序的执行速度就越快，但是目前还没有足够的方法来确定给定时间限制的 Spark 集群的适当大小，或者在给定使用的机器数量的情况下预测执行时间。一个人只能运行这些应用程序并使用她/他的经验来确定集群的大小并预测预期的执行时间。执行时间的错误估计会导致代价高昂的超支和过长的执行，因此需要提供精确时间保证的分析大小/预测技术。本文通过提出基于模型检查的解决方案来解决这个问题。该方法利用有向无环图 (DAG) 来抽象 Spark 程序执行流的结构，用与执行相关的数据注释每个节点（Spark 阶段），并将全局执行时间的识别表述为可达性问题。为了避免众所周知的状态空间爆炸问题，本文还提出了一种减少生成的抽象模型大小的技术。这导致使用的内存和/或验证时间显着减少，使我们的方法可以在给定可用资源的情况下预测 Spark 应用程序的执行时间。通过使用定时自动机和约束 LTL over clocks 逻辑来正式编码和分析生成的模型，来评估所提出的缩减技术的好处。该方法还在一些现实案例研究中成功验证。由于优化不是 Spark 特定的，我们声称它可以应用于广泛的应用程序，其底层模型可以抽象为 DAG。