Modern data analytics systems use long-running executors to run an application’s entire DAG. Executors exhibit salient time-varying resource requirements. Yet, existing schedulers simply reserve resources for executors statically, and use the peak resource demand to guide executor placement. This leads to low utilization and poor application performance. We present Elasecutor, a novel executor scheduler for data analytics systems. Elasecutor dynamically allocates and explicitly sizes resources to executors over time according to the predicted time/varying resource demands. Rather than placing executors using their peak demand, Elasecutor strategically assigns them to machines based on a concept called dominant remaining resource to minimize resource fragmentation. Elasecutor further adaptively reprovisions resources in order to tolerate inaccurate demand prediction and reschedules tasks to deal with inadequate reprovisioning resources on one machine. Testbed evaluation on a 35-node cluster with our Spark-based prototype implementation shows that Elasecutor reduces makespan by more than 36% on average, and improves cluster utilization by up to 55% compared to existing work.

中文翻译：

---

Elasecutor：数据分析系统中的弹性执行器调度

现代数据分析系统使用运行时间较长的执行程序来运行应用程序的整个DAG。执行者表现出明显的时变资源需求。但是，现有的调度程序只是简单地为执行者静态地保留资源，并使用高峰资源需求来指导执行者的位置。这导致利用率低和应用程序性能差。我们介绍了Elasecutor，这是一种用于数据分析系统的新型执行程序调度程序。Elasecutor会根据预测的时间/不断变化的资源需求，动态地为执行者动态分配资源并显式调整其大小。Elasecutor不会使用高峰执行者来放置执行者，而是根据称为主导剩余资源以最大程度地减少资源碎片。Elasecutor进一步自适应地调配资源，以容许不准确的需求预测，并重新安排任务以处理一台机器上的调配资源不足。使用我们基于Spark的原型实现对35个节点的群集进行的测试台评估显示，与现有工作相比，Elasecutor平均可将制造跨度降低36％以上，并将群集利用率提高多达55％。