Analyzing large datasets with distributed dataflow systems requires the use of clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. However, picking the appropriate resources in both type and number can often be challenging, as the selected configuration needs to match a distributed dataflow job's resource demands and access patterns. A good cluster configuration avoids hardware bottlenecks and maximizes resource utilization, avoiding costly overprovisioning. We propose a collaborative approach for finding optimal cluster configurations based on sharing and learning from historical runtime data of distributed dataflow jobs. Collaboratively shared data can be utilized to predict runtimes of future job executions through the use of specialized regression models. However, training prediction models on historical runtime data that were produced by different users and in diverse contexts requires the models to take these contexts into account.

中文翻译：

---

分布式数据流作业集群配置的协同优化

使用分布式数据流系统分析大型数据集需要使用集群。公共云提供商提供了可用于此类集群的大量资源。然而，选择合适的资源类型和数量通常具有挑战性，因为所选配置需要匹配分布式数据流作业的资源需求和访问模式。良好的集群配置可避免硬件瓶颈并最大限度地提高资源利用率，避免代价高昂的过度配置。我们提出了一种基于共享和学习分布式数据流作业的历史运行时数据来寻找最佳集群配置的协作方法。通过使用专门的回归模型，可以利用协作共享的数据来预测未来作业执行的运行时间。然而，