Recently, there have been several proposals in the area of geo-distributed big data processing. In this work, we aim to address a limitation of the existing solutions, namely to optimize task allocation across geographically distributed data centers, in a way that both the total traffic and the running time of the whole processing in complex multi-stage flows are targeted. Apart from proposing concrete efficient solutions for this combinatorial problem, we advocate to take a critical stand on the broadly spread claim that transferring distributed data to a single or fewer places is too costly. In our proposal, we judiciously reduce the participation of some data centers in the flow execution, and we show that, in a wide range of settings, this yields significant benefits. We show that a stochastic solution is superior to a fast greedy one, at the expense of optimization time of up to a few minutes. Compared to a state-of-the-art solution, we manage to decrease total traffic by 44% and running time by 37% on average. In several cases, the improvements can reach 1-2 orders of magnitude. Moreover, we provide evidence that simple heuristics are inferior. Our experimental evaluation comprises both extensive simulations and real runs in Spark.

最近，在地理分布大数据处理领域中已经提出了一些建议。在这项工作中，我们旨在解决现有解决方案的局限性，即优化跨地理分布的数据中心的任务分配，以针对复杂多级流程中的总流量和整个处理的运行时间为目标。除了提出针对此组合问题的具体有效解决方案外，我们还主张对广泛传播的主张持批判立场，因为这种主张将分布式数据传输到单个或更少的地方过于昂贵。在我们的建议中，我们明智地减少了一些数据中心在流程执行中的参与，并且我们证明了，在各种各样的设置中，这产生了明显的好处。我们证明了随机解决方案优于快速贪婪的解决方案，以花费几分钟的优化时间为代价。与最新解决方案相比，我们设法将总流量减少了44％，将运行时间平均减少了37％。在某些情况下，这些改进可以达到1-2个数量级。此外，我们提供的证据表明，简单的启发式方法较差。我们的实验评估包括Spark中的大量模拟和实际运行。