We study the problem of task reallocation for load-balancing of MapReduce jobs in applications that process large datasets. In this context, we propose a novel strategy based on cooperative agents used to optimize the task scheduling in a single MapReduce job. The novelty of our strategy lies in the ability of agents to identify opportunities within a current unbalanced allocation, which in turn triggers concurrent and one-to-many negotiations amongst agents to locally reallocate some of the tasks within a job. Our contribution is that tasks are reallocated according to the proximity of the resources and they are performed in accordance to the capabilities of the nodes in which agents are situated. To evaluate the adaptivity and responsiveness of our approach, we implement a prototype test-bed and conduct a vast panel of experiments in a heterogeneous environment and by exploring varying hardware configurations. This extensive experimentation reveals that our strategy significantly improves the overall runtime over the classical Hadoop data processing.

我们在处理大型数据集的应用程序中研究了任务重新分配的问题，以实现MapReduce作业的负载平衡。在这种情况下，我们提出了一种基于协作代理的新颖策略，用于优化单个MapReduce作业中的任务调度。我们策略的新颖之处在于代理能够识别当前不平衡分配中的机会的能力，这又会触发代理之间的并发和一对多谈判，以在本地重新分配工作中的某些任务。我们的贡献是根据资源的接近程度重新分配任务，并根据代理所位于的节点的功能来执行任务。为了评估我们方法的适应性和响应能力，我们实现了原型测试台，并在异构环境中并通过探索各种硬件配置进行了大量的实验。这项广泛的实验表明，与传统的Hadoop数据处理相比，我们的策略显着改善了总体运行时间。