Tasks in multi-task learning often correlate, conflict, or even compete with each other. As a result, a single solution that is optimal for all tasks rarely exists. Recent papers introduced the concept of Pareto optimality to this field and directly cast multi-task learning as multi-objective optimization problems, but solutions returned by existing methods are typically finite, sparse, and discrete. We present a novel, efficient method that generates locally continuous Pareto sets and Pareto fronts, which opens up the possibility of continuous analysis of Pareto optimal solutions in machine learning problems. We scale up theoretical results in multi-objective optimization to modern machine learning problems by proposing a sample-based sparse linear system, for which standard Hessian-free solvers in machine learning can be applied. We compare our method to the state-of-the-art algorithms and demonstrate its usage of analyzing local Pareto sets on various multi-task classification and regression problems. The experimental results confirm that our algorithm reveals the primary directions in local Pareto sets for trade-off balancing, finds more solutions with different trade-offs efficiently, and scales well to tasks with millions of parameters.

中文翻译：

---

多任务学习中的高效连续帕累托探索

多任务学习中的任务经常相互关联、冲突，甚至相互竞争。因此，很少存在对所有任务都最佳的单一解决方案。最近的论文将帕累托最优的概念引入该领域，并将多任务学习直接转化为多目标优化问题，但现有方法返回的解决方案通常是有限的、稀疏的和离散的。我们提出了一种新颖、有效的方法，可以生成局部连续的帕累托集和帕累托前沿，这开辟了在机器学习问题中连续分析帕累托最优解的可能性。我们通过提出一个基于样本的稀疏线性系统，将多目标优化的理论结果扩展到现代机器学习问题，机器学习中可以应用标准的无 Hessian 求解器。我们将我们的方法与最先进的算法进行比较，并展示其在各种多任务分类和回归问题上分析局部帕累托集的用法。实验结果证实，我们的算法揭示了局部帕累托集中用于权衡平衡的主要方向，有效地找到了更多具有不同权衡的解决方案，并且可以很好地扩展到具有数百万个参数的任务。