This paper reports the results and post-challenge analyses of ChaLearn’s AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a “meta-learner”, “data ingestor”, “model selector”, “model/learner”, and “evaluator”. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark ( http://autodl.chalearn.org ), the open-sourced code of the winners, and a free “AutoDL self-service.”

中文翻译：

---

ChaLearn AutoDL Challenge 2019 的获胜解决方案和挑战后分析

本文报告了 ChaLearn 的 AutoDL 挑战系列的结果和挑战后分析，该系列有助于整理出大量用于深度学习 (DL) 的 AutoML 解决方案，这些解决方案已在各种设置中引入，但缺乏公平的比较。所有输入数据模态（时间序列、图像、视频、文本、表格）都被格式化为张量，所有任务都是多标签分类问题。代码提交是在隐藏任务上执行的，时间和计算资源有限，推动了快速获得结果的解决方案。在这种情况下，深度学习方法占主导地位，尽管流行的神经架构搜索 (NAS) 是不切实际的。解决方案依赖于经过微调的预训练网络，其架构与数据模态相匹配。挑战后的测试没有显示超出规定时间限制的改进。虽然没有一个组件是特别原始或新颖的，但出现了一个高级模块化组织，其特点是“元学习器”、“数据摄取器”、“模型选择器”、“模型/学习器”和“评估器”。这种模块化使消融研究成为可能，这揭示了（平台外）元学习、集成和高效数据管理的重要性。异构模块组合的实验进一步证实了获胜解决方案的（局部）最优性。我们的挑战遗产包括一个永恒的基准（异构模块组合的实验进一步证实了获胜解决方案的（局部）最优性。我们的挑战遗产包括一个永恒的基准（异构模块组合的实验进一步证实了获胜解决方案的（局部）最优性。我们的挑战遗产包括一个永恒的基准（ http://autodl.chalearn.org )、获奖者的开源代码和免费的“AutoDL 自助服务”。