当前位置:
X-MOL 学术
›
arXiv.cs.LO
›
论文详情
Our official English website, www.x-mol.net, welcomes your
feedback! (Note: you will need to create a separate account there.)
LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
arXiv - CS - Logic in Computer Science Pub Date : 2021-01-15 , DOI: arxiv-2101.06223 Yuhuai Wu, Markus Rabe, Wenda Li, Jimmy Ba, Roger Grosse, Christian Szegedy
arXiv - CS - Logic in Computer Science Pub Date : 2021-01-15 , DOI: arxiv-2101.06223 Yuhuai Wu, Markus Rabe, Wenda Li, Jimmy Ba, Roger Grosse, Christian Szegedy
While designing inductive bias in neural architectures has been widely
studied, we hypothesize that transformer networks are flexible enough to learn
inductive bias from suitable generic tasks. Here, we replace architecture
engineering by encoding inductive bias in the form of datasets. Inspired by
Peirce's view that deduction, induction, and abduction form an irreducible set
of reasoning primitives, we design three synthetic tasks that are intended to
require the model to have these three abilities. We specifically design these
synthetic tasks in a way that they are devoid of mathematical knowledge to
ensure that only the fundamental reasoning biases can be learned from these
tasks. This defines a new pre-training methodology called "LIME" (Learning
Inductive bias for Mathematical rEasoning). Models trained with LIME
significantly outperform vanilla transformers on three very different large
mathematical reasoning benchmarks. Unlike dominating the computation cost as
traditional pre-training approaches, LIME requires only a small fraction of the
computation cost of the typical downstream task.
中文翻译:
LIME:学习归纳偏见的数学推理基元
尽管在神经体系结构中设计归纳偏置已得到广泛研究,但我们假设变压器网络足够灵活,可以从合适的通用任务中学习归纳偏置。在这里,我们通过以数据集的形式编码归纳偏差来代替体系结构工程。受皮尔士认为演绎,归纳和绑架构成不可还原的推理原语集合的启发,我们设计了三个综合任务,旨在要求模型具有这三个能力。我们以没有数学知识的方式专门设计了这些合成任务,以确保只能从这些任务中学习基本的推理偏见。这定义了一种新的预训练方法,称为“ LIME”(学习数学推理的归纳偏差)。在三个非常不同的大型数学推理基准上,使用LIME训练的模型明显优于香草变压器。与将计算成本控制为传统的预训练方法不同,LIME仅需要典型下游任务的计算成本的一小部分。
更新日期:2021-01-18
中文翻译:
LIME:学习归纳偏见的数学推理基元
尽管在神经体系结构中设计归纳偏置已得到广泛研究,但我们假设变压器网络足够灵活,可以从合适的通用任务中学习归纳偏置。在这里,我们通过以数据集的形式编码归纳偏差来代替体系结构工程。受皮尔士认为演绎,归纳和绑架构成不可还原的推理原语集合的启发,我们设计了三个综合任务,旨在要求模型具有这三个能力。我们以没有数学知识的方式专门设计了这些合成任务,以确保只能从这些任务中学习基本的推理偏见。这定义了一种新的预训练方法,称为“ LIME”(学习数学推理的归纳偏差)。在三个非常不同的大型数学推理基准上,使用LIME训练的模型明显优于香草变压器。与将计算成本控制为传统的预训练方法不同,LIME仅需要典型下游任务的计算成本的一小部分。