Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code and trained models available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.

中文翻译：

---

PICARD：增量解析语言模型的约束自回归解码

用于文本数据的大型预训练语言模型具有不受约束的输出空间；在每个解码步骤中，它们可以生成 10,000 个子词标记中的任何一个。当针对 SQL 等受约束的形式语言进行微调时，这些模型通常会生成无效代码，使其无法使用。我们提出了 PICARD（可在 https://github.com/ElementAI/picard 获得的代码和训练模型），这是一种通过增量解析来约束语言模型的自回归解码器的方法。PICARD 通过在每个解码步骤拒绝不可接受的令牌来帮助找到有效的输出序列。在具有挑战性的 Spider 和 CoSQL 文本到 SQL 的翻译任务中，我们展示了 PICARD 将微调的 T5 模型与可通过的性能转换为最先进的解决方案。