Large-scale language models pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. However, as our systematic examination reveals, it is still challenging for such models to generate coherent long passages of text ($>$1000 tokens), especially when the models are fine-tuned to the target domain on a small corpus. To overcome the limitation, we propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution. Our method first produces domain-specific content keywords and then progressively refines them into complete passages in multiple stages. The simple design allows our approach to take advantage of pretrained language models at each stage and effectively adapt to any target domain given only a small set of examples. We conduct a comprehensive empirical study with a broad set of evaluation metrics, and show that our approach significantly improves upon the fine-tuned GPT-2 in terms of domain-specific quality and sample efficiency. The coarse-to-fine nature of progressive generation also allows for a higher degree of control over the generated content.

中文翻译：

---

长文本的渐进式生成

在大量文本语料库（如 GPT-2）上预训练的大规模语言模型是强大的开放域文本生成器。然而，正如我们的系统检查所揭示的那样，这些模型生成连贯的长文本段落（$>$1000 标记）仍然具有挑战性，尤其是当模型在小型语料库中针对目标域进行微调时。为了克服这个限制，我们提出了一种以渐进方式生成文本的简单而有效的方法，其灵感来自于生成从低分辨率到高分辨率的图像。我们的方法首先生成特定领域的内容关键字，然后在多个阶段逐步将它们细化为完整的段落。简单的设计使我们的方法能够在每个阶段利用预训练的语言模型，并有效地适应任何仅给定一小组示例的目标领域。我们使用广泛的评估指标进行了全面的实证研究，并表明我们的方法在特定领域的质量和样本效率方面显着改进了微调的 GPT-2。渐进生成的由粗到细的特性还允许对生成的内容进行更高程度的控制。