当前位置: X-MOL 学术arXiv.cs.CL › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
arXiv - CS - Computation and Language Pub Date : 2020-11-21 , DOI: arxiv-2011.10819
Ondřej Dušek, Zdeněk Kasner

A major challenge in evaluating data-to-text (D2T) generation is measuring the semantic accuracy of the generated text, i.e. checking if the output text contains all and only facts supported by the input data. We propose a new metric for evaluating the semantic accuracy of D2T generation based on a neural model pretrained for natural language inference (NLI). We use the NLI model to check textual entailment between the input data and the output text in both directions, allowing us to reveal omissions or hallucinations. Input data are converted to text for NLI using trivial templates. Our experiments on two recent D2T datasets show that our metric can achieve high accuracy in identifying erroneous system outputs.

中文翻译:

利用自然语言推理评估数据到文本生成的语义准确性

评估数据到文本(D2T)生成的主要挑战是测量生成的文本的语义准确性,即检查输出文本是否包含输入数据支持的所有事实。我们提出了一种新的度量标准,用于基于针对自然语言推理(NLI)训练的神经模型评估D2T生成的语义准确性。我们使用NLI模型检查两个方向上输入数据和输出文本之间的文本含义,从而使我们能够发现遗漏或幻觉。使用普通模板将输入数据转换为NLI的文本。我们在两个最新的D2T数据集上进行的实验表明,我们的度量标准可以在识别错误​​的系统输出方面实现高精度。
更新日期:2020-11-25
down
wechat
bug