Large neural models have brought a new challenge to natural language generation (NLG): it has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.

中文翻译：

---

测量自然语言生成模型中的归因

大型神经模型给自然语言生成（NLG）带来了新的挑战：确保自由生成的模型输出的安全性和可靠性已成为当务之急。为此，我们提出了一个评估框架，可归因于已识别来源（AIS），规定与外部世界相关的 NLG 输出将根据独立的、提供的来源进行验证。我们定义了 AIS 和两阶段注释管道，以允许注释者根据注释指南评估模型输出。我们在跨越三个任务的生成数据集（两个会话 QA 数据集、一个摘要数据集和一个表到文本数据集）上成功验证了这种方法。我们在附录中提供了完整的注释指南，并在 https://github.com/google-research-datasets/AIS 上公开发布注释数据。