Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context. In many scenarios, a desirable behavior is that LLMs give precedence to contextual knowledge when it conflicts with the parametric knowledge, and fall back to using their parametric knowledge when the context is irrelevant. This enables updating and correcting the model's knowledge by in-context editing instead of retraining. Previous works have shown that LLMs are inclined to ignore contextual knowledge and fail to reliably fall back to parametric knowledge when presented with irrelevant context. In this work, we discover that, with proper prompting methods, instruction-finetuned LLMs can be highly controllable by contextual knowledge and robust to irrelevant context. Utilizing this feature, we propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. To better evaluate the robustness of model editors, we collect a new dataset, that contains irrelevant questions that are more challenging than the ones in existing datasets. Empirical results show that our method outperforms current state-of-the-art methods by a large margin. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs (and vice versa). The source code can be found at https://github.com/thunlp/EREN.

中文翻译：

---

适用于大型语言模型的稳健且可扩展的模型编辑

大型语言模型 (LLM) 可以使用参数知识（模型权重中编码的知识）或上下文知识（上下文中呈现的知识）进行预测。在许多情况下，理想的行为是法学硕士在上下文知识与参数知识冲突时优先考虑上下文知识，并在上下文不相关时回退到使用参数知识。这使得可以通过上下文编辑而不是重新训练来更新和纠正模型的知识。之前的研究表明，法学硕士倾向于忽略上下文知识，并且在出现不相关上下文时无法可靠地回归到参数知识。在这项工作中，我们发现，通过适当的提示方法，经过教学微调的法学硕士可以通过上下文知识实现高度可控，并对不相关的上下文具有鲁棒性。利用这一特性，我们提出了EREN（Edit models by READing Notes）来提高LLM编辑的可扩展性和鲁棒性。为了更好地评估模型编辑器的稳健性，我们收集了一个新的数据集，其中包含比现有数据集中的问题更具挑战性的不相关问题。实证结果表明，我们的方法大大优于当前最先进的方法。与现有技术不同，它可以整合来自多个编辑的知识，并正确响应语法相似但语义不相关的输入（反之亦然）。源代码可以在 https://github.com/thunlp/EREN 找到。