Contemporary organizations increasingly adopt conversational agents (CAs) as intelligent and natural language-based solutions for providing services and information. CAs offer new forms of personalization, speed, (cost-)effectiveness, and automation. However, despite their hype in research and practice, many organizations still fail to seize CAs’ potential because they lack knowledge of how to evaluate and improve the quality of CAs to sustain them in organizational operations. We aim to fill this knowledge gap by conducting a design science research project in which we aggregate insights from the literature and practice to derive an applicable set of quality criteria for CAs. Our article contributes to CA research and guides practitioners by providing a blueprint to structure the evaluation of CAs and to discover areas for systematic improvement.

当代组织越来越多地采用会话代理 (CA) 作为基于自然语言的智能解决方案来提供服务和信息。CA 提供新形式的个性化、速度、（成本）效益和自动化。然而，尽管在研究和实践上大肆宣传，许多组织仍然未能抓住CA的潜力，因为他们缺乏如何评估和提高CA质量以维持其在组织运营中的知识。我们的目标是通过开展一个设计科学研究项目来填补这一知识空白，在该项目中，我们汇总了文献和实践中的见解，得出一套适用于 CA 的质量标准。我们的文章通过提供构建 CA 评估的蓝图并发现需要系统改进的领域，为 CA 研究做出贡献并指导从业者。