This work contributes a research protocol for evaluating human-AI interaction in the context of specific AI products. The research protocol enables UX and HCI researchers to assess different human-AI interaction solutions and validate design decisions before investing in engineering. We present a detailed account of the research protocol and demonstrate its use by employing it to study an existing set of human-AI interaction guidelines. We used factorial surveys with a 2 × 2 mixed design to compare user perceptions when a guideline is applied versus violated, under conditions of optimal versus sub-optimal AI performance. The results provided both qualitative and quantitative insights into the UX impact of each guideline. These insights can support creators of user-facing AI systems in their nuanced prioritization and application of the guidelines.

这项工作提供了一个研究协议，用于评估特定人工智能产品背景下的人机交互。该研究协议使用户体验和人机交互研究人员能够在投资工程之前评估不同的人机交互解决方案并验证设计决策。我们详细介绍了该研究方案，并通过使用它来研究一套现有的人类与人工智能交互指南来展示其用途。我们使用 2 × 2 混合设计的因子调查来比较用户在最佳与次优 AI 性能条件下应用与违反指南时的感受。结果为每条指南的用户体验影响提供了定性和定量的见解。这些见解可以支持面向用户的人工智能系统的创建者对指南进行细致入微的优先级排序和应用。