Evaluation metrics such as precision, recall and normalized discounted cumulative gain have been widely applied in ad hoc retrieval experiments. They have facilitated the assessment of system performance in various topics over the past decade. However, the effectiveness of such metrics in capturing users’ in-situ search experience, especially in complex search tasks that trigger interactive search sessions, is limited. To address this challenge, it is necessary to adaptively adjust the evaluation strategies of search systems to better respond to users’ changing information needs and evaluation criteria. In this work, we adopt a taxonomy of search task states that a user goes through in different scenarios and moments of search sessions, and perform a meta-evaluation of existing metrics to better understand their effectiveness in measuring user satisfaction. We then built models for predicting task states behind queries based on in-session signals. Furthermore, we constructed and meta-evaluated new state-aware evaluation metrics. Our analysis and experimental evaluation are performed on two datasets collected from a field study and a laboratory study, respectively. Results demonstrate that the effectiveness of individual evaluation metrics varies across task states. Meanwhile, task states can be detected from in-session signals. Our new state-aware evaluation metrics could better reflect in-situ user satisfaction than an extensive list of the widely used measures we analyzed in this work in certain states. Findings of our research can inspire the design and meta-evaluation of user-centered adaptive evaluation metrics, and also shed light on the development of state-aware interactive search systems.

精确率、召回率和归一化贴现累积增益等评价指标已广泛应用于即席检索实验中。在过去的十年中，它们促进了对各个主题的系统性能的评估。然而，此类指标在捕获用户现场搜索体验方面的有效性是有限的，尤其是在触发交互式搜索会话的复杂搜索任务中。为了应对这一挑战，有必要自适应地调整搜索系统的评估策略，以更好地响应用户不断变化的信息需求和评估标准。在这项工作中，我们采用了搜索任务状态的分类法，即用户在搜索会话的不同场景和时刻经历的情况，并对现有指标进行元评估，以更好地了解它们在衡量用户满意度方面的有效性。然后，我们构建了模型，用于根据会话中信号预测查询背后的任务状态。此外，我们构建并元评估了新的状态感知评估指标。我们的分析和实验评估分别是在现场研究和实验室研究中收集的两个数据集上进行的。结果表明，各个评估指标的有效性因任务状态而异。同时，可以从会话中信号检测任务状态。我们新的状态感知评估指标可以更好地反映现场用户满意度，而不是我们在本工作中分析的某些州广泛使用的衡量指标的广泛列表。我们的研究结果可以启发以用户为中心的自适应评估指标的设计和元评估，并为状态感知交互式搜索系统的开发提供启示。