Discovery environments are ubiquitous in academic libraries but studying their effectiveness and use in an academic environment has mostly centered around user satisfaction, experience, and task analysis. This study aims to create a quantitative, reproducible framework to test the relevancy of results and the overall success of Washington State University’s discovery environment (Primo by Ex Libris). Within this framework, the authors use bibliographic citations from student research papers submitted as part of a required university class as the proxy for relevancy. In the context of this study, the researchers created a testing model that includes: (1) a process to produce machine-generated keywords from a corpus of research papers to compare against a set of human-created keywords, (2) a machine process to query a discovery environment to produce search result lists to compare against citation lists, and (3) four metrics to measure the comparative success of different search strategies and the relevancy of the results. This framework is used to move beyond a sentiment or task-based analysis to measure if materials cited in student papers appear in the results list of a production discovery environment. While this initial test of the framework produced fewer matches between researcher-generated search results and student bibliography sources than expected, the authors note that faceted searches represent a greater success rate when compared to open-ended searches. Future work will include comparative (A/B) testing of commonly deployed discovery layer configurations and limiters to measure the impact of local decisions on discovery layer efficacy as well as noting where in the results list a citation match occurs.

中文翻译：

---

测量发现环境中相关性的框架

发现环境在学术图书馆中无处不在，但研究它们在学术环境中的有效性和使用主要集中在用户满意度、体验和任务分析上。本研究旨在创建一个定量的、可重复的框架，以测试结果的相关性和华盛顿州立大学发现环境（Ex Libris 的 Primo）的整体成功。在此框架内，作者使用学生研究论文中的书目引用作为必修大学课程的一部分作为相关性的代理。在本研究的背景下，研究人员创建了一个测试模型，其中包括：（1）从研究论文的语料库中生成机器生成的关键字，以与一组人工创建的关键字进行比较的过程，(2) 查询发现环境以生成搜索结果列表以与引文列表进行比较的机器过程，以及 (3) 衡量不同搜索策略的比较成功和结果相关性的四个指标。该框架用于超越基于情绪或任务的分析，以衡量学生论文中引用的材料是否出现在生产发现环境的结果列表中。虽然该框架的初步测试在研究人员生成的搜索结果和学生书目来源之间产生的匹配比预期的要少，但作者指出，与开放式搜索相比，分面搜索代表了更高的成功率。