This study evaluated the efficacy of two proposed methods in an operational standard-setting study conducted for a high-stakes language proficiency test of the U.S. government. The goal was to seek low-cost modifications to the existing Yes/No Angoff method to increase the validity and reliability of the recommended cut scores using a convergent mixed-methods study design. The study used the Yes/No ratings as the baseline method in two rounds of ratings, while differentiating the two methods by incorporating item maps and an Ordered Item Booklet, each of which is an integral tool of the Mapmark and the Bookmark methods. The results showed that the internal validity evidence is similar across both methods, especially after Round 2 ratings. When procedural validity evidence was considered, however, a preference emerged for the method where panelists conducted the initial ratings unbeknownst to the empirical item difficulty information, and then such information was provided on an item map as part of the Round 1 feedback. The findings highlight the importance of evaluating both internal and procedural validity evidence when considering standard-setting methods.

中文翻译：

---

评估语言能力评估中“是/否 Angoff”标准制定方法的方法增强

本研究评估了针对美国政府高风险语言能力测试进行的操作标准制定研究中两种拟议方法的有效性。目标是寻求对现有是/否 Angoff 方法的低成本修改，以使用收敛混合方法研究设计提高推荐分数的有效性和可靠性。该研究在两轮评级中使用是/否评级作为基线方法，同时通过合并项目地图和订购项目手册来区分这两种方法，这两种方法都是地图标记和书签方法的组成部分。结果表明，两种方法的内部有效性证据相似，尤其是在第二轮评级之后。然而，当考虑程序有效性证据时，出现了一种偏好的方法，即小组成员在不知道经验项目难度信息的情况下进行初始评级，然后在项目地图上提供此类信息作为第一轮反馈的一部分。研究结果强调了在考虑标准制定方法时评估内部和程序有效性证据的重要性。