ABSTRACT

The aim of the current research was to provide recommendations to facilitate the development and use of anchoring vignettes (AVs) for cross-cultural comparisons in education. Study 1 identified six factors leading to order violations and ties in AV responses based on cognitive interviews with 15-year-old students. The factors were categorized into three domains: varying levels of AV format familiarity, differential interpretations of content, and individual differences in processing information. To inform the most appropriate approach to treat order violations and re-scaling method, Study 2 conducted Monte Carlo simulations with the manipulation of three factors and incorporation of five responses styles. Study 2 found that the AV approach improved accuracy in score estimation by successfully controlling for response styles. The results also revealed the reordering approach to treat order violations produced the most accurate estimation combined with the re-scaling method assigning the middle value among possible scores. Along with strategies to develop AVs, strengths and limitations of the implemented nonparametric AV approach were discussed in comparison to item response theory modeling for response styles.

摘要

当前研究的目的是提供建议，以促进锚定小插曲 (AV) 的开发和使用，用于教育中的跨文化比较。研究 1 根据对 15 岁学生的认知访谈确定了导致 AV 响应中的顺序违规和联系的六个因素。这些因素分为三个领域：不同程度的 AV 格式熟悉程度、对内容的不同解释以及处理信息的个体差异。为了告知处理订单违规和重新缩放方法的最合适方法，研究 2 进行了蒙特卡罗模拟，操作三个因素并结合了五种响应方式。研究 2 发现 AV 方法通过成功控制响应方式提高了分数估计的准确性。结果还揭示了处理订单违规的重新排序方法产生了最准确的估计，结合重新调整方法在可能的分数中分配中间值。除了开发 AV 的策略外，还讨论了所实施的非参数 AV 方法的优势和局限性，并与响应风格的项目响应理论建模进行了比较。