We investigated the potential impact of differential item functioning (DIF) on group-level mean and standard deviation estimates using empirical and simulated data in the context of large-scale assessment. For the empirical investigation, PISA 2018 cognitive domains (Reading, Mathematics, and Science) data were analyzed using Jackknife sampling to explore the impact of DIF on the country scores and their standard errors. We found that the countries that have a large number of DIF items tend to increase the difference of the country scores computed with and without the DIF adjustment. In addition, standard errors of the country score differences also increased with the number of DIF items. For the simulation study, we evaluated bias and root mean squared error (RMSE) of the group mean and standard deviation estimates using the multigroup item response theory (IRT) model to explore the extent to which DIF items create a bias of the group mean scores and how effectively the DIF adjustment corrects the bias under various conditions. We found that the DIF adjustment reduced the bias by 50% on average. The implications and limitations of the study are further discussed.

中文翻译：

---

在大规模评估的背景下，差异项目功能对小组分数报告的影响

我们在大规模评估的背景下使用经验和模拟数据调查了差异项目功能 (DIF) 对组级均值和标准差估计的潜在影响。对于实证调查，PISA 2018 认知领域（阅读、数学和科学）数据使用 Jackknife 抽样进行分析，以探索 DIF 对国家分数及其标准误差的影响。我们发现，拥有大量 DIF 项目的国家往往会增加使用和不使用 DIF 调整计算的国家分数的差异。此外，国家得分差异的标准误差也随着 DIF 项目数量的增加而增加。对于模拟研究，我们使用多组项目反应理论 (IRT) 模型评估了组均值和标准差估计的偏差和均方根误差 (RMSE)，以探索 DIF 项目在多大程度上产生组均值偏差以及 DIF 的有效性adjustment 纠正各种条件下的偏差。我们发现 DIF 调整平均减少了 50% 的偏差。进一步讨论了该研究的意义和局限性。