Abstract

In several observational contexts where different raters evaluate a set of items, it is common to assume that all raters draw their scores from the same underlying distribution. However, a plenty of scientific works have evidenced the relevance of individual variability in different type of rating tasks. To address this issue the intra-class correlation coefficient (ICC) has been used as a measure of variability among raters within the Hierarchical Linear Models approach. A common distributional assumption in this setting is to specify hierarchical effects as independent and identically distributed from a normal with the mean parameter fixed to zero and unknown variance. The present work aims to overcome this strong assumption in the inter-rater agreement estimation by placing a Dirichlet Process Mixture over the hierarchical effects’ prior distribution. A new nonparametric index \(\lambda\) is proposed to quantify raters polarization in presence of group heterogeneity. The model is applied on a set of simulated experiments and real world data. Possible future directions are discussed.

中文翻译：

---

评估者间一致性分析中的混合极化：贝叶斯非参数指数

摘要

在不同的评估者评估一组项目的几种观察环境中，通常假设所有评估者都从相同的基础分布中得出分数。然而，大量科学著作已经证明了不同类型的评级任务中个体差异的相关性。为了解决这个问题，在分层线性模型方法中，类内相关系数 (ICC) 被用作评估者之间变异性的衡量标准。此设置中的常见分布假设是将分层效应指定为与均值参数固定为零且方差未知的正态独立且同分布。目前的工作旨在通过将狄利克雷过程混合放置在分层效应的先验分布上来克服评估者间一致性估计中的这种强烈假设。提出了一种新的非参数指数\(\lambda\)来量化存在群体异质性的情况下评分者的极化。该模型应用于一组模拟实验和现实世界数据。讨论了未来可能的方向。