In multi-class classification, the output of a probabilistic classifier is a probability distribution of the classes. In this work, we focus on a statistical assessment of the reliability of probabilistic classifiers for multi-class problems. Our approach generates a Pearson \(\chi ^2\) statistic based on the k-nearest-neighbors in the prediction space. Further, we develop a Bayesian approach for estimating the expected power of the reliability test that can be used for an appropriate sample size k. We propose a sampling algorithm and demonstrate that this algorithm obtains a valid prior distribution. The effectiveness of the proposed reliability test and expected power is evaluated through a simulation study. We also provide illustrative examples of the proposed methods with practical applications.

在多类分类中，概率分类器的输出是类的概率分布。在这项工作中，我们专注于对多类问题的概率分类器可靠性的统计评估。我们的方法基于预测空间中的k最近邻生成 Pearson \(\chi ^2\)统计量。此外，我们开发了一种贝叶斯方法来估计可用于适当样本量k的可靠性测试的预期功效. 我们提出了一种采样算法，并证明该算法获得了有效的先验分布。通过模拟研究评估所提出的可靠性测试和预期功率的有效性。我们还提供了具有实际应用的拟议方法的说明性示例。