ABSTRACT

Human ratings are ubiquitous in creativity research. Yet, the process of rating responses to creativity tasks – typically several hundred or thousands of responses, per rater – is often time-consuming and expensive. Planned missing data designs, where raters only rate a subset of the total number of responses, have been recently proposed as one possible solution to decrease overall rating time and monetary costs. However, researchers also need ratings that adhere to psychometric standards, such as a certain degree of reliability, and psychometric work with planned missing designs is currently lacking in the literature. In this work, we introduce how judge response theory and simulations can be used to fine-tune planning of missing data designs. We provide open code for the community and illustrate our proposed approach by a cost-effectiveness calculation based on a realistic example. We clearly show that fine-tuning helps to save time (to perform the ratings) and monetary costs, while simultaneously targeting expected levels of reliability.

摘要

人类评分在创造力研究中无处不在。然而，对创造力任务的反应进行评级的过程（通常每个评估者有数百或数千个反应）通常既耗时又昂贵。最近提出了计划中的缺失数据设计，即评级者仅对响应总数的子集进行评级，作为减少总体评级时间和金钱成本的一种可能的解决方案。然而，研究人员还需要符合心理测量标准的评级，例如一定程度的可靠性，而目前文献中缺乏计划缺失设计的心理测量工作。在这项工作中，我们介绍了如何使用判断响应理论和模拟来微调缺失数据设计的规划。我们为社区提供开放代码，并通过基于实际示例的成本效益计算来说明我们提出的方法。我们清楚地表明，微调有助于节省时间（执行评级）和金钱成本，同时瞄准预期的可靠性水平。