Physics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more efficient and consistent than human coding, allowing much larger and broader datasets to be analyzed than is practical with human coders. Existing work that uses these tools, however, does not investigate norms that allow for trustworthy quantitative claims without full reliance on cross-checking with human coding, which defeats the purpose of using these automated tools. Here we propose a four-part method for making such claims with supervised natural language processing: evaluating a trained model, calculating statistical uncertainty, calculating systematic uncertainty from the trained algorithm, and calculating systematic uncertainty from novel data sources. We provide evidence for this method using data from two distinct short response survey questions with two distinct coding schemes. We also provide a real-world example of using these practices to machine code a dataset unseen by human coders. We offer recommendations to guide physics education researchers who may use machine-coding methods in the future.

中文翻译：

---

大规模评估机器编码可信度的方法

物理教育研究人员有兴趣使用机器学习和自然语言处理工具从自然语言和文本数据中提出定量主张，例如对调查问题的开放式回答。我们的愿望是，这种形式的机器编码可能比人类编码更高效、更一致，从而可以分析比人类编码人员实际情况更大、更广泛的数据集。然而，使用这些工具的现有工作并没有研究允许可信定量声明的规范，而无需完全依赖与人类编码的交叉检查，这违背了使用这些自动化工具的目的。在这里，我们提出了一种通过监督自然语言处理提出此类主张的四部分方法：评估训练模型，计算统计不确定性，根据训练算法计算系统不确定性，以及根据新数据源计算系统不确定性。我们使用来自两个不同的短响应调查问题和两种不同编码方案的数据为该方法提供了证据。我们还提供了一个使用这些实践对人类编码员看不到的数据集进行机器编码的真实示例。我们提供建议来指导未来可能使用机器编码方法的物理教育研究人员。