Here, we describe the development and validation of an automatic assessment system that examines students’ hand-drawn visual representations in free-response items. The data were collected from 1,028 students in the second through 11th grades in South Korea using two items from the Test About Particles in a Gas questionnaire (Novick & Nussbaum, 1981). Students’ free responses, which include hand drawings and writing, were coded for two dimensions — structural (particulate/continuous/other) and distributional (expanded/concentrated/other). Machine learning (ML) models were trained to assess the responses on the particulate nature of matter. For classifying hand drawings, a pre-trained Inception-v3 model followed by a support vector machine was trained and its performance was evaluated. The assessment model yielded high machine-human agreement (MHA) (kappa = 0.732–0.926, accuracy = 0.820–0.942, precision = 0.817–0.941, recall = 0.820–0.942, F1 = 0.818–0.941, and area under the curve [AUC] = 0.906–0.990). Students’ written responses were tokenized, and a dictionary of scientific semantic scores was prepared. The final model for the overall assessment of both drawing and writing yielded high MHA (kappa = 0.800–0.881, accuracy = 0.859–0.956, precision = 0.865–0.957, recall = 0.859–0.956, F1 = 0.859–0.956, and AUC = 0.944–0.995), which varied by the final classifiers of the models. There were some variances in the performance of the assessment model according to the school level. This study suggests that artificial intelligence can be used to automate assessments of students’ representations of scientific concepts in free-response items, particularly those drawn in a pencil-and-paper format.

在这里，我们描述了一个自动评估系统的开发和验证，该系统检查学生在自由回答项目中的手绘视觉表示。这些数据是从韩国二年级到十一年级的 1,028 名学生收集的，使用气体问卷中关于粒子测试的两个项目（Novick 和 Nussbaum，1981 年）。学生的自由反应，包括手绘和写作，被编码为两个维度——结构（颗粒/连续/其他）和分布（扩展/集中/其他）。训练机器学习 (ML) 模型以评估对物质微粒性质的反应。为了对手绘图进行分类，对预训练的 Inception-v3 模型和支持向量机进行了训练，并对其性能进行了评估。评估模型产生了很高的人机一致性 (MHA)（kappa = 0.732–0.926，准确性 = 0.820–0.942，精度 = 0.817–0.941，召回率 = 0.820–0.942，F1 = 0.818–0.941，曲线下面积 [AUC ] = 0.906–0.990）。学生的书面回答被标记化，并准备了一本科学语义分数词典。绘画和写作整体评估的最终模型产生了高 MHA（kappa = 0.800-0.881，准确度 = 0.859-0.956，精确度 = 0.865-0.957，召回率 = 0.859-0.956，F1 = 0.859-0.956，AUC = 0.944 –0.995)，这因模型的最终分类器而异。根据学校水平，评估模型的性能存在一些差异。