Abstract

This study is an exploratory analysis of examinee behavior in a large-scale language proficiency test. Despite a number-right scoring system with no penalty for guessing, we found that 16% of examinees omitted at least one answer and that women were more likely than men to omit answers. Item-response theory analyses treating the omitted responses as missing rather than wrong showed that examinees had underperformed by skipping the answers, with a greater underperformance among more able participants. An analysis of omitted answer patterns showed that reading passage items were most likely to be omitted, and that native language-translation items were least likely to be omitted. We hypothesized that since reading passage items were most tempting to skip, then among examinees who did answer every question there might be a tendency to guess at these items. Using cluster analyses, we found that underperformance on the reading items was more likely than underperformance on the non-reading passage items. In large-scale operational tests, examinees must know the optimal strategy for taking the test. Test developers must also understand how examinee behavior might impact the validity of score interpretations.

摘要

本研究是对大规模语言能力测试中考生行为的探索性分析。尽管采用了数字正确的评分系统，对猜对没有惩罚，但我们发现 16% 的考生至少遗漏了一个答案，而且女性比男性更容易遗漏答案。项目反应理论分析将遗漏的反应视为缺失而不是错误，表明考生因跳过答案而表现不佳，能力更强的参与者表现更差。对遗漏答案模式的分析表明，阅读文章项目最有可能被遗漏，而母语翻译项目最不可能被遗漏。我们假设，由于阅读段落项目最容易跳过，那么在确实回答了所有问题的考生中，可能会有猜测这些项目的倾向。使用聚类分析，我们发现在阅读项目上表现不佳的可能性大于在非阅读文章项目上表现不佳的可能性。在大规模的操作性考试中，考生必须知道最佳的应试策略。测试开发人员还必须了解考生的行为如何影响分数解释的有效性。