In this study, we examine “what went wrong” in our professional development program for encouraging cognitively demanding instruction, focusing on the difficulties we encountered in using an observational tool for evaluating this type of instruction and reaching inter-rater reliability. We do so through the lens of a discursive theory of teaching and learning. Data consisted of 10 coders’ coding sheets while learning to apply the Coding Rubric for Video Observations tool on a set of recorded mathematics lessons. We show that some discrepancies between novice coders and experts were found in narratives about valued actions relating to social aspects of teaching, such as teacher explanations and students’ struggle and discussion. These were relatively easy to detect from the written evaluations in the coding sheets. More problematic to pinpoint were places where discrepancies were found between novice and experts’ stories about the mathematical objects discussed. These required re-observing the recorded lessons. The analysis revealed that novice coders referred to objects about which the observed teacher mainly explained calculations, whereas the experts searched for (often, signaling the absence of) mathematical objects that invited exploration. We discuss these findings based on a theory of different Pedagogical Discourses that form the background for expert and novice coders’ interpretations of the coding manual. We conclude with practical implications for the process of achieving inter-rater reliability on observation tools for cognitively demanding instruction.

在这项研究中，我们检查了我们的专业发展计划中“出了什么问题”以鼓励认知要求高的教学，重点关注我们在使用观察工具评估此类教学和达到评估者间可靠性时遇到的困难。我们通过教学和学习的话语理论来做到这一点。数据包括 10 个编码员的编码表，同时学习在一组记录的数学课程中应用视频观察工具的编码规则。我们表明，在与教学的社会方面相关的有价值的行动的叙述中，例如教师解释和学生的斗争和讨论，新手编码员和专家之间存在一些差异。这些相对容易从编码表中的书面评估中检测出来。更难以查明的是新手和专家关于所讨论数学对象的故事之间存在差异的地方。这些需要重新观察录制的课程。分析表明，新手编码员提到的对象是被观察的老师主要解释计算的对象，而专家则搜索（通常表示不存在）邀请探索的数学对象。我们根据不同教学话语的理论讨论这些发现，这些话语形成了专家和新手编码员对编码手册的解释的背景。我们总结了在认知要求高的教学的观察工具上实现评估者间可靠性的过程的实际意义。分析表明，新手编码员提到的对象是被观察的老师主要解释计算的对象，而专家则搜索（通常表示不存在）邀请探索的数学对象。我们根据不同教学话语的理论讨论这些发现，这些话语形成了专家和新手编码员对编码手册的解释的背景。我们总结了在认知要求高的教学的观察工具上实现评估者间可靠性的过程的实际意义。分析表明，新手编码员提到的对象是被观察的老师主要解释计算的对象，而专家则搜索（通常表示不存在）邀请探索的数学对象。我们根据不同教学话语的理论讨论这些发现，这些话语形成了专家和新手编码员对编码手册的解释的背景。我们总结了在认知要求高的教学的观察工具上实现评估者间可靠性的过程的实际意义。我们根据不同教学话语的理论讨论这些发现，这些话语形成了专家和新手编码员对编码手册的解释的背景。我们总结了在认知要求高的教学的观察工具上实现评估者间可靠性的过程的实际意义。我们根据不同教学话语的理论讨论这些发现，这些话语形成了专家和新手编码员对编码手册的解释的背景。我们总结了在认知要求高的教学的观察工具上实现评估者间可靠性的过程的实际意义。