Skip to main content
SpringerLink
Log in

Detector-driven classroom interviewing: focusing qualitative researcher time by selecting cases in situ

  • Research Article
  • Published:
Educational technology research and development Aims and scope Submit manuscript

Abstract

In this paper, we propose a new method for selecting cases for in situ, immediate interview research: detector-driven classroom interviewing (DDCI). Published work in educational data mining and learning analytics has yielded highly scalable measures that can detect key aspects of student interaction with computer-based learning in close to real-time. These measures detect a variety of constructs and make it possible to increase the precision and time-efficiency of this form of research. We review four examples that show how the method can be used to study why students become frustrated and how they respond, how anxiety influences how students respond to frustration, how metacognition interacts with affect, and how to improve the design of an adaptive learning system. Lastly, we compare DDCI to other mixed-methods approaches and outline opportunities for detector-driven classroom interviewing in research and practice, including research opportunities, design improvement opportunities, and pedagogical opportunities for teachers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available on request from the corresponding author, Ryan S. Baker. The data are not publicly available due to IRB restrictions for the privacy of research participants.

References

  • Aagaard, J. (2015). Drawn to distraction: A qualitative study of off-task use of educational technology. Computers & Education, 87, 90–97.

    Article  Google Scholar 

  • Aldiabat, K. M., & Le Navenec, C. L. (2018). Data saturation: The mysterious step in grounded theory methodology. The Qualitative Report, 23(1), 245–261.

    Google Scholar 

  • Andres, J. M. A. L., Hutt, S., Ocumpaugh, J., Baker, R. S., Nasiar, N., & Porter, C. (2022). How anxiety affects affect: A quantitative ethnographic investigation using affect detectors and data-targeted interviews. In Advances in quantitative ethnography: Third international conference, ICQE 2021, Virtual Event, November 6–11, 2021, Proceedings 3 (pp. 268–283). Springer.

  • Andres, J. M. A. L., Ocumpaugh, J., Baker, R., Slater, S., Paquette, S., Jiang, Y., Bosch, N., Munshi, A., Moore, A., Biswas, G. (2019). Affect sequences and learning in Betty’s Brain. In Proceedings of the 9th international learning analytics and knowledge conference (pp. 383–390).

  • Ashcraft, M. H. (2002). Math anxiety: Personal, educational, and cognitive consequences. Current Directions in Psychological Science, 11(5), 181–185. https://doi.org/10.1111/1467-8721.00196

    Article  Google Scholar 

  • Azevedo, R., & Gašević, D. (2019). Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: Issues and challenges. Computers in Human Behavior, 96, 207–210. https://doi.org/10.1016/j.chb.2019.03.025

    Article  Google Scholar 

  • Azevedo, R., Moos, D. C., Johnson, A. M., & Chauncey, A. D. (2010). Measuring cognitive and metacognitive regulatory processes during hypermedia learning: Issues and challenges. Educational Psychologist, 45(4), 210–223.

    Article  Google Scholar 

  • Baird, B., Smallwood, J., Mrazek, M. D., Kam, J. W., Franklin, M. S., & Schooler, J. W. (2012). Inspired by distraction: Mind wandering facilitates creative incubation. Psychological Science, 23(10), 1117–1122. https://doi.org/10.1177/0956797612446024

    Article  Google Scholar 

  • Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-task behavior in the cognitive tutor classroom: When students “game the system”. In Proceedings of ACM CHI 2004: Computer-human interaction (pp. 383–390).

  • Baker, R. S., Nasiar, N., Ocumpaugh, J. L., Hutt, S., Andres, J. M. A. L., Slater, S., Schofield, M., Moore, A., Paquette, L., Munshi, A., & Biswas, G. (2021). Affect-targeted interviews for understanding student frustration. In Proceedings of the international conference on artificial intelligence and education.

  • Baker, R. S., Ocumpaugh, J. L., & Andres, J. M. A. L. (2020). BROMP quantitative field observations: A review. In R. Feldman (Ed.), Learning science: Theory, research, and practice (pp. 127–156). McGraw-Hill.

    Google Scholar 

  • Baker, R. S. J., & Rossi, L. M. (2013). Assessing the disengaged behavior of learners. In R. Sottilare, A. Graesser, X. Hu, & H. Holden (Eds.), Design recommendations for intelligent tutoring systems—Volume 1—Learner modeling (Vol. 1, pp. 155–166). U.S. Army Research Lab.

    Google Scholar 

  • Baker, R., & Siemens, G. (2014). Educational data mining and learning analytics. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (2nd ed., pp. 253–274). Cambridge University Press.

    Chapter  Google Scholar 

  • Barrett, L. F. (2017). The theory of constructed emotion: An active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(1), 1–23.

    Google Scholar 

  • Barriball, K. L., & While, A. (1994). Collecting data using a semi-structured interview: A discussion paper. Journal of Advanced Nursing-Institutional Subscription, 19(2), 328–335.

    Article  Google Scholar 

  • Biswas, G., Baker, R. S., & Paquette, L. (2017). Data mining methods for assessing self-regulated learning. In Handbook of self-regulation of learning and performance (pp. 388–403). Routledge.

  • Bosch, N., Huang, E., Angrave, L., & Perry, M. (2019). Modeling improvement for underrepresented minorities in online STEM education. In Proceedings of the 27th ACM conference on user modeling, adaptation and personalization (pp. 327–335).

  • Bosch, N., Zhang, Y., Paquette, L., Baker, R. S., Ocumpaugh, J., & Biswas, G. (2021). Students’ verbalized metacognition during computerized learning. In Proceedings of the 2021 CHI conference on human factors in computing systems (CHI ’21) (pp. 680:1–680:12). https://doi.org/10.1145/3411764.3445809

  • Botelho, A. F., Baker, R., Ocumpaugh, J., & Heffernan, N. (2018). Studying affect dynamics and chronometry using sensor-free detectors. In Proceedings of the 11th international conference on educational data mining (pp. 157–166).

  • Botelho, A. F., Varatharaj, A., Inwegen, E. G. V., & Heffernan, N. T. (2019). Refusing to try: Characterizing early stopout on student assignments. In Proceedings of the 9th international conference on learning analytics & knowledge (pp. 391–400). https://doi.org/10.1145/3303772.3303806

  • Briggs, C. L. (1986). Learning how to ask: A sociolinguistic appraisal of the role of the interview in social science research. Cambridge University Press.

    Book  Google Scholar 

  • Browning, M., Behrens, T. E., Jocham, G., O’Reilly, J. X., & Bishop, S. J. (2015). Anxious individuals have difficulty learning the causal statistics of aversive environments. Nature Neuroscience, 18(4), 590. https://doi.org/10.1038/nn.3961

    Article  Google Scholar 

  • Clarke, V., & Braun, V. (2017). Thematic analysis. The Journal of Positive Psychology, 12(3), 297–298. https://doi.org/10.1080/17439760.2016.1262613

    Article  Google Scholar 

  • Clements, M. (1982). Careless errors made by sixth-grade children on written mathematical tasks. Journal for Research in Mathematics Education, 13(2), 136–144. https://doi.org/10.1016/j.cedpsych.2019.01.007

    Article  Google Scholar 

  • Costanza-Chock, S. (2018). Design justice, AI, and escape from the matrix of domination. Journal of Design and Science. https://doi.org/10.21428/96c8d426

    Article  Google Scholar 

  • De Angeli, A., & Brahnam, S. (2008). I hate you! Disinhibition with virtual partners. Interacting with Computers, 20(3), 302–310. https://doi.org/10.1016/j.intcom.2008.02.004

    Article  Google Scholar 

  • DeFalco, J. A., Rowe, J. P., Paquette, L., Georgoulas-Sherry, V., Brawner, K., Mott, B. W., Baker, R. S., & Lester, J. C. (2018). Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence and Education, 28(2), 152–193. https://doi.org/10.1007/s40593-017-0152-1

    Article  Google Scholar 

  • Dillon, J., Bosch, N., Chetlur, M., Wanigasekara, N., Ambrose, G. A., Sengupta, B., & D'Mello, S. K. (2016). Student emotion, co-occurrence, and dropout in a MOOC context. In Proceedings of the international conference on educational data mining.

  • D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082. https://doi.org/10.1037/a0032674

    Article  Google Scholar 

  • D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., & Graesser, A. (2010). A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In International conference on intelligent tutoring systems (pp. 245–254). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-13388-6_29

  • Dumas, J. S., Dumas, J. S., & Redish, J. (1999). A practical guide to usability testing. Intellect Books.

    Google Scholar 

  • Eder, D., & Fingerson, L. (2002). Interviewing children and adolescents. In J. K. Gubrium & J. A. Holstein (Eds.), Handbook of interview research. Sage.

    Google Scholar 

  • El-Nasr, M. S., Durga, S., Shiyko, M., & Sceppa, C. (2015). Data-driven retrospective interviewing (DDRI): A proposed methodology for formative evaluation of pervasive games. Entertainment Computing, 11, 1–19. https://doi.org/10.1016/j.entcom.2015.07.002

    Article  Google Scholar 

  • Emmel, N. (2013). Purposeful sampling. In Sampling and choosing cases in qualitative research: A realist approach (pp. 33–45). https://doi.org/10.4135/9781473913882.n3

  • Endler, N. S., & Kocovski, N. L. (2001). State and trait anxiety revisited. Journal of Anxiety Disorders, 15(3), 231–245. https://doi.org/10.1016/S0887-6185(01)00060-3

    Article  Google Scholar 

  • Erbas, A. K., & Okur, S. (2012). Researching students’ strategies, episodes, and metacognitions in mathematical problem solving. Quality & Quantity, 46, 89–102.

    Article  Google Scholar 

  • Goffman, E. (1959). The presentation of self in everyday life. Doubleday.

    Google Scholar 

  • Graesser, A. C. (2011). Learning, thinking, and emoting with discourse technologies. American Psychologist, 66(8), 746. https://doi.org/10.1037/a0024974

    Article  Google Scholar 

  • Hershkovitz, A., Baker, R. S., Gobert, J., Wixon, M., & Pedro, M. S. (2013). Discovery with models: A case study on carelessness in computer-based science inquiry. American Behavioral Scientist, 57(10), 1480–1499. https://doi.org/10.1177/0002764213479365

    Article  Google Scholar 

  • Hoeber, O., Hoeber, L., El Meseery, M., Odoh, K., & Gopi, R. (2016). Visual Twitter analytics (Vista): Temporally changing sentiment and the discovery of emergent themes within sport event tweets. Online Information Review, 40(1), 25–41. https://doi.org/10.1108/OIR-02-2015-0067

    Article  Google Scholar 

  • Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudik, M., & Wallach, H. (2019, May). Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1–16). https://doi.org/10.1145/3290605.3300830

  • Hunting, R. P. (1997). Clinical interview methods in mathematics education research and practice. The Journal of Mathematical Behavior, 16(2), 145–165.

    Article  Google Scholar 

  • Hutt, S., Baker, R. S., Ocumpaugh, J., Munshi, A., Andres, J. M. A. L., Karumbaiah, S., Slater S., Biswas G., Paquette L., Bosch, N. & van Velsen, M. (2022). Quick red fox: An app supporting a new paradigm in qualitative research on AIED for STEM. In Artificial intelligence in STEM education: The paradigmatic shifts in research, education, and technology (pp. 319–332).

  • Hutt, S., Grafsgaard, J. F., & D'Mello, S. K. (2019). Time to scale: Generalizable affect detection for tens of thousands of students across an entire school year. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1–14).

  • Jerolmack, C., & Khan, S. (2014). Talk is cheap: Ethnography and the attitudinal fallacy. Sociological Methods & Research, 43(2), 178–209.

    Article  Google Scholar 

  • Jiang, Y., Paquette, L., Baker, R. S., & Clarke-Midura, J. (2015) Comparing novice and experienced students in virtual performance assessments. In Proceedings of the 8th international conference on educational data mining (pp. 136–143).

  • Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. In European conference on technology enhanced learning (pp. 82–96). Springer, Cham. https://doi.org/10.1007/978-3-319-66610-5_7

  • Johnston-Wilder, S., Brindley, J., & Dent, P. (2014). A survey of mathematics anxiety and mathematical resilience among existing apprentices. The Gatsby Foundation.

    Google Scholar 

  • Kallio, H., Pietilä, A. M., Johnson, M., & Kangasniemi, M. (2016). Systematic methodological review: Developing a framework for a qualitative semi-structured interview guide. Journal of Advanced Nursing, 72(12), 2954–2965.

    Article  Google Scholar 

  • Knapp, N. F. (1997). Interviewing Joshua: On the importance of leaving room for serendipity. Qualitative Inquiry, 3(3), 326–342. https://doi.org/10.1177/107780049700300305

    Article  Google Scholar 

  • Kvale, S., & Brinkmann, S. (2009). Interviews: Learning the craft of qualitative research interviewing. Sage.

    Google Scholar 

  • Labov, W. (1972). Some principles of linguistic methodology. Language in Society, 1(1), 97–120.

    Article  Google Scholar 

  • Leary, H., Lee, V. R., & Recker, M. (2021). It’s more than just technology adoption: Understanding variations in teachers’ use of an online planning tool. TechTrends, 65(3), 269–277. https://doi.org/10.1007/s11528-020-00576-3

    Article  Google Scholar 

  • Leech, N. L., & Onwuegbuzie, A. J. (2007). An array of qualitative data analysis tools: A call for data analysis triangulation. School Psychology Quarterly, 22(4), 557.

    Article  Google Scholar 

  • Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: The Betty’s Brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208.

    Google Scholar 

  • Lindquist, K. A., Barrett, L. F., Bliss-Moreau, E., & Russell, J. A. (2006). Language and the perception of emotion. Emotion, 6(1), 125.

    Article  Google Scholar 

  • Luo, G. (2015). MLBCD: A machine learning tool for big clinical data. Health Information Science and Systems, 3(1), 1–19. https://doi.org/10.1186/s13755-015-0011-0

    Article  Google Scholar 

  • Miller, W. L., Baker, R., Labrum, M., Petsche, K., Liu, Y.-H., & Wagner, A. (2015) Automated detection of proactive remediation by teachers in Reasoning Mind classrooms. In Proceedings of the 5th international learning analytics and knowledge conference (pp. 290–294). https://doi.org/10.1145/2723576.2723607

  • Munshi, A., Biswas, G., Baker, R., Ocumpaugh, J., Hutt, S., & Paquette, L. (2023). Analysing adaptive scaffolds that help students develop self-regulated learning behaviours. Journal of Computer Assisted Learning, 39(2), 351–368.

    Article  Google Scholar 

  • Munshi, A., Rajendran, R., Ocumpaugh, J., Biswas, G., Baker, R. S., & Paquette, L. (2018, July). Modeling learners’ cognitive and affective states to scaffold SRL in open-ended learning environments. In Proceedings of the 26th conference on user modeling, adaptation and personalization (pp. 131–138).

  • Nathan, M. J., & Petrosino, A. (2003). Expert blind spot among preservice teachers. American Educational Research Journal, 40(4), 905–928. https://doi.org/10.3102/00028312040004905

    Article  Google Scholar 

  • Nawaz, S., Kennedy, G., Bailey, J., Mead, C., & Horodyskyj, L. (2018). Struggle town? Developing profiles of student confusion in simulation-based learning environments. In 35th International conference on innovation, practice and research in the use of educational technologies in tertiary education, ASCILITE (pp. 224–233).

  • Nelson, L. K. (2020). Computational grounded theory: A methodological framework. Sociological Methods & Research, 49(1), 3–42. https://doi.org/10.1177/0049124117729703

    Article  Google Scholar 

  • Ocumpaugh, J., Hutt, S., Andres, J. M. A. L., Baker, R. S., Biswas, G., Bosch, N., Paquette, L., & Munshi, A. (2021). Using qualitative data from targeted interviews to inform rapid AIED development. In Proceedings of the 29th international conference on computers in education (pp. 69–74).

  • Ogan, A., Finkelstein, S., Mayfield, E., D'Adamo, C., Matsuda, N., & Cassell, J. (2012). “Oh dear Stacy!” Social interaction, elaboration, and learning with teachable agents. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 39–48). https://doi.org/10.1145/2207676.2207684

  • Paquette, L., Grant, T., Zhang, Y., Biswas, G., & Baker, R. (2021). Using epistemic networks to analyze self-regulated learning in an open-ended problem-solving environment. In International conference on quantitative ethnography (pp. 185–201). Springer, Cham.

  • Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Sage.

    Google Scholar 

  • Ravitch, S. M., & Carl, N. M. (2019). Qualitative research: Bridging the conceptual, theoretical, and methodological. Sage Publications.

    Google Scholar 

  • Rittle-Johnson, B., & Koedinger, K. R. (2005). Designing knowledge scaffolds to support mathematical problem solving. Cognition and Instruction, 23(3), 313–349. https://doi.org/10.1207/s1532690xci2303_1

    Article  Google Scholar 

  • Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21(2), 267–280. https://doi.org/10.1016/j.learninstruc.2010.07.004

    Article  Google Scholar 

  • Saldana, J. (2011). Fundamentals of qualitative research. OUP USA.

    Google Scholar 

  • Schofield, J. W. (1995). Computers and classroom culture. Cambridge University Press.

    Book  Google Scholar 

  • Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts beyond words: When language overshadows insight. Journal of Experimental Psychology: General, 122(2), 166. https://doi.org/10.1037/0096-3445.122.2.166

    Article  Google Scholar 

  • Seidman, I. (2006). Interviewing as qualitative research: A guide for researchers in education and the social sciences. Teachers College Press.

    Google Scholar 

  • Spradley, J. P. (2016). The ethnographic interview. Waveland Press.

    Google Scholar 

  • Stawarczyk, D., Majerus, S., Maj, M., Van der Linden, M., & D’Argembeau, A. (2011). Mind-wandering: Phenomenology and function as assessed with a novel experience sampling method. Acta Psychologica, 136(3), 370–381. https://doi.org/10.1016/j.actpsy.2011.01.002

    Article  Google Scholar 

  • Strauss, A., & Corbin, J. (1990). Basics of qualitative research. Sage.

    Google Scholar 

  • Tynan, R. (2005). The effects of threat sensitivity and face giving on dyadic psychological safety and upward communication 1. Journal of Applied Social Psychology, 35(2), 223–247. https://doi.org/10.1111/j.1559-1816.2005.tb02119.x

    Article  Google Scholar 

  • Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. (1994). The think aloud method: A practical approach to modelling cognitive. Academic Press.

    Google Scholar 

  • Verduyn, P., & Lavrijsen, S. (2015). Which emotions last longest and why: The role of event importance and rumination. Motivation and Emotion, 39(1), 119–127. https://doi.org/10.1007/s11031-014-9445-y

    Article  Google Scholar 

  • Vermeeren, A. P. O. S., Bekker, M. M., Kesteren, I. V., & Ridder, H. D. (2007). Experiences with structured interviewing of children during usability tests. In Proceedings of HCI 2007 The 21st British HCI Group annual conference University of Lancaster, UK 21 (pp. 1–9). https://doi.org/10.14236/ewic/HCI2007.14

  • Ward, M. D. (1981). The observer effect in classroom visitation. Unpublished doctoral dissertation, Brigham Young University.

  • Wengraf, T. (2001). Qualitative research interviewing: Biographic narrative and semi-structured methods. Sage.

    Book  Google Scholar 

  • Xia, M., Asano, Y., Williams, J. J., Qu, H., & Ma, X. (2020). Using information visualization to promote students’ reflection on “gaming the system” in online learning. In Proceedings of the seventh ACM conference on Learning@ Scale (pp. 37–49). https://doi.org/10.1145/3386527.3405924

Download references

Funding

Funding was provided by National Science Foundation (Grant No. DRL-1561567).

Author information

Authors and Affiliations

  1. Graduate School of Education, University of Pennsylvania, Philadelphia, USA

    Ryan S. Baker, Stephen Hutt, Jaclyn Ocumpaugh, J. M. Alexandra Andres & Nidhi Nasiar

  2. School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, USA

    Nigel Bosch & Luc Paquette

  3. Computer Science, School of Engineering, Vanderbilt University, Nashville, USA

    Gautam Biswas & Anabil Munshi

Authors
  1. Ryan S. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen Hutt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nigel Bosch
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jaclyn Ocumpaugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gautam Biswas
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Luc Paquette
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. J. M. Alexandra Andres
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Nidhi Nasiar
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Anabil Munshi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryan S. Baker.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Baker, R.S., Hutt, S., Bosch, N. et al. Detector-driven classroom interviewing: focusing qualitative researcher time by selecting cases in situ. Education Tech Research Dev (2023). https://doi.org/10.1007/s11423-023-10324-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11423-023-10324-y

Keywords

Search

Navigation