Skip to main content
SpringerLink
Log in

Improving collaborative problem-solving skills via automated feedback and scaffolding: a quasi-experimental study with CPSCoach 2.0

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

We present CPSCoach 2.0, an automated system that provides feedback, instructional scaffolding, and practice to help individuals improve three collaborative problem-solving (CPS) skills drawn from a theoretical CPS framework: construction of shared knowledge, negotiation/coordination, and maintaining team function. CPSCoach 2.0 was developed and tested in the context of computer-mediated collaboration (video conferencing) with an educational game. It automatically analyzes users’ speech during a round of collaborative gameplay to provide personalized feedback and to select a target CPS skill for improvement. After multiple cycles of iterative testing and refinement, we tested CPSCoach 2.0 in a user study where 21 dyads (n = 42) completed four rounds of feedback and scaffolding embedded within five rounds of game-play in a single session. Using a quasi-experimental matching procedure, we found that the use of CPSCoach 2.0 was associated with improvement in CPS skill development compared to matched controls. Further, users found the automated feedback to be moderately accurate and had positive perceptions of the system, and these impressions were stronger for those who received higher scores overall. Results demonstrate the use of automated feedback and instructional scaffolds to support the development of CPS skills.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. The only publication on CPSCoach includes a short conference paper (Stewart et al. 2023). The present paper reports results on a new study with a substantially revised system.

References

  • Alterman, R., Harsch, K.: A more reflective form of joint problem solving. Int. J. Comput. Support Collab. Learn 12, 9–33 (2017). https://doi.org/10.1007/s11412-017-9250-1

    Article  Google Scholar 

  • Amon, M.J., Vrzakova, H., D’Mello, S.K.: Beyond Dyadic coordination: multimodal behavioral irregularity in triads predicts facets of collaborative problem solving. Cogn. Sci.. Sci. (2019). https://doi.org/10.1111/cogs.12787

    Article  Google Scholar 

  • Andrews-Todd, J., Forsyth, C.M.: Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Comput. Human Behav. 104, 105759 (2020)

    Article  Google Scholar 

  • Aran, O., Gatica-Perez. D (2010) Fusing Audio-Visual Nonverbal Cues to Detect Dominant People in Group Conversations. In: 2010 20th International Conference on Pattern Recognition. pp 3687–3690

  • Azevedo, R., Bernard, R.M.: A meta-analysis of the effects of feedback in computer-based instruction. J. Educ. Comput. Res. 13, 111–127 (1995)

    Article  Google Scholar 

  • Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Royal Stat. Soc. Series B-Methodol. 57, 289–300 (1995)

    MathSciNet  Google Scholar 

  • Beyan, C., Carissimi, N., Capozzi, F., et al (2016) Detecting Emergent Leader in a Meeting Environment Using Nonverbal Visual Features Only. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, New York, NY, USA, pp 317–324

  • Bosch, N., D’Mello, S., Baker, R., et al (2015) Automatic detection of learning-centered affective states in the wild. In: Proceedings of the 20th international conference on intelligent user interfaces. pp 379–388

  • Bransford, JD., Brown, AL., Cocking RR (2000) How people learn

  • Breideband, T., Martinez, G., Sukumar, PT., et al (2022) Collaborating from Home during COVID-19: Examining Individual Sleep Patterns and Sleep Alignment

  • Brynjolfsson, E., Horton, JJ., Ozimek, A., et al (2020) COVID-19 and remote work: An early look at US data

  • Calacci, D., Lederman, O., Shrier, D., Pentland, A., “Sandy” (2016) Breakout: An Open Measurement and Intervention Tool for Distributed Peer Learning Groups. CoRR abs/1607.0:

  • Chi, M., Wylie, R.: The ICAP framework: linking cognitive engagement to active learning outcomes. Educ. Psychol. 49, 219–243 (2014)

    Article  Google Scholar 

  • Chi, M., Siler, S., Jeong, H., et al.: Learning from human tutoring. Cogn. Sci.. Sci. 25, 471–533 (2001)

    Article  Google Scholar 

  • Chopade, P., Edwards, D., Khan, SM., et al (2019) CPSX: Using AI-machine learning for mapping human-human interaction and measurement of cps teamwork skills. In: 2019 IEEE International Symposium on Technologies for Homeland Security (HST). pp 1–6

  • Cook, T.D., Campbell, D.T., Shadish, W.: Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin, Boston, MA (2002)

    Google Scholar 

  • Cukurova, M., Luckin, R., Millán, E., Mavrikis, M.: The NISPI framework: analysing collaborative problem-solving from students’ physical interactions. Comput. Educ.. Educ. 116, 93–109 (2018). https://doi.org/10.1016/j.compedu.2017.08.007

    Article  Google Scholar 

  • Cukurova, M., Zhou, Q., Spikol, D., Landolfi, L, (2020) Modelling collaborative problem-solving competence with transparent learning analytics: is video data enough? In: Proceedings of the tenth international conference on learning analytics & knowledge. association for computing machinery, New York, NY, USA, pp 270–275

  • D’Mello, S.K., King, B., Graesser, A.: Towards spoken human-computer tutorial dialogues. Hum. Comput. Interact.comput. Interact 25, 289–323 (2010)

    Article  Google Scholar 

  • Davor, Č., Margaret Anne, DS., Davor, Č., Margaret Anne, DS (2005) Collaboration support for novice team programming. Proceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work pp 136–139. https://doi.org/10.1145/1099203.1099229

  • Deci, EL., Ryan, RM (1982) Intrinsic motivation inventory measurement instrument

  • de Kok, I, Heylen, D. (2009) Multimodal end-of-turn prediction in multi-party meetings. In: Proceedings of the 2009 International Conference on Multimodal Interfaces. ACM, New York, NY, USA, pp 91–98

  • Dede, C.: Comparing frameworks for 21st century skills. In: Bellanca, J., Brandt, R. (eds.) 21st century skills: Rethinking how students learn, pp. 51–76. Solution Tree Press, Bloomington IN (2010)

    Google Scholar 

  • Devlin, J., Chang, M-W., Lee, K., Toutanova, K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv arXiv:1810.04805

  • Diehl, M., Stroebe, W.: Productivity loss in brainstorming groups: toward the solution of a riddle. J. Pers. Soc. Psychol. 53, 497–509 (1987)

    Article  Google Scholar 

  • Dielmann, A., Garau, G., Bourlard, H (2010) Floor holder detection and end of speaker turn prediction in meetings. In: Proceedings of the International Conference on Speech and Language Processing, Interspeech. ISCA

  • Dowell, N.M.M., Nixon, T.M., Graesser, A.C.: Group communication analysis: a computational linguistics approach for detecting sociocognitive roles in multiparty interactions. Behav. Res. Methods. Res. Methods 51, 1007–1041 (2019)

    Article  Google Scholar 

  • Dowell, N.M.M., Lin, Y., Godfrey, A., Brooks, C.: Exploring the relationship between emergent sociocognitive roles, collaborative problem-solving skills, and outcomes: a group communication analysis. J. Learn. Anal. 7, 38–57 (2020)

    Google Scholar 

  • Elliot, A., McGregor, H.: A 2 x 2 achievement goal framework. J. Pers. Soc. Psychol. 80, 501–519 (2001). https://doi.org/10.1037//0022-3514.80.3.501

    Article  CAS  PubMed  Google Scholar 

  • Eloy, L, E.B. Stewart, A., Jean Amon, M., et al (2019) Modeling team-level multimodal dynamics during multiparty collaboration. In: 2019 international conference on multimodal interaction. association for computing machinery, New York, NY, USA, pp 244–258

  • Erik, B., John, JH., Adam, O., et al (2020) COVID-19 and remote work: An early look at US data

  • Faucett, H.A., Lee, M.L., Carter, S.: I should listen more: real-time sensing and feedback of non-verbal communication in video telehealth. Proc. ACM Hum-Comput. Interact. 1(44), 1–44 (2017). https://doi.org/10.1145/3134679

    Article  Google Scholar 

  • Fiore, S.M., Graesser, A., Greiff, S.: Collaborative problem-solving education for the twenty-first-century workforce. Nat. Hum. Behav.behav. 2, 367–369 (2018)

    Article  Google Scholar 

  • Forbes-Riley, K., Litman, D.J.: Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Commun.commun. 53, 1115–1136 (2011). https://doi.org/10.1016/j.specom.2011.02.006

    Article  Google Scholar 

  • Friedrich, H., Esther, C., Juergen, B., et al.: A Framework for Teachable Collaborative Problem Solving Skills. Springer, Netherlands (2015)

    Google Scholar 

  • Fusaroli, R., Rkaczaszek-Leonardi, J., Tylén, K.: Dialog as interpersonal synergy. New Ideas Psychol. 32, 147–157 (2014)

    Article  Google Scholar 

  • Gigone, D., Hastie, R.: The common knowledge effect: Information sharing and group judgment. J. Pers. Soc. Psychol. 65, 959–974 (1993)

    Article  Google Scholar 

  • Graesser, A.C., Fiore, S.M., Greiff, S., et al.: Advancing the science of collaborative problem solving. Psychol. Sci. Public Interest 19, 59–92 (2018). https://doi.org/10.1177/1529100618808244

    Article  PubMed  Google Scholar 

  • Griffin, P., Care, E., McGaw, B.: The Changing Role of Education and Schools. In: Griffin, P., McGaw, B., Care, E. (eds.) Assessment and Teaching of 21st Century Skills, pp. 1–15. Springer, Netherlands, Dordrecht (2012a)

    Chapter  Google Scholar 

  • Griffin, P., McGaw, B., Care, E.: Assessment and teaching of 21st century skills. Springer, New York (2012b)

    Book  Google Scholar 

  • Gutwin, C., Bateman, S., Arora, G., Coveney, A. (2017) Looking away and catching up: dealing with brief attentional disconnection in synchronous groupware. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing. ACM, New York, NY, USA, pp 2221–2235

  • Hesse, F., Care, E., Buder, J., et al.: A Framework for Teachable Collaborative Problem Solving Skills. In: Griffin, P., Care, E. (eds.) Assessment and Teaching of 21st Century Skills: Methods and Approach, pp. 37–56. Springer, Netherlands, Dordrecht (2015)

    Chapter  Google Scholar 

  • Hill, GW. (1982) Group Versus Individual Performance : Are TV + 1 Heads Better Than One ? 91:517–539

  • Hong, S., Suh, M., Kim, T.S., et al.: Design for collaborative information-seeking: understanding user challenges and deploying collaborative dynamic queries. Proc. ACM Hum-Comput. Interact (2019). https://doi.org/10.1145/3359208

    Article  Google Scholar 

  • Janis, I.L.: Groupthink: Psychological studies of policy decisions and fiascoes. Houghton Mifflin, Boston (1982)

    Google Scholar 

  • Jiangang, H., Lei, C., Michael, F., et al.: CPS-Rater: automated sequential annotation for conversations in collaborative problem-solving activities. ETS Res. Report Series 2017, 1–9 (2017). https://doi.org/10.1002/ets2.12184

    Article  Google Scholar 

  • Jokinen, K., Furukawa, H., Nishida, M., Yamamoto, S.: Gaze and turn-taking behavior in casual conversational interactions. ACM Trans. Interact Intell. Syst. 3(12), 1–12 (2013). https://doi.org/10.1145/2499474.2499481

    Article  Google Scholar 

  • Karau, S.J., Williams, K.D.: Social loafing: a meta-analytic review and theoretical integration. J. Pers. Soc. Psychol. 65, 681–706 (1993)

    Article  Google Scholar 

  • Kendon, A.: Some functions of gaze-direction in social interaction. Acta Psychol. (amst) Psychol. (Amst) 26, 22–63 (1967). https://doi.org/10.1016/0001-6918(67)90005-4

    Article  CAS  PubMed  Google Scholar 

  • Kerr, N.L.: Motivation losses in small groups: a social dilemma analysis. J. Pers. Soc. Psychol. 45, 819–828 (1983)

    Article  Google Scholar 

  • Kerr, N.L., Bruun, S.E.: Dispensability of member effort and group motivation losses: free-rider effects. J. Pers. Soc. Psychol. 44, 78–94 (1983)

    Article  Google Scholar 

  • Kerr, N.L., Tindale, R.S.: Group performance and decision making. Annu. Rev. Psychol.. Rev. Psychol. 55, 623–655 (2004)

    Article  Google Scholar 

  • Koedinger, K.R., McLaughlin, E.A., Heffernan, N.T.: A quasi-experimental evaluation of an on-line formative assessment and tutoring system. J. Educ. Comput. Res. 43, 489–510 (2010)

    Article  Google Scholar 

  • Kori, K., Pedaste, M., Leijen, Ä., Mäeots, M.: Supporting reflection in technology-enhanced learning. Educ. Res. Rev. 11, 45–55 (2014)

    Article  Google Scholar 

  • Krafft, PM., Baker, CL., Tenenbaum, JB. others (2016) Modeling human ad hoc coordination. In: Thirtieth AAAI Conference on Artificial Intelligence

  • Kütt, G.H., Tanprasert, T., Rodolitz, J., et al.: Effects of shared gaze on audio- versus text-based remote collaborations. Proc. ACM Hum-Comput. Interact (2020). https://doi.org/10.1145/3415207

    Article  Google Scholar 

  • Laughlin, P.R., Ellis, A.L.: Demonstrability and social combination processes on mathematical intellective tasks. J. Exp. Soc. Psychol. 22, 177–189 (1986)

    Article  Google Scholar 

  • Laughlin, P.R., Kerr, N.L., Davis, J.H., et al.: Group size, member ability, and social decision schemes on an intellective task. J. Pers. Soc. Psychol. 31, 522 (1975)

    Article  Google Scholar 

  • Laughlin, P.R., Hatch, E.C., Silver, J.S., Boh, L.: Groups perform better than the best individuals on letters-to-numbers problems: effects of group size. J. Pers. Soc. Psychol. 90, 644 (2006)

    Article  PubMed  Google Scholar 

  • Lewis, J.R.: The system usability scale: past, present, and future. Int. J. Hum. Comput. Interact.comput. Interact. 34, 577–590 (2018)

    Article  Google Scholar 

  • Li, H., Rosenthal, R., Rubin, D.B.: Reliability of measurement in psychology: from Spearman-Brown to maximal reliability. Psychol. Methods 1, 98–107 (1996). https://doi.org/10.1037/1082-989X.1.1.98

    Article  Google Scholar 

  • McGregor, M., Tang, JC. (2017) More to meetings: challenges in using speech-based technology to support meetings. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing. ACM, New York, NY, USA, pp 2208–2220

  • Mercier, E.M., Higgins, S.E., da Costa, L.: Different leaders: emergent organizational and intellectual leadership in children’s collaborative learning groups. Int. J. Comput. Support Collab. Learn (2014). https://doi.org/10.1007/s11412-014-9201-z

    Article  Google Scholar 

  • Michael, F., Su-Youn, Y., Jiangang, H., et al (2016) Automated classification of collaborative problem solving interactions in simulated science tasks. Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications pp 31–41

  • Miguel, J., Andres, L., Mercedes, M., et al (2014) An exploratory analysis of confusion among students using Newton’s Playground. Proceedings of the 22nd International Conference on Computers in Education

  • Müller, P., Huang, MX., Bulling, A. (2018) Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour. In: 23rd International Conference on Intelligent User Interfaces. pp 153–164

  • Murray, G., Oertel, C. (2018) Predicting group performance in task-based interaction. In: Proceedings of the 20th ACM international conference on multimodal interaction. ACM, New York, NY, USA, pp 14–20

  • NASEM: How people learn II: Learners, contexts, and cultures. National Academies Press, Washington, DC (2018)

    Google Scholar 

  • Nelson, L.M.: Collaborative problem solving. Inst. Des. Theories and Models 2, 241–267 (1999)

    Google Scholar 

  • Nihei, F., Nakano, YI., Hayashi, Y., et al (2014) Predicting influential statements in group discussions using speech and head motion information. In: Proceedings of the 16th international conference on multimodal interaction. ACM, New York, NY, USA, pp 136–143

  • Nijstad, B.A., Stroebe, W., Lodewijkx, H.F.M.: Production blocking and idea generation: does blocking interfere with cognitive processes? J. Exp. Soc. Psychol. 39, 531–548 (2003)

    Article  Google Scholar 

  • OECD (2015a) PISA 2015 Results in Focus. OECD Publishing

  • OECD (2015b) PISA 2015 Collaborative problem solving framework. organisation for economic co-operation and development (OECD)

  • Olsen, J.K., Sharma, K., Rummel, N., Aleven, V.: Temporal analysis of multimodal data to predict collaborative learning outcomes. British J. Educ. Technol. 51, 1527–1547 (2020). https://doi.org/10.1111/bjet.12982

    Article  Google Scholar 

  • Olsen, JK., Finkelstein, S. (2017) Through the (thin-slice) looking glass: An initial look at rapport and co-construction within peer collaboration. In: Proceedings of the 12th International Conference on Computer Supported Collaborative Work. International Society of the Learning Sciences, Philadelphia, PA, USA

  • Otsuka, K., Kasuga, K., Köhler, M. (2018) Estimating visual focus of attention in multiparty meetings using deep convolutional neural networks. In: Proceedings of the 20th ACM international conference on multimodal interaction. ACM, New York, NY, USA, pp 191–199

  • Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng.knowl. Data Eng. 22, 1345–1359 (2010)

    Article  Google Scholar 

  • Pugh, S.L., Rao, A., Stewart, A.E.B., D’Mello, S.K.: Do Speech-Based Collaboration Analytics Generalize Across Task Contexts? In: LAK22: 12th International Learning Analytics and Knowledge Conference, pp. 208–218. ACM, New York (2022)

    Chapter  Google Scholar 

  • Pugh, SL., Subburaj, SK., Rao, AR., et al (2021) Say What? Automatic Modeling of Collaborative Problem Solving Skills from Student Speech in the Wild. In: Proceedings of The 14th International Conference on Educational Data Mining (EDM21)

  • Rahimi, S., Shute, V.J., Fulwider, C., et al.: Timing of learning supports in educational games can impact students’ outcomes. Comput. Educ.. Educ. 190, 104600 (2022)

    Article  Google Scholar 

  • Roschelle, J., Teasley, S.D.: The Construction of Shared Knowledge in Collaborative Problem Solving. In: O’Malley, C. (ed.) Computer Supported Collaborative Learning, pp. 69–97. Springer, Berlin Heidelberg, Berlin, Heidelberg (1995)

    Chapter  Google Scholar 

  • Rosenthal, R., Rosnow, R.: Essentials of behavioral research: Methods and data analysis. McGraw-Hill, New York (1984)

    Google Scholar 

  • Samrose, S., McDuff, D., Sim, R., et al (2021) Meetingcoach: An intelligent dashboard for supporting effective & inclusive meetings. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. pp 1–13

  • Sanchez-Cortes, D., Aran, O., Mast, MS., Gatica-Perez, D. (2010) Identifying Emergent Leadership in Small Groups Using Nonverbal Communicative Cues. In: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction. ACM, New York, NY, USA, pp 39:1--39:4

  • Schlösser, C., Harrer, A., Kienle, A. (2018) Supporting dyadic chat communication with eye tracking based reading awareness. In: 2018 IEEE 18th international conference on advanced learning technologies (ICALT). pp 149–151

  • Schulze, J., Krumm, S.: The “virtual team player”: a review and initial model of knowledge, skills, abilities, and other characteristics for virtual collaboration. Organ. Psychol. Rev. 7, 66–95 (2017). https://doi.org/10.1177/2041386616675522

    Article  Google Scholar 

  • Scoular, C., Care, E.: Monitoring patterns of social and cognitive student behaviors in online collaborative problem solving assessments. Comput. Human Behav. 104, 105874 (2020). https://doi.org/10.1016/j.chb.2019.01.007

    Article  Google Scholar 

  • Shree Krishna, S., Angela, EBS., Arjun Ramesh, R., Sidney, KD. (2020) Multimodal, Multiparty Modeling of Collaborative Problem Solving Performance. Proceedings of the 2020 Conference on Multimodal Interaction

  • Shute, V.: Focus on formative feedback. Rev. Educ. Res. 78, 153–189 (2008)

    Article  Google Scholar 

  • Shute, V.J., Ventura, M., Kim, Y.J.: Assessment and learning of qualitative physics in Newton’s playground. J. Educ. Res. 106, 423–430 (2013)

    Article  Google Scholar 

  • Shute, V., Rahimi, S., Smith, G., et al.: Maximizing learning without sacrificing the fun: Stealth assessment, adaptivity and learning supports in educational games. J. Comput. Assist. Learn.comput. Assist. Learn 37, 127–141 (2021a). https://doi.org/10.1111/jcal.12473

    Article  Google Scholar 

  • Shute, V.J., Smith, G., Kuba, R., et al.: The design, development, and testing of learning supports for the physics playground game. Int. J. Artif. Intell. Educ.artif. Intell. Educ. 31, 357–379 (2021b). https://doi.org/10.1007/s40593-020-00196-1

    Article  Google Scholar 

  • Sinha, T., Cassell. J. (2015) We click, we align, we learn: Impact of influence and convergence processes on student learning and rapport building. In: Proceedings of the 1st Workshop on Modeling INTERPERsonal SynchrONy And infLuence. pp 13–20

  • Smagorinsky, P.: Deconflating the ZPD and instructional scaffolding: retranslating and reconceiving the zone of proximal development as the zone of next development. Learn. Cult. Soc. Interact. 16, 70–75 (2018)

    Article  Google Scholar 

  • Southwell, R., Pugh, S., Perkoff, M., et al (2022) Challenges and feasibility of automatic speech recognition for modeling student collaborative discourse in classrooms. In: Proceedings of the 15th International Educational Data Mining Conference (EDM 22)

  • Steiner, I.D.: Group processes and group productivity. Academic, New York (1972)

    Google Scholar 

  • Stephen, T.P., von Alina, A., D, Kurt P,: Computational psychometrics for the measurement of collaborative problem solving skills. Front. Psychol. 8, 2029 (2017). https://doi.org/10.3389/fpsyg.2017.02029

    Article  Google Scholar 

  • Stewart, A., Vrzakova, H., Sun, C., et al.: I say, you say, we say: using spoken language to model socio-cognitive processes during computer-supported collaborative problem solving. Proc. ACM Hum. Comput. Interact. 3(39), 1–19 (2019)

    Article  Google Scholar 

  • Stewart, A.E.B., Keirn, Z., D’Mello, S.K.: Multimodal modeling of collaborative problem-solving facets in triads. User Model User-Adapt Interact 31, 713–751 (2021)

    Article  Google Scholar 

  • Stewart, AEB., D’Mello, SK. (2018) Connecting the dots towards collaborative AIED: linking group makeup to process to learning. In: international conference on artificial intelligence in education. pp 545–556

  • Stewart, AEB., Keirn, ZA., D’Mello, SK. (2018) Multimodal modeling of coordination and coregulation patterns in speech rate during triadic collaborative problem solving. In: Proceedings of the 20th ACM international conference on multimodal interaction. ACM, New York, NY, USA, pp 21–30

  • Stewart, A., Rao, A., Michaels, A., et al (2023) CPSCoach: The design and implementation of intelligent collaborative problem solving feedback. In: Proceedings of the 24th international conference on artificial intelligence in education (AIED 2023), pp 695–700

  • Stoeffler, K., Rosen, Y., Bolsinova, M., von, DA. (2018) Gamified Assessment of Collaborative Skills with Chatbots. In: Book cover Book cover International Conference on Artificial Intelligence in Education. pp 343–347

  • Stuart, EA., Rubin, DB. (2008) Best practices in quasi-experimental designs. Best practices in quantitative methods 155–176

  • Subburaj, SK., Stewart, AEB., Rao, AR., D’Mello, SK. (2020) Multimodal, Multiparty Modeling of Collaborative Problem Solving Performance. In: Proceedings of the 2020 Conference on Multimodal Interaction

  • Sun, C., Shute, V., Stewart, A., et al.: Toward a generalized competency model of collaborative problem solving. Comput. Educ.. Educ. 143, 103672 (2020)

    Article  Google Scholar 

  • Sun, C., Shute, V.J., Stewart, A.E.B., et al.: The relationship between collaborative problem solving processes and objective outcomes in a game-based learning environment. Comput Human Behav 128, 107120 (2022)

    Article  Google Scholar 

  • Swiecki, Z., Ruis, A.R., Farrell, C., Shaffer, D.W.: Assessing individual contributions to Collaborative Problem Solving: A network analysis approach. Comput Human Behav 104, 105876 (2020). https://doi.org/10.1016/j.chb.2019.01.009

    Article  Google Scholar 

  • Tian, S., Zhang, A.X., Karger, D.: A System for Interleaving Discussion and Summarization in Online Collaboration. Proc. ACM Hum-Comput. Interact. (2021). https://doi.org/10.1145/3432940

    Article  Google Scholar 

  • Vaswani, A., Shazeer, N., Parmar, N., et al (2017) Attention is all you need. In: Advances in neural information processing systems. pp 5998–6008

  • Virtaneva, M., Feshchenko, P., Hossain, A., et al (2021) COVID-19 remote work: Body stress, self-efficacy, teamwork, and perceived productivity of knowledge workers. In: Scandinavian Conference on Information Systems. Association for Information Systems

  • Vrzakova, H., Amon, MJ., Stewart, A., D’Mello, SK. (2019) Dynamics of visual attention in multiparty collaborative problem solving using multidimensional recurrence quantification analysis. In: proceedings of the ACM CHI conference on human factors in computing systems (CHI 2019). ACM, New York

  • Vrzakova, H., Amon, MJ., Rees, M. et al (2020) Looking for a Deal? Social Visual Attention during Negotiations via Mixed Media Videoconferencing (in press). In: Proceedings of the ACM: Human Computer Interaction, Computer Supported Collaborative Work (CSCW 2020)

  • Webb, M., Gibson, D.: Technology enhanced assessment in complex collaborative settings. Educ Inf Technol (Dordr) 20, 675–695 (2015). https://doi.org/10.1007/s10639-015-9413-5

    Article  Google Scholar 

  • Wolf, T., Debut, L., Sanh, V., et al (2019) HuggingFace’s Transformers: State-of-the-art Natural Language Processing

  • Zhou, G., Moulder, R., Sun, C., D’Mello, SK. (2022) Investigating temporal dynamics underlying successful collaborative problem solving behaviors with multilevel vector autoregression. In: Proceedings of the 15th international educational data mining conference (EDM 22).

Download references

Acknowledgements

This research was supported by the NSF National AI Institute for Student-AI Teaming (iSAT) under DRL 2019805 and by DUE 1745442/1660877. The opinions expressed are those of the authors and do not represent views of the NSF. We also thank Chen Sun, Arjun Rao, Valerie Shute, Sam Pugh, and Quinton Beck-White for their contributions to the research.

Author information

Authors and Affiliations

  1. University of Colorado Boulder, Boulder, USA

    Sidney K. D’Mello

  2. Arizona State University, Glendale, USA

    Nicholas Duran & Amanda Michaels

  3. University of Pittsburgh, Pittsburgh, USA

    Angela E. B. Stewart

Authors
  1. Sidney K. D’Mello
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas Duran
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amanda Michaels
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Angela E. B. Stewart
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.D. and A.S wrote the main manuscript text. N.D. and A.G. provided reviews and feedback. A.G. and A.S. conducted the user studies. All authors intellectually contributed to the research.

Corresponding author

Correspondence to Sidney K. D’Mello.

Ethics declarations

Conflict of interest

The authors have no conflicts of interests.

Ethical approval

All research conducted has been approved by the cognizant IRBs and participants provided informed consent prior to the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Details on Study Procedure

We provide the following additional details to complement the overview provided in Sect. 3.2.

Round 1–3 gameplay. A randomly selected participant was chosen as the Controller (i.e., who interacts with the game mechanics) and the other as the Contributor (i.e., who provides suggestions) prior to the start of the first round. After completing 10 min of gameplay (Round 1), participants were separated into breakout rooms again and informed that they would receive feedback on their collaboration.

Rounds 2 & 3 gameplay & intervention on Facet A. Upon entering the breakout room, participants were shown a brief four-minute video that explained the three facets of the CPS framework and how their scores were generated. Participants then received feedback on all three facets and self-reported their perceived accuracy of the feedback for each facet. The facet with the lowest Round 1 score (Facet A) was selected for improvement, and this was communicated to the participants along with the Scaffold 1 Intervention. Participants had a maximum of 10 min to engage with the intervention, upon which the experimenter intervened. Participants who completed the intervention before the 10-min interval had elapsed simply informed the experimenter who instructed them to wait for their partner.

When both partners were back in the main room, a second 10-min round of gameplay (Round 2) commenced with the same participants assigned to the Controller vs. Contributor roles. When 10-min had elapsed, they were sent to separate breakout rooms where they received feedback on the same facet selected for improvement in the previous round (i.e., Facet A). They once again self-reported their perceptions of feedback accuracy for this facet only, upon which they received the Scaffold 2 Intervention to further improve on the same facet (i.e., Facet A). When both participants were done with the intervention or 10 min had elapsed, participants re-entered the main room and completed a third 10-min round of collaborative gameplay (Round 3).

Rounds 4 & 5 gameplay & intervention on Facet B. After Round 3, participants once again were moved to separate breakout rooms. They were then given a five-minute break. When they returned, they were informed that they would now focus on improving performance on a different facet (i.e., Facet B – the one with the second lowest score during Round 3) using the following instructions.

“In the following round you will change roles with your teammate. If you were controlling the game, you will now be observing your teammate and offering suggestions. If you were previously observing, you will now have control of gameplay. We'd like you to now focus on a different aspect of collaboration: <Facet Name>”

They received the Scaffold 1 Intervention, but this time for Facet B, and engaged with it as before. Once they were done or time had elapsed, they returned to the main room. They were informed that they would now switch roles with the previous Controller now becoming the Contributor and vice versa. They completed another 10-min of gameplay (Round 4) in their new roles. Next, they again moved to separate breakout rooms where they received feedback for Facet B, self-reported its perceived accuracy, and engaged with the Scaffold 2 Intervention for Facet B. Then, they returned to the main room and completed one last round (Round 5) with their new roles.

Appendix B: Histogram of Variables Analyzed

figure a

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

D’Mello, S.K., Duran, N., Michaels, A. et al. Improving collaborative problem-solving skills via automated feedback and scaffolding: a quasi-experimental study with CPSCoach 2.0. User Model User-Adap Inter (2024). https://doi.org/10.1007/s11257-023-09387-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11257-023-09387-6

Keywords

Search

Navigation