Skip to main content
SpringerLink
Log in

Mining User-Object Interaction Data for Student Modeling in Intelligent Learning Environments

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

This work seeks to contribute to the development of intelligent environments by presenting an approach oriented to the identification of On-Task and Off-Task behaviors in educational settings. This is accomplished by monitoring and analyzing the user-object interactions that users manifest while performing academic activities with a tangible-intangible hybrid system in a university intelligent environment configuration. With the proposal of a framework and the Orange Data Mining tool and the Neural Network, Random Forest, Naive Bayes, and Tree classification models, training and testing was carried out with the user-object interaction records of the 13 students (11 for training and two for testing) to identify representative sequences of behavior from user-object interaction records. The two models that had the best results, despite the small number of data, were the Neural Network and Naive Bayes. Although a more significant amount of data is necessary to perform a classification adequately, the process allowed exemplifying this process so that it can later be fully incorporated into an intelligent educational system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

REFERENCES

  1. Virone, G., et al., Behavioral patterns of older adults in assisted living, IEEE Trans. Inf. Technol. Biomed., 2008, vol. 12, no. 3, pp. 387–398. https://doi.org/10.1109/TITB.2007.904157

    Article  Google Scholar 

  2. Pantic, M., et al., Human computing and machine understanding of human behavior: a survey, Proc. 8th Int. Conf. on Multimodal Interfaces, Banff Alberta, 2006.

  3. Favela, J., Activity, behavior, and context: the ABC of pervasive healthcare research, Proc. 8th Int. Conf. on Intelligent Environments, Guanajuato, 2012.

  4. Nef, T., et al., Evaluation of three state-of-the-art classifiers for recognition of activities of daily living from smart home ambient data, Sensors, 2012, vol. 15.5, pp. 11725–11740.

    Article  Google Scholar 

  5. Vinh, L.T., et al., Semi-Markov conditional random fields for accelerometer-based activity recognition, Appl. Intellig., 2011, vol. 35, pp. 226–241.

    Article  Google Scholar 

  6. Moore, D.W., et al., Increasing on-task behavior in students in a regular classroom: effectiveness of a self-management procedure using a tactile prompt, J. Behav. Educ., 2013, vol. 22, pp. 302–311.

    Article  Google Scholar 

  7. Godwin, K.E., et al., Off-task behavior in elementary school children, Learn. Instr., 2016, vol. 44, pp. 128–143.

    Article  Google Scholar 

  8. Doddannara, L.S., et al., Exploring the relationships between design, students’ affective states, and disengaged behaviors within an ITS, in Proc. 16th Int. Conf. on Artificial Intelligence in Education AIED 2013, Memphis, TN, USA, July 9–13, 2013, Berlin, Heidelberg: Springer, 2013.

  9. Wang Li-Chun, and Ming-Puu Chen, The effects of game strategy and preference-matching on flow experience and programming performance in game-based learning, Innov. Educ. Teach. Int., 2010, vol. 47.1, pp. 39–52.

    Article  Google Scholar 

  10. Schaufeli, W.B. and Bakker, A., UWES Utrecht Work Engagement Scale Preliminary Manual, Utrecht Univ., 2003.

    Google Scholar 

  11. Bumbacco, C. and Scharfe, E., Why attachment matters: first-year post-secondary students’ experience of burnout, disengagement, and drop-out, J. Coll. Stud. Retention: Res., Theory Pract., 2023, vol. 24, no. 4, pp. 988–1001.

    Google Scholar 

  12. Favela, J., Behavior-aware computing: applications and challenges, IEEE Pervasive Comput., 2013, vol. 12, no. 3, pp. 14–17.

    Article  Google Scholar 

  13. Aztiria, A., et al., Learning frequent behaviors of the users in intelligent environments, IEEE Trans. Syst., Man, Cybernet., 2013, vol. 43, no. 6, pp. 1265–1278.

    Article  Google Scholar 

  14. Zhang, J., Mohd, Y.Z., and Haron, H., Interactivity recognition graph neural network (IR-GNN) model for improving human-object interaction detection, Electronics, 2013, vol. 12, no. 2, p. 470. https://doi.org/10.3390/electronics12020470

    Article  Google Scholar 

  15. Qing Ye and Xiuju Xu, Human-object interaction detection based on graph model, Proc. 3rd SPIE Int. Conf. on Artificial Intelligence and Computer Engineering (ICAICE 2022), Wuhan, 2022. https://doi.org/10.1117/12.2671248.

  16. Mazzamuto, M., et al., A wearable device application for human-object interactions detection, Proc. Int. Conf. on Computer Vision Theory and Applications (VISAPP), Lisbon, 2023.

  17. Luo, T., et al., From detection to understanding: a survey on representation learning for human-object interaction, Neurocomputing, 2023, vol. 543, p. 126243.

  18. Deunsol, J. and Minsu, C., Relational context learning for human-object interaction detection, Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Vancouver, 2023.

  19. Ye Qing, et al., Human object interaction detection based on feature optimization and key human-object enhancement, J. Vis. Commun. Image Representation, 2023, vol. 93, p. 103824.

  20. SungJin, N., Frishkoff, G., and Collins-Thompson,K., Predicting students disengaged behaviors in an online meaning-generation task, IEEE Trans. Learn. Technol., 2017, vol. 11.3, pp. 362–375.

    Google Scholar 

  21. Fu Rong, et al., Learning behavior analysis in classroom based on deep learning, Proc. 10th IEEE Int. Conf. on Intelligent Control and Information Processing (ICICIP), Marrakesh, 2019.

  22. Jayarajah Kasthuri, et al., Need accurate user behaviour? Pay attention to groups!, Proc. ACM Int. Joint Conf. on Pervasive and Ubiquitous Computing, Osaka, 2015.

  23. Baker, R.S., et al., Off-task behavior in the cognitive tutor classroom: when students game the system, Proc. SIGCHI Conf. on Human Factors in Computing Systems, Vienna, 2004.

  24. Baker, R.S., Modeling and understanding students’ off-task behavior in intelligent tutoring systems, Proc. SIGCHI Conf. on Human Factors in Computing Systems, San Jose, 2007.

  25. Zidianakis, E., et al., Building a sensory infrastructure to support interaction and monitoring in ambient intelligence environments, in Proc. 2nd Int. Conf. on Distributed, Ambient, and Pervasive Interactions: DAPI 2014, Held as Part of HCI Int. 2014, Heraklion, Crete, Greece, June 22–27, 2014, Springer Int. Publ., 2014.

  26. D’Mello, S.K., et al., Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms, Proc. ACM Int. Conf. on Multimodal Interaction, Seattle, 2015.

  27. Belapurkar, N., Sagar, S., and Baris, A., The case for ambient sensing for human activity detection, Proc. 8th Int. Conf. on the Internet of Things, Santa Barbara, 2018.

  28. Leonidis, A., et al., An intelligent task assignment and personalization system for students’ online collaboration, in Proc. 6th Int. Conf. on Universal Access in Human-Computer Interaction, Applications and Services: UACHI 2011, Held as Part of HCI Int. 2011, Orlando, FL, USA July 9–14, 2011, Berlin, Heidelberg: Springer, 2011, part 6.

  29. Rai, S. and Xiaolin Hu, Behavior pattern detection for data assimilation in agent-based simulation of smart environments, Proc. IEEE/WIC/ACM Int. Joint Conf. on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), Atlanta, 2013, vol. 2.

  30. Wang Qunbo, Wenjun Wu, and Yuxing Qi, A learning analytic model for smart classroom, in Proc. International Workshops: MWDA, BAH, KGMA, DMMOOC, DS on Web and Big Data: APWeb-WAIM, Macau, China, July 23–25, 2018, Springer Int. Publ., 2018.

  31. Korozi, M., et al., LECTOR: towards reengaging students in the educational process inside smart classrooms, in Proc. 9th Int. Conf. on Intelligent Human Computer Interaction: IHCI 2017, Evry, France, Dec. 11–13, 2017, Springer Int. Publ., 2017.

  32. White, R.W., Awadallah, A.H., and Sim, R., Task completion detection: a study in the context of intelligent systems, Proc. 42nd Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, Paris, 2019.

  33. Montebello, M., Assisting education through real-time learner analytics, Proc. IEEE Frontiers in Education Conf. (FIE), San Jose, 2018.

  34. Matsui, K., Tatsuhiko, K., and Keiya, S., Challenges for data collecting of teacher and student’ behavior in different types of class using video and wearable device, Proc. Joint 8th Int. Conf. on Informatics, Electronics & Vision (ICIEV) and 3rd IEEE Int. Conf. on Imaging, Vision & Pattern Recognition (icIVPR), Spokane, WA, 2019.

  35. Prabono, A.G., Lee, S.-L., and Yahya, B.N., Context-based similarity measure on human behavior pattern analysis, Soft Comput., 2019, vol. 23, pp. 5455–5467.

    Article  Google Scholar 

  36. Biswas, G. and Sulcer, B., Visual exploratory data analysis methods to characterize student progress in intelligent learning environments, Proc. IEEE Int. Conf. on Technology for Education, Mumbai, 2010.

  37. Bruno, B., et al., Analysis of human behavior recognition algorithms based on acceleration data, Proc. IEEE Int. Conf. on Robotics and Automation, Karlsruhe, 2013.

  38. Guivarch, V., et al., Hybrid system to analyze user’s behaviour, Proc. IEEE Symp. Series on Computational Intelligence (SSCI), Athens, 2016.

  39. Mathioudakis, G., et al., Ami-ria: real-time teacher assistance tool for an ambient intelligence classroom, Proc. 5th Int. Conf. on Mobile, Hybrid, and Online Learning (eLmL 2013), Nice, 2013.

  40. Atallah, L. and Guang-Zhong, Y., The use of pervasive sensing for behaviour profiling – a survey, Pervasive Mobile Comput., 2009, vol. 5.5, pp. 447–464.

    Article  Google Scholar 

  41. Papamitsiou, Z., Karapistoli, E., and Econo-mides, A.A., Applying classification techniques on temporal trace data for shaping student behavior models, Proc. 6th Int. Conf. on Learning Analytics & Knowledge, Edinburgh, 2016.

  42. Soto-Mendoza, V., et al., Detecting abnormal behaviours of institutionalized older adults through a hybrid-inference approach, Pervasive Mobile Comput., 2017, vol. 40, pp. 708–723.

    Article  Google Scholar 

  43. Lattner, A.D., et al., Sequential pattern mining for situation and behavior prediction in simulated robotic soccer, in Proc. RoboCup 2005: Robot Soccer World Cup IX 9, Berlin, Heidelberg: Springer, 2006.

  44. Magnusson, M.S., Discovering hidden time patterns in behavior: T-patterns and their detection, Behav. Res. Methods, Instrum. Comput., 2000, vol. 32.1, pp. 93–110.

    Article  Google Scholar 

  45. Casarrubea, M., et al., T-pattern analysis for the study of temporal structure of animal and human behavior: a comprehensive review, J. Neurosci. Methods, 2015, vol. 239, pp. 34–46.

    Article  Google Scholar 

  46. Santoyo, C., et al., Observational analysis of the organization of on-task behavior in the classroom using complementary data analyses, An. Psicol., 2017, vol. 33, no. 3, pp. 497–514.

    Google Scholar 

  47. Srikant, R. and Rakesh Agrawal, Mining sequential patterns: generalizations and performance improvements, in Proc. 5th Int. Conf. on Extending Database Technology “Advances in Database Technology” – EDBT’96: Avignon, France, March 25–29, 1996, Berlin, Heidelberg: Springer, 1996.

  48. Hernández-Calderón, J.G., et al., A system to match behaviors and performance of learners from user-object interactions: model and architecture, Int. J. Inf. Technol. Syst. Approach, 2019, vol. 12.2, pp. 82–103.

    Article  Google Scholar 

  49. Cantabella, M., et al., Analysis of student behavior in learning management systems through a Big Data framework, Future Gener. Comput. Syst., 2019, vol. 90, pp. 262–272.

    Article  Google Scholar 

  50. Xohua-Chacón, J.A., Benítez-Guerrero, E.I., and Mezura-Godoy, C., A tangible system for learning relational algebra, Rev. Colomb. Comput., 2018, vol. 19.1, pp. 39–55.

    Article  Google Scholar 

  51. United States National Commission for the Protection of Human Subjects of Biomedical, and Behavioral Research, The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research, vol. 1: Department of Health, Education, and Welfare, National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1978.

  52. Buza, K. and Schmidt-Thieme, L., Motif-based classification of time series with bayesian networks and SVMS, in Proc. 32nd Annu. Conf. of the Gesellschaft für Klassifikation eV, Joint Conf. with the British Classification Society (BCS) and the Dutch/Flemish Classification Society (VOC) Advances in Data Analysis, Data Handling and Business Intelligence, Hamburg: Helmut-Schmidt Univ., July 16–18, 2008, Berlin, Heidelberg: Springer, 2010.

  53. Ren, J.-M., and Jang, J.-S.R., Time-constrained sequential pattern discovery for music genre classification, Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Prague, 2011.

  54. Camacho, V. and Alejandra, V., Detección de emociones negativas en las redes sociales, Ph.D. Thesis, 2017.

  55. Lu, O.H.T., et al., Applying learning analytics for the early prediction of students’ academic performance in blended learning, J. Educ. Technol. Soc., 2018, vol. 21.2, pp. 220–232.

    Google Scholar 

  56. Jiménez, S., Juárez-Ramírez, R., Castillo, V.H., et al., The role of personality in motivation to use an affective feedback system, Program. Comput. Software, 2021, vol. 47, pp. 793–802. https://doi.org/10.1134/S0361768821080156

    Article  Google Scholar 

  57. Aguilar-Cisneros, J., Valerdi, R., and Sullivan, B.P., Students’ systems thinking competency level detection through software cost estimation concept modeling, Program. Comput. Software, 2022, vol. 48, pp. 499–512. https://doi.org/10.1134/S0361768822080060

    Article  Google Scholar 

  58. Kochegurova, E.A. and Martynova, Y.A., Aspects of continuous user identification based on free texts and hidden monitoring, Program. Comput. Software, 2020, vol. 46, pp. 12–24. https://doi.org/10.1134/S036176882001003X

    Article  Google Scholar 

  59. Del-Ángel-Flores, H., López-Domínguez, E., Hernández-Velázquez, Y., et al., Usability evaluation of a mobile learning platform focused on learning monitoring and customization based on a laboratory study, Program. Comput. Software, 2022, vol. 48, pp. 583–597. https://doi.org/10.1134/S0361768822080102

    Article  Google Scholar 

Download references

ACKNOWLEDGMENTS

The first author gratefully acknowledges CONAHCYT for scholarship no. 421557 for graduate studies.

Funding

This work was supported by ongoing institutional funding. No additional grants to carry out or direct this particular research were obtained.

Author information

Authors and Affiliations

  1. Facultad de Estadística e Informática, Universidad Veracruzana, Veracruz, Xalapa, México

    J. G. Hernández-Calderón, E. Benítez-Guerrero, J. R. Rojano-Cáceres & Carmen Mezura-Godoy

Authors
  1. J. G. Hernández-Calderón
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. E. Benítez-Guerrero
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. R. Rojano-Cáceres
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carmen Mezura-Godoy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to J. G. Hernández-Calderón, E. Benítez-Guerrero, J. R. Rojano-Cáceres or Carmen Mezura-Godoy.

Ethics declarations

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s Note.

Pleiades Publishing remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hernández-Calderón, J.G., Benítez-Guerrero, E., Rojano-Cáceres, J.R. et al. Mining User-Object Interaction Data for Student Modeling in Intelligent Learning Environments. Program Comput Soft 49, 657–670 (2023). https://doi.org/10.1134/S036176882308008X

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S036176882308008X

Keywords:

Search

Navigation