Skip to main content
SpringerLink
Log in

Human Event Recognition in Smart Classrooms Using Computer Vision: A Systematic Literature Review

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

The field of human event recognition using visual data in smart environments has emerged as a fruitful and successful area of study, with extensive research and development efforts driving significant advancements. These advancements have not only provided valuable insights, but also led to practical applications in various domains. In this context, human actions, activities, interactions, and behaviors can all be considered as events of interest in smart environments. However, when it comes to smart classrooms, there is a lack of unified consensus on the definition of the term “human event.” This lack of agreement presents a significant challenge for educators, researchers, and developers, as it hampers their ability to precisely identify and classify the specific situations that are relevant within the educational context. The aim of this paper is to address this challenge by conducting a systematic literature review of relevant events in smart classrooms, with a focus on their applications in assistive technology. The review encompasses a comprehensive analysis of 227 published documents spanning from 2012 to 2022. It delves into key algorithms, methodologies, and applications of vision-based event recognition in smart environments. As the primary outcome, the review identifies the most significant events, classifying them according to single person behavior, or multiple-person interactions, or object-person interactions. It also examines their practical applications within the educational context. The paper concludes with a discussion on the relevance and practicality of vision-based human event recognition in smart classrooms, particularly in the post-COVID era.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

REFERENCES

  1. Ducatel, K., Technologies de la societe de l’information, Societe de l’information conviviale, Scenarios for ambient intelligence in 2010 U. europeenne, Institut d’etudes de prospectives technologiques and U. europeenne, 2001.

  2. Cook, D.J., Augusto, J.C., and Jakkula, V.R., Ambient intelligence: Technologies, applications, and opportunities, Pervasive Mobile Comput., 2009, vol. 5, no. 4, pp. 277–298.

    Article  Google Scholar 

  3. Saina, M.K. and Goel, N., How smart are smart classrooms? A review of smart classroom technologies, ACM Comput. Surv., 2019, vol. 52, no. 6, pp. 1–28. https://doi.org/10.1145/3365757

    Article  Google Scholar 

  4. Guinard, D., Fischer, M., and Trifa, V., Sharing using social networks in a composable web of things, Proc. 8th IEEE Int. Conf. on Pervasive Computing and Communications Workshops (PERCOM Workshops), Mannheim, 2010, pp. 702–707. https://doi.org/10.1109/percomw.2010.5470524

  5. Radosavljevic, V., Radosavljevic, S., and Jelic, G., Ambient intelligence-based smart classroom model, Interact. Learn. Environ., 2022, vol. 30, no. 2. https://doi.org/10.1080/10494820.2019.1652836

  6. Kwet, M. and Prinsloo, P., The “smart” classroom: A new frontier in the age of the smart university, Teach. High. Educ., 2020, vol. 25, no. 4, pp. 510–526. https://doi.org/10.1080/13562517.2020.1734922

    Article  Google Scholar 

  7. Candamo, J., Shreve, M., Goldgof, D.B., Sapper, D.B., and Kasturi, R., Understanding transit scenes: A survey on human behavior-recognition algorithms, IEEE Trans. Intell. Transp. Syst., 2009, vol. 11, no. 1, pp. 206–224. https://doi.org/10.1109/TITS.2009.2030963

    Article  Google Scholar 

  8. Ahad, M.A.R., Vision and sensor-based human activity recognition: Challenges ahead, in Advancements in Instrumentation and Control in Applied System Applications, IGI Global, 2020, pp. 17–35. https://doi.org/10.4018/978-1-7998-2584-5

    Book  Google Scholar 

  9. Beddiar, D.R., Nini, B., Sabokrou, M., and Hadid, A., Vision based human activity recognition: A survey, Multimedia Tools Appl., 2020, vol. 79, no. 41, pp. 30509–30555.

    Article  Google Scholar 

  10. Zhang, H.-B., Zhang, Y.-X., Zhong, B., Lei, Q., Yang, L., and Du, J.-X., and Chen, D.-S., A comprehensive survey of vision based human action recognition methods, Sensors (Basel), 2019, vol. 19, no. 5, p. 1005.

    Article  Google Scholar 

  11. Kong, Y. and Fu, Y., Human action recognition and prediction: A survey, 2018. arXiv:1806.11230

  12. Tripathi, R.K. and Jalal, A.S., and Agrawal, S.C., Suspicious human activity recognition: A review, Artif. Intell. Rev., 2018, vol. 50, no. 2, pp. 283–339. https://doi.org/10.1007/s10462-017-9545-7

    Article  Google Scholar 

  13. Jegham, I., Khalifa, A.B., Alouani, I., and Mahjoub, M.A., Vision-based human action recognition: An overview and real world challenges, Forensic Sci. Int. Digit. Investig., 2020, vol. 32, p. 200901. https://doi.org/10.1016/j.fsidi.2019.200901

  14. Paredes-Valles, F., Scheper, K.Y., and De Croon, G.C., Unsupervised learning of a hierarchical spiking neural network for optical flow estimation: From events to global motion perception, IEEE Trans. Pattern Anal. Mach. Intell., 2019, vol. 42, no. 8, pp. 2051–2064.

    Article  Google Scholar 

  15. Garcia-Garcia, B., Bouwmans, T., and Silva, A.J.R., Background subtraction in real applications: Challenges, current models and future directions, Comput. Sci. Rev., 2020, vol. 35, p. 100204.

  16. Ahmad, M., Ahmed, I., Ullah, K., Khan, I., and Adnan, A., Robust background subtraction based person’s counting from overhead view, Proc. 9th IEEE Annu. Ubiquitous Computing, Electronics and Mobile Communication Conf. (UEMCON), New York, 2018, pp. 746–752.

  17. Fan, Y., Wen, G., Li, D., Qiu, S., Levine, M.D., and Xiao, F., Video anomaly detection and localization via Gaussian mixture fully convolutional variational autoencoder, Comput. Vis. Image Underst., 2020, vol. 195, p. 102920. https://doi.org/10.1016/j.cviu.2020.102920

  18. Cho, J., Jung, Y., Kim, D.-S., Lee, S., and Jung, Y., Moving object detection based on optical flow estimation and a Gaussian mixture model for advanced driver assistance systems, Sensors (Basel), 2019, vol. 19, no. 14, p. 3217.

    Article  Google Scholar 

  19. Yang, H. and Qu, S., Real-time vehicle detection and counting in complex traffic scenes using background subtraction model with low-rank decomposition, IET Intell. Transp. Syst., 2018, vol. 21, no. 1, pp. 75–85.

    Article  Google Scholar 

  20. Martins, I., Carvalho, P., Corte-Real, L., and Alba-Castro, J.L., Bmog: Boosted gaussian mixture model with controlled complexity for background subtraction, Pattern Anal. Appl., 2018, vol. 21, no. 3, pp. 641–654.

    Article  MathSciNet  Google Scholar 

  21. Zerrouki, N., Harrou, F., Sun, Y., and Houacine, A., Visionbased human action classification using adaptive boosting algorithm, IEEE Sens. J., 2018, vol. 12, no. 12, pp. 5115–5121.

    Article  Google Scholar 

  22. Boregowda, L. and Rajagopal, A., US Patent Appl, 11/227,505, 2007.

  23. Bird, N.D., Masoud, O., Papanikolopoulos, N.P., and Isaacs, A., Detection of loitering individuals in public transportation areas, IEEE Trans. Intell. Transp. Syst., 2005, vol. 6, no. 2, pp. 167–177. https://doi.org/10.1109/TITS.2005.848370

    Article  Google Scholar 

  24. Zhang, R., Vogler, C., and Metaxas, D., Human gait recognition at sagittal plane, Image Vis. Comput., 2007, vol. 25, no. 3, pp. 321–330. https://doi.org/10.1016/j.imavis.2005.10.007

    Article  Google Scholar 

  25. Kukreja, V., Kumar, D., and Kaur, A., Deep learning in Human Gait Recognition: An overview, Proc. IEEE Int. Conf. on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, 2021, pp. 9–13.

  26. Sevik, U., Karakullukcu, E., Berber, T., Akba, Y., and Turkyılmaz, S., Automatic classification of skin burn colour images using texture-based feature extraction, IET Image Process., 2019, vol. 13, no. 11, pp. 2018–2028.

    Article  Google Scholar 

  27. Taylor, M.J. and Morris, T., Adaptive skin segmentation via feature-based face detection, in Real-Time Image and Video Processing, International Society for Optics and Photonics, 2014, vol. 9139, p. 91390P. https://doi.org/10.1117/12.2052003

    Book  Google Scholar 

  28. Rodriguez, S.A., Fremont, V., Bonnifait, P., and Cherfaoui, V., An embedded multi-modal system for object localization and tracking, IEEE Intell. Transp. Syst. Mag., 2012, vol. 4, no. 4, pp. 42–53. https://doi.org/10.1109/MITS.2012.2217855

    Article  Google Scholar 

  29. Lu, Y., Lu, C., and Tang, C.-K., Online video object detection using association LSTM, Proc. IEEE Int. Conf. on Computer Vision, Venice, 2017, pp. 2344–2352. https://doi.org/10.1109/ICCV.2017.257

  30. Messing, R., Pal, C., and Kautz, H., Activity recognition using the velocity histories of tracked keypoints, Proc. 12th IEEE Int. Conf. on Computer Vision, Kyoto, 2009, pp. 104–111. https://doi.org/10.1109/ICCV.2009.5459154

  31. Leichter, I., Lindenbaum, M., and Rivlin, E., Mean shift tracking with multiple reference color histograms, Comput. Vis. Image Underst., 2010, vol. 114, no. 3, pp. 400–408. https://doi.org/10.1016/j.cviu.2009.12.006

    Article  Google Scholar 

  32. Wu, Y., Lim, J., and Yang, M.-H., Online object tracking: a benchmark, Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Portland, OR, 2013, pp. 2411–2418. https://doi.org/10.1109/TPAMI.2014.2388226

  33. Alpaydin, E., Introduction to Machine Learning, MIT Press, 2020. https://doi.org/10.1016/j.neuroimage.2010.11.004

    Book  Google Scholar 

  34. Li, Y., On incremental and robust subspace learning, Pattern Recognit., 2004, vol. 37, no. 7, pp. 1509–1518. https://doi.org/10.1016/j.patcog.2003.11.010

    Article  Google Scholar 

  35. Mairesse, F., Gasic, M., Jurcicek, F., Keizer, S., Thomson, B., Yu, K., and Young, S., Spoken language understanding from unaligned data using discriminative classification models, Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, Taipei, 2009, pp. 4749–4752. https://doi.org/10.1109/ICASSP.2009.4960692

  36. Haigh, K.Z., Kiff, L.M., and Ho, G., The independent lifestyle assistant: lessons learned, Assist. Technol., 2006, vol. 18, no. 1, pp. 87–106. https://doi.org/10.1080/10400435.2006.10131909

    Article  Google Scholar 

  37. Chen, L., and Khalil, I., Activity recognition: Approaches, practices, and trends, in Activity Recognition in Pervasive Intelligent Environments, Springer, 2011, pp. 1–31. https://doi.org/10.2991/978-94-91216-05-3_1

  38. Chen, L. and Nugent, C.D., Human Activity Recognition and Behaviour Analysis, Springer, 2019.

    Book  Google Scholar 

  39. Bouchard, B., Giroux, S., and Bouzouane, A., A smart home agent for plan recognition of cognitively-impaired patients, J. Comput. (Taipei), 2006, vol. 1, no. 5, pp. 53–62.

    Google Scholar 

  40. Chen, L., Nugent, C.D., and Wang, H., A knowledge-driven approach to activity recognition in smart homes, IEEE Trans. Knowl. Data Eng., 2011, vol. 24, no. 6, pp. 961–974. https://doi.org/10.1109/TKDE.2011.51

    Article  Google Scholar 

  41. Granada, R.L., Pereira, R.F., Monteiro, J., Barros, R.C., Ruiz, D., and Meneguzzi, F., Hybrid activity and plan recognition for video streams, Proc. Workshops at the 31st AAAI Conf. on Artificial Intelligence, San Francisco, 2017.

  42. Kautz, H., Etzioni, O., Fox, D., Weld, D., and Shastri, L., Foundations of assisted cognition systems, Technical Report, Univ. of Washington: Computer Science Department, 2003.

    Google Scholar 

  43. Chen, E.S., Melton, G.B., Engelstad, M.E., and Sarkar, I.N., Standardizing clinical document names using the HL7/LOINC document ontology and LOINC codes, in Proc. AMIA Annu. Symp., American Medical Informatics Association, 2010, vol. 2010, p. 101.

  44. Hakeem, A. and Shah, M., Multiple agent event detection and representation in videos, Proc. 20th AAAI National Conf. on Artificial Intelligence, Pittsburgh, 2005, pp. 89–94.

  45. Georis, B., Maziere, M., Bremond, F., and Thonnat, M., A video interpretation platform applied to bank agency monitoring, Proc. Conf. on Intelligent Distributed Surveillance Systems (IDSS-04), London, 2004. https://doi.org/10.1049/ic:20040097

  46. SanMiguel, J.C., Martinez, J.M., and Garcia, A., An ontology for event detection and its application in surveillance video, Proc. 6th IEEE Int. Conf. on Advanced Video and Signal Based Surveillance, Genova, 2009, pp. 220–225. https://doi.org/10.1109/AVSS.2009.28

  47. Francois, A.R., Nevatia, R., Hobbs, J., Bolles, R.C., and Smith, J.R., VERL: An ontology framework for representing and annotating video events, IEEE Multimed., 2005, vol. 12, no. 4, pp. 76–86. https://doi.org/10.1109/AVSS.2009.28

    Article  Google Scholar 

  48. Akdemir, U., Turaga, P., and Chellappa, R., An ontology based approach for activity recognition from video, Proc. 16th ACM Int. Conf. on Multimedia, Vancouver, 2008, pp. 709–712. https://doi.org/10.1145/1459359.1459466

  49. Yao, B., Hagras, H., Alhaddad, M.J., and Alghazzawi, D., A fuzzy logic-based system for the automation of human behavior recognition using machine vision in intelligent environments, Soft Comput., 2015, vol. 19, no. 2, pp. 499–506. https://doi.org/10.1007/s00500-014-1270-4

    Article  Google Scholar 

  50. Ikizler, N. and Forsyth, D., Searching video for complex activities with finite state models, Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Minneapolis, MN, 2007, pp. 1–8.

  51. Ballan, L., Bertini, M., Del Bimbo, A., Seidenari, L., and Serra, G., Event detection and recognition for semantic annotation of video, Multimedia Tools Appl., 2011, vol. 51, no. 1, pp. 279–302. http://hdl.handle.net/11380/979935

    Article  Google Scholar 

  52. Xiao, Y. and Watson, M., Guidance on conducting a systematic literature review, J. Plann. Educ. Res., 2019, vol. 39, no. 1, pp. 93–112.

    Article  Google Scholar 

  53. Hu, W., Tan, T., Wang, L., and Maybank, S., A survey on visual surveillance of object motion and behaviors, IEEE Trans. Syst. Man Cybern., 2004, vol. 34, no. 3, pp. 334–352.

    Article  Google Scholar 

  54. Valera, M. and Velastin, S.A., Intelligent distributed surveillance systems: a review, IEE Proc. Vis. Image Signal Process., 2005, vol. 152, no. 2, pp. 192–204. https://doi.org/10.1049/ip-vis:20041147

    Article  Google Scholar 

  55. Turaga, P., Chellappa, R., Subrahmanian, V.S., and Udrea, O., Machine recognition of human activities: A survey, IEEE Trans. Circ. Syst. Video Tech., 2008, vol. 18, no. 11, pp. 1473–1488. https://doi.org/10.1109/TCSVT.2008.2005594

    Article  Google Scholar 

  56. Poppe, R., A survey on vision-based human action recognition, Image Vis. Comput., 2010, vol. 28, no. 6, pp. 976–990. https://doi.org/10.1016/j.imavis.2009.11.014

    Article  Google Scholar 

  57. Aggarwal, J.K. and Xia, L., Human activity recognition from 3d data: a review, Pattern Recognit. Lett., 2014, vol. 48, pp. 70–80. https://doi.org/10.1016/j.patrec.2014.04.011

    Article  Google Scholar 

  58. Aggarwal, J.K. and Ryoo, M.S., Human activity analysis: A review, ACM Comput. Surv., 2011, vol. 43, no. 3, pp. 1–43. https://doi.org/10.1145/1922649.1922653

    Article  Google Scholar 

  59. Chaaraoui, A.A., Climent-Perez, P., and Florez-Revuelta, F., A review on vision techniques applied to human behaviour analysis for ambient-assisted living, Expert Syst. Appl., 2012, vol. 39, no. 12, pp. 10873–10888. https://doi.org/10.1016/j.eswa.2012.03.005

    Article  Google Scholar 

  60. Popoola, O.P. and Wang, K., Video-based abnormal human behavior recognition–A review, IEEE Trans. Syst. Man Cybern. C, 2012, vol. 42, no. 6, pp. 865–878 https://doi.org/10.1109/TSMCC.2011.2178594

    Article  Google Scholar 

  61. Sodemann, A.A., Ross, M.P., and Borghetti, B.J., A review of anomaly detection in automated surveillance, IEEE Trans. Syst. Man Cybern. C, 2012, vol. 42, no. 6, pp. 1257–1272. https://doi.org/10.1109/TSMCC.2012.2215319

    Article  Google Scholar 

  62. Borges, P.V.K., Conci, N., and Cavallaro, A., Video-based human behavior understanding: A survey, IEEE Trans. Circ. Syst. Video Tech., 2013, vol. 23, no. 11, pp. 1993–2008. https://doi.org/10.1109/TCSVT.2013.2270402

    Article  Google Scholar 

  63. Chaquet, J.M., Carmona, E.J., and Fernandez-Caballero, A., A survey of video datasets for human action and activity recognition, Comput. Vis. Image Underst., 2013, vol. 117, no. 6, pp. 633–659. https://doi.org/10.1016/j.cviu.2013.01.013

    Article  Google Scholar 

  64. Vishwakarma, S. and Agrawal, A., A survey on activity recognition and behavior understanding in video surveillance, Vis. Comput., 2013, vol. 29, no. 10, pp. 983–1009. https://doi.org/10.1007/s00371-012-0752-6

    Article  Google Scholar 

  65. Guo, G. and Lai, A., A survey on still image based human action recognition, Pattern Recognit., 2014, vol. 47, no. 10, pp. 3343–3361. https://doi.org/10.1016/j.patcog.2014.04.018

    Article  Google Scholar 

  66. Lowe, S.A. and Laighin, G.O., Monitoring human health behaviour in one’s living environment: A technological review, Med. Eng. Phys., 2014, vol. 36, no. 2, pp. 147–168. https://doi.org/10.1016/j.medengphy.2013.11.010

    Article  Google Scholar 

  67. Amiribesheli, M., Benmansour, A., and Bouchachia, A., A review of smart homes in healthcare, J. Ambient Intell. Humaniz. Comput., 2015, vol. 6, no. 4, pp. 495–517. https://doi.org/10.1007/s12652-015-0270-2

    Article  Google Scholar 

  68. Hurney, P., Waldron, P., Morgan, F., Jones, E., and Glavin, M., Review of pedestrian detection techniques in automotive farinfrared video, IET Intell. Transp. Syst., 2015, vol. 9, no. 8, pp. 824–832. https://doi.org/10.1049/iet-its.2014.0236

    Article  Google Scholar 

  69. Ziaeefard, M. and Bergevin, R., Semantic human activity recognition: A literature review, Pattern Recognit., 2015, vol. 48, no. 8, pp. 2329–2345. https://doi.org/10.1016/j.patcog.2015.03.006

    Article  Google Scholar 

  70. Ramezani, M. and Yaghmaee, F., A review on human action analysis in videos for retrieval applications, Artif. Intell. Rev., 2016, vol. 46, no. 4, pp. 485–514. https://doi.org/10.1007/s10462-016-9473-y

    Article  Google Scholar 

  71. Subetha, T. and Chitrakala, S., A survey on human activity recognition from videos, Proc. IEEE Int. Conf. on Information Communication and Embedded Systems (ICICES), Chennai, 2016, pp. 1–7. https://doi.org/10.1109/ICICES.2016.7518920

  72. Mahata, J. and Phadikar, A., Recent advances in human behaviour understanding: A survey, Proc. IEEE Conf. on Devices for Integrated Circuit (DevIC), Kalyani, 2017, pp. 751–755. https://doi.org/10.1109/DEVIC.2017.8074052

  73. Rashmi, S., Bhat, S., and Sushmitha, V., Evaluation of human action recognition techniques intended for video analytics, Proc. IEEE Int. Conf. on Smart Technologies for Smart Nation (SmartTechCon), Bengaluru, 2017, pp. 357–362. https://doi.org/10.1109/SmartTechCon.2017.8358396

  74. Rohit, K., Mistree, K., and Lavji, J., A review on abnormal crowd behavior detection, Proc. IEEE Int. Conf. on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, 2017, pp. 1–3. https://doi.org/10.1109/ICIIECS.2017.8275999

  75. Sargano, A.B., Angelov, P., and Habib, Z., A comprehensive review on handcrafted and learning-based action representation approaches for human activity recognition, applied sciences, Appl. Sci. (Basel), 2017, vol. 7, no. 1, p. 110. https://doi.org/10.3390/app7010110

    Article  Google Scholar 

  76. Lussier, M., Lavoie, M., Giroux, S., Consel, C., Guay, M., Macoir, J., Hudon, C., Lorrain, D., Talbot, L., Langlois, F., et al., Early detection of mild cognitive impairment with in-home monitoring sensor technologies using functional measures: A systematic review, IEEE J. Biomed. Health Inform., 2018, vol. 23, no. 2, pp. 838–847. https://doi.org/10.1109/JBHI.2018.2834317

    Article  Google Scholar 

  77. Dhiman, C. and Vishwakarma, D.K., A review of state-of-theart techniques for abnormal human activity recognition, Eng. Appl. Artif. Intell., 2019, vol. 77, pp. 21–45. https://doi.org/10.1016/j.engappai.2018.08.014

    Article  Google Scholar 

  78. Fahim, M. and Sillitti, A., Anomaly detection, analysis and prediction techniques in IoT environment: a systematic literature review, IEEE Access, 2019, vol. 7, pp. 81664–81681. https://doi.org/10.1109/ACCESS.2019.2921912

    Article  Google Scholar 

  79. Iguernaissi, R., Merad, D., Aziz, K., and Drap, P., People tracking in multi-camera systems: A review, Multimedia Tools Appl., 2019, vol. 78, no. 8, pp. 10773–10793. https://doi.org/10.1007/s11042-018-6638-5

    Article  Google Scholar 

  80. Lentzas, A. and Vrakas, D., Non-intrusive human activity recognition and abnormal behavior detection on elderly people: A review, Artif. Intell. Rev., 2020, vol. 53, pp. 1975–2021.

    Article  Google Scholar 

  81. Nigam, S., Singh, R., and Misra, A., A review of computational approaches for human behavior detection, Arch. Comput. Methods Eng., 2019, vol. 26, no. 4, pp. 831–863. https://doi.org/10.1007/s11831-018-9270-7

    Article  Google Scholar 

  82. Sikandar, T., Ghazali, K.H., and Rabbi, M.F., ATM crime detection using image processing integrated video surveillance: A systematic review, Multimedia Syst., 2019, vol. 25, no. 3, pp. 229–251. https://doi.org/10.1007/s00530-018-0599-4

    Article  Google Scholar 

  83. Singh, T. and Vishwakarma, D.K., Video benchmarks of human action datasets: A review, Artif. Intell. Rev., 2019, vol. 52, no. 2, pp. 1107–1154. https://doi.org/10.1007/s10462-018-9651-1

    Article  Google Scholar 

  84. Tripathi, R.K., Jalal, A.S., and Agrawal, S.C., Abandoned or removed object detection from visual surveillance: A review, Multimedia Tools Appl., 2019, vol. 78, no. 6, pp. 7585–7620. https://doi.org/10.1007/s11042-018-6472-9

    Article  Google Scholar 

  85. Wang, J., Chen, Y., Hao, S., Peng, X., and Hu, L., Deep learning for sensor-based activity recognition: A survey, Pattern Recognit. Lett., 2019, vol. 119, pp. 3–11. https://doi.org/10.1016/j.patrec.2018.02.010

    Article  Google Scholar 

  86. Bakar, U., Ghayvat, H., Hasanm, S., and Mukhopadhyay, S., Activity and anomaly detection in smart home: a survey, in Next Generation Sensors and Systems, Springer, 2016, pp. 191–220. https://doi.org/10.1007/978-3-319-21671-3_9

    Book  Google Scholar 

  87. Zhang, C. and Jia, Q.-S., A review of occupant behavior models in residential building: Sensing, modeling, and prediction, Proc. IEEE Chinese Control and Decision Conf. (CCDC), Yinchuan, 2016, pp. 2032–2037. https://doi.org/10.1109/CCDC.2016.7531318

  88. Al-Shamayleh, A.S., Ahmad, R., Abushariah, M.A., Alam, K.A., and Jomhari, N., A systematic literature review on vision based gesture recognition techniques, Multimedia Tools Appl., 2018, vol. 77, no. 21, pp. 28121–28184. https://doi.org/10.1007/s11042-018-5971-z

    Article  Google Scholar 

  89. Mabrouk, A.B. and Zagrouba, E., Abnormal behavior recognition for intelligent video surveillance systems: A review, Expert Syst. Appl., 2018, vol. 91, pp. 480–491. https://doi.org/10.1016/j.eswa.2017.09.029

    Article  Google Scholar 

  90. Sreenu, G. and Durai, M.S., Intelligent video surveillance: A review through deep learning techniques for crowd analysis, J. Big Data, 2019, vol. 6, no. 1, p. 48. https://doi.org/10.1186/s40537-019-0212-5

    Article  Google Scholar 

  91. Al-Zoubi, S.M. and Younes, M.A.B., Low academic achievement: Causes and results, Theory Pract. Lang. Stud., 2015, vol. 5, no. 11, pp. 2262–2268. https://doi.org/10.17507/tpls.0511.09

    Article  Google Scholar 

  92. Kim, H., Lee, S., Kim, Y., Lee, S., Lee, D., Ju, J., and Myung, H., Weighted joint-based human behavior recognition algorithm using only depth information for low-cost intelligent video-surveillance system, Expert Syst. Appl., 2016, vol. 45, pp. 131–141. Available at: https://www.sciencedirect.com/science/article/abs/pii/ S0957417415006648.

    Article  Google Scholar 

  93. Mandryk, R.L. and Atkins, M.S., A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies, Int. J. Hum. Comput. Stud., 2007, vol. 65, no. 4, pp. 329–347. https://doi.org/10.1016/j.ijhcs.2006.11.011

    Article  Google Scholar 

  94. Sidney, K.D., Craig, S.D., Gholson, B., Franklin, S., Picard, R., and Graesser, A.C., Integrating affect sensors in an intelligent tutoring system, Proc. Affective Interactions: The Computer in the Affective Loop Workshop, 2005, pp. 7–13.

  95. Su, M.C., Cheng, C.T., Chang, M.C., and Hsieh, Y.Z., A video analytic in-class student concentration monitoring system, IEEE Trans. Consum. Electron., 2021, vol. 67, no. 4, pp. 294–304.

    Article  Google Scholar 

  96. Admoni, H. and Scassellati, B., Social eye gaze in humanrobot interaction: A review, J. Hum. Robot Interact., 2017, vol. 6, no. 1, pp. 25–63. https://doi.org/10.5898/JHRI.6.1.Admoni

    Article  Google Scholar 

  97. Jokinen, K., Furukawa, H., Nishida, M., and Yamamoto, S., Gaze and turn-taking behavior in casual conversational interactions, ACM Trans. Interact. Intell. Syst., 2013, vol. 3, no. 2, pp. 1–30.

    Article  Google Scholar 

  98. Andrist, S., Mutlu, B., and Tapus, A., Look like me: matching robot personality via gaze to increase motivation, Proc. 33rd Annu. ACM Conf. on Human Factors in Computing Systems, Seoul, 2015, pp. 3603–3612. https://doi.org/10.1145/2702123.2702592

  99. Ishi, C.T., Liu, C., Ishiguro, H., and Hagita, N., Head motion during dialogue speech and nod timing control in humanoid robots, Proc. 5th ACM/IEEE Int. Conf. on Human-Robot Interaction (HRI), Osaka, 2010, pp. 293–300. https://doi.org/10.1109/HRI.2010.5453183

  100. Otsuka, K., Takemae, Y., and Yamato, J., A probabilistic inference of multiparty-conversation structure based on Markovswitching models of gaze patterns, head directions, and utterances, Proc. 7th Int. Conf. on Multimodal Interfaces, Torento, 2005, pp. 191–198. https://doi.org/10.1145/1088463.1088497

  101. Huang, C.-M. and Mutlu, B., Learning-based modeling of multimodal behaviors for humanlike robots, Proc. 9th ACM/IEEE Int. Conf. on Human-Robot Interaction (HRI), Bielefeld, 2014, pp. 57–64. https://doi.org/10.1145/2559636.2559668

  102. Admoni, H., Dragan, A., Srinivasa, S.S., and Scassellati, B., Deliberate delays during robot-to-human handovers improve compliance with gaze communication, Proc. ACM/IEEE Int. Conf. on Human-Robot Interaction, Bielefeld, 2014, pp. 49–56. https://doi.org/10.1145/2559636.2559682

  103. Rich, C., Ponsler, B., Holroyd, A., and Sidner, C.L., Recognizing engagement in human-robot interaction, Proc. 5th ACM/IEEE Int. Conf. on Human-Robot Interaction (HRI), Osaka, 2010, pp. 375–382. https://doi.org/10.1109/HRI.2010.5453163

  104. Sakita, K., Ogawara, K., Murakami, S., Kawamura, K., and Ikeuchi, K., Flexible cooperation between human and robot by interpreting human intention from gaze information, Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Sendai, 2004, vol. 1, pp. 846–851. https://doi.org/10.1109/IROS.2004.1389458

  105. Andrist, S., Mutlu, B., and Gleicher, M., Conversational gaze aversion for virtual agents, in Proc. Int. Workshop on Intelligent Virtual Agents, Springer, 2013, pp. 249–262. https://doi.org/10.1007/978-3-642-40415-3_22.

  106. Nie, X. B., Xiong, C., and Zhu, S.-C., Joint action recognition and pose estimation from video, Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Boston, MA, 2015, pp. 1293–1301.

  107. Luvizon, D.C., Picard, D., and Tabia, H., 2D/3D pose estimation and action recognition using multitask deep learning, Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, 2018, pp. 5137–5146. arXiv:1802.09232v2

  108. Gao, C., Ye, S., Tian, H., and Yan, Y., Multi-scale single-stage pose detection with adaptive sample training in the classroom scene, Knowl. Base. Syst., 2021, vol. 222, p. 107008.

  109. Yang, Y. and Ramanan, D., Articulated human detection with flexible mixtures of parts, IEEE Trans. Pattern Anal. Mach. Intell., 2012, vol. 35, no. 12, pp. 2878–2890. https://doi.org/10.1109/TPAMI.2012.261

    Article  Google Scholar 

  110. Cherian, A., Mairal, J., Alahari, K., and Schmid, C., Mixing body-part sequences for human pose estimation, Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Columbus, OH, 2014, pp. 2353–2360. https://doi.org/10.1109/CVPR.2014.302.

  111. Kang, J., Han, X., Song, J., Niu, Z., and Li, X., The identification of children with autism spectrum disorder by SVM approach on EEG and eye-tracking data, Comput. Biol. Med., 2020, vol. 120, p. 103722.

  112. Valtakari, N.V., Hooge, I.T., Viktorsson, C., Nyström, P., Falck-Ytter, T., and Hessels, R.S., Eye tracking in human interaction: possibilities and limitations, Behav. Res. Methods, 2021, vol. 53, pp. 1592–1608.

    Article  Google Scholar 

  113. Yaneva, V., Eraslan, S., Yesilada, Y., and Mitkov, R., Detecting high-functioning autism in adults using eye tracking and machine learning, IEEE Trans. Neural Syst. Rehabil. Eng., 2020, vol. 28, no. 6, pp. 1254–1261.

    Article  Google Scholar 

  114. Yaneva, V., Ha, L.A., Eraslan, S., Yesilada, Y., and Mitkov, R., Detecting autism based on eye-tracking data from web searching tasks, Proc. Internet of Accessible Things, Halifax, 2018, pp. 1–10. https://doi.org/10.1145/3192714.3192819

    Book  Google Scholar 

  115. Jaiswal, S., Valstar, M.F., Gillott, A., and Daley, D., Automatic detection of ADHD and ASD from expressive behaviour in RGBD data, Proc. 12th IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG 2017), Washington, 2017, pp. 762–769.

  116. Gao, R., Deng, K., and Xie, M., Deep learning-assisted ADHD diagnosis, Proc. 3rd Int. Symp. on Artificial Intelligence for Medicine Sciences, Beijing, 2020, pp. 142–147.

  117. Singh, J., and Goyal, G., Decoding depressive disorder using computer vision, Multimedia Tools Appl., 2021, vol. 80, pp. 8189–8212.

  118. Hernndez-Vela, A., Reyes, M., Igual, L., Moya, J., Violant, V., and Escalera, S., Adhd indicators modelling based on dynamic time warping from rgbd data: a feasibility study, Proc. 6th CVC Workshop on the Progress of Research & Development, Barcelona: Computer Vision Center, Citeseer, 2011, pp. 59–62.

  119. Rautaray, S.S. and Agrawal, A., Vision based hand gesture recognition for human computer interaction: A survey, Artif. Intell. Rev., 2015, vol. 43, no. 1, pp. 1–54. https://doi.org/10.1007/s10462-012-9356-9

    Article  Google Scholar 

  120. Hsu, R.C., Su, P.C., Hsu, J.L., and Wang, C.Y., Real-time interaction system of human-robot with hand gestures, Proc. IEEE Eurasia Conf. on IOT, Communication and Engineering (ECICE), Yunlin, 2020, pp. 396–398.

  121. Binh, H.T., Trung, N.Q., Nguyen, H.-A.T., and Duy, B.T., Detecting student engagement in classrooms for intelligent tutoring systems, Proc. 23rd IEEE Int. Computer Science and Engineering Conf. (ICSEC), Phuket, 2019, pp. 145–149. https://doi.org/10.1109/ICSEC47112.2019.8974739

  122. Fang, C.-Y., Kuo, M.-H., Lee, G.-C., and Chen, S.-W., Student gesture recognition system in classroom 2.0, Proc. 14th IASTED Int. Conf. on Computers and Advanced Technology in Education, CATE 2011, Cambridge, 2011, pp. 290–297. https://doi.org/10.2316/P.2011.734-010

  123. Nazare, T.S. and Ponti, M., Hand-raising gesture detection with Lienhart-Maydt method in videoconference and distance learning, in Proc. Iberoamerican Congress on Pattern Recognition, Springer, 2013, pp. 512–519. https://doi.org/10.1007/978-3-642-41827-3_64.

  124. Hariharan, B., Padmini, S., and Gopalakrishnan, U., Gesture recognition using kinect in a virtual classroom environment, Proc. 4th IEEE Int. Conf. on Digital Information and Communication Technology and its Applications (DICTAP), Banff, 2014, pp. 118–124. https://doi.org/10.1109/DICTAP.2014.6821668

  125. Salous, S., Newton, J., Leroy, L., and Chendeb, S., Gestural recognition by a four kinect. module in a CAVE S “Le SAS,” in RoCHI, 2015, pp. 111–114.

  126. Kapgate, S., Sahu, P., Das, M., and Gupta, D., Human following robot using kinect in embedded platform, Proc. 1st IEEE Int. Conf. on the Paradigm Shifts in Communication, Embedded Systems, Machine Learning and Signal Processing (PCEMS), Nagpur, 2022, pp. 119–123.

  127. Corera, S. and Krishnarajah, N., Capturing hand gesture movement: A survey on tools, techniques and logical considerations, Proc. Chi Sparks, 2011.

    Google Scholar 

  128. Shanthakumar, V.A., Peng, C., Hansberger, J., Cao, L., Meacham, S., and Blakely, V., Design and evaluation of a hand gesture recognition approach for real-time interactions, Multimedia Tools Appl., 2020, vol. 79, pp. 17707–17730.

    Article  Google Scholar 

  129. Tsai, T.H., Huang, C.C., and Zhang, K.L., Design of hand gesture recognition system for human-computer interaction, Multimedia Tools Appl., 2020, vol. 79, pp. 5989–6007.

    Article  Google Scholar 

  130. Nguyen, K.H., US Patent 6 072 494, 2000.

  131. Cote, M., Payeur, P., and Comeau, G., Comparative study of adaptive segmentation techniques for gesture analysis in unconstrained environments, Proc. IEEE Int. Workshop on Imaging Systems and Techniques (IST 2006), Minori, 2006, pp. 28–33. https://doi.org/10.1109/IST.2006.1650770

  132. Köpüklü, O., Gunduz, A., Kose, N., and Rigoll, G., Real-time hand gesture detection and classification using convolutional neural networks, Proc. 14th IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG 2019), Lille, 2019, pp. 1–8.

  133. Liu, D., Zhang, L., Luo, T., Tao, L., and Wu, Y., Towards interpretable and robust hand detection via pixel-wise prediction, Pattern Recognit., 2020, vol. 105, p. 107202.

  134. Sun, Z., Chen, J., Mukherjee, M., Liang, C., Ruan, W., and Pan, Z., Online multiple object tracking based on fusing global and partial features, Neurocomputing, 2022, vol. 470, pp. 190–203.

    Article  Google Scholar 

  135. Huang, L., Zhang, B., Guo, Z., Xiao, Y., Cao, Z., and Yuan, J., Survey on depth and RGB image-based 3D hand shape and pose estimation, Virtual Reality and Intellig, Hardware, 2021, vol. 3, no. 3, pp. 207–234.

    Google Scholar 

  136. Song, T., Zhao, H., Liu, Z., Liu, H., Hu, Y., and Sun, D., Intelligent human hand gesture recognition by local-global fusing quality-aware features, Future Gener. Comput. Syst., 2021, vol. 115, pp. 298–303.

    Article  Google Scholar 

  137. Song, T., Zhao, H., Liu, Z., Liu, H., Hu, Y., and Sun, D., Intelligent human hand gesture recognition by local-global fusing quality-aware features, Future Gener. Comput. Syst., 2021, vol. 115, pp. 298–303.

    Article  Google Scholar 

  138. Dang, T.L., Tran, S.D., Nguyen, T.H., Kim, S., and Monet, N., An improved hand gesture recognition system using keypoints and hand bounding boxes, Array (N. Y.), 2022, vol. 16, no. 3, p. 100251.

  139. Aloysius, N. and Geetha, M., Understanding vision-based continuous sign language recognition, Multimedia Tools Appl., 2020, vol. 79, nos. 31–32, pp. 22177–22209.

    Article  Google Scholar 

  140. Amin, S.U., Alsulaiman, M., Muhammad, G., Mekhtiche, M.A., and Hossain, M.S., Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion, Future Gener. Comput. Syst., 2019, vol. 101, pp. 542–554.

    Article  Google Scholar 

  141. Li, Z., Lin, D., and Tang, X., Nonparametric discriminant analysis for face recognition, IEEE Trans. Pattern Anal. Mach. Intell., 2009, vol. 31, no. 4, pp. 755–761. https://doi.org/10.1109/TPAMI.2008.174

    Article  Google Scholar 

  142. Ren, Y. and Zhang, F., Hand gesture recognition based on MEB-SVM, Proc. IEEE Int. Conf. on Embedded Software and Systems, Hangzhou, 2009, pp. 344–349. https://doi.org/10.1109/ICESS.2009.21

  143. Afroze, S. and Hoque, M.M., Towards lip motion based speaking mode detection using residual neural networks, in Proc. Int. Conf. on Soft Computing and Pattern Recognition, Cham: Springer Int. Publ., 2020, pp. 166–175.

  144. Afroze, S. and Hoque, M.M., Talking vs non-talking: a vision based approach to detect human speaking mode, Proc. IEEE Int. Conf. on Electrical, Computer and Communication Engineering (ECCE), Cox’sBazar, 2019, pp. 1–6.

  145. Wuerkaixi, A., Zhang, Y., Duan, Z., and Zhang, C., Rethinking audio-visual synchronization for active speaker detection, Proc. 32nd IEEE Int. Workshop on Machine Learning for Signal Processing (MLSP), Xi’an, 2022, pp. 01–06.

  146. Haider, F. and Al Moubayed, S., Towards speaker detection using lips movements for human machine multiparty dialogue, Proc. 25th Swedish Phonetics Conf. (FONETIK), Stockholm, 2012, vol. 117.

  147. Lie, W.-N. and Hsieh, H.-C., Lips detection by morphological image processing, Proc. 4th IEEE Int. Conf. on Signal Processing ICSP’98, Beijing, 1998, vol. 2, pp. 1084–1087.

  148. Bendris, M., Charlet, D., and Chollet, G., Lip activity detection for talking faces classification in TV-Content, Proc. Int. Conf. on Machine Vision, Kaifeng, 2010, pp. 187–190.

  149. Khan, I., Abdullah, H., and Zainal, M.S.B., Efficient eyes and mouth detection algorithm using combination of Viola Jones and skin color pixel detection, Int. J. Eng., 2013, vol. 3, no. 4, p. 8269.

    Google Scholar 

  150. Huang, H.-Y. and Lin, Y.-C., An efficient mouth detection based on face localization and edge projection, Int. J. Comput. Theory Eng., 2013, vol. 5, no. 3, p. 514.

    Article  Google Scholar 

  151. Azim, T., Jaffar, M.A., and Mirza, A.M., Fully automated real time fatigue detection of drivers through fuzzy expert systems, Appl. Soft Comput., 2014, vol. 18, pp. 25–38.

    Article  Google Scholar 

  152. Navarathna, R., Lucey, P., Dean, D., Fookes, C., and Sridharan, S., Lip detection for audio-visual speech recognition in-car environment, Proc.10th IEEE Int. Conf. on Information Science, Signal Processing and their Applications (ISSPA 2010), Kuala Lumpur, 2010, pp. 598–601.

  153. Eveno, N., Caplier, A., and Coulon, P.-Y., Accurate and quasiautomatic lip tracking, IEEE Trans. Circ. Syst. Video Tech., 2004, vol. 14, no. 5, pp. 706–715.

    Article  Google Scholar 

  154. Bouvier, C., Benoit, A., Caplier, A., and Coulon, P.-Y., Open or closed mouth state detection: static supervised classification based on log-polar signature, in Proc. Int. Conf. on Advanced Concepts for Intelligent Vision Systems, Springer, 2008, pp. 1093–1102.

  155. Saenko, K., Livescu, K., Siracusa, M., Wilson, K., Glass, J., and Darrell, T., Visual speech recognition with loosely synchronized feature streams, Proc. 10th IEEE Int. Conf. on Computer Vision (ICCV’05), Beijing, 2005, vols. 1, 2, pp. 1424–1431.

  156. Faraj, M.I. and Bigun, J., Person verification by lip-motion, Proc. IEEE Conf. on Computer Vision and Pattern Recognition Workshop (CVPRW’06), New York, 2006, p. 37.

  157. Polur, P.R. and Miller, G.E., Experiments with fast Fourier transform, linear predictive and cepstral coefficients in dysarthric speech recognition algorithms using hidden Markov model, IEEE Trans. Neural Syst. Rehabil. Eng., 2005, vol. 13, no. 4, pp. 558–561.

    Article  Google Scholar 

  158. Katila, J. and Raudaskoski, S., Interaction analysis as an embodied and interactive process: Multimodal, co-operative, and intercorporeal ways of seeing video data as complementary professional visions, Hum. Stud., 2020, vol. 43, no. 3, pp. 445–470.

    Article  Google Scholar 

  159. Chen, W., Knowledge-aware learning analytics for smart learning, Procedia Comput. Sci., 2019, vol. 159, pp. 1957–1965.

    Article  Google Scholar 

  160. Herdel, V., Kuzminykh, A., Hildebrandt, A., and Cauchard, J.R., Drone in love: Emotional perception of facial expressions on flying robots, Proc. CHI Conf. on Human Factors in Computing Systems, Yokohama, 2021, pp. 1–20.

  161. Kupper, Z., Ramseyer, F., Hoffmann, H., Kalbermatten, S., and Tschacher, W., Video-based quantification of body movementduring social interaction indicates the severity of negative symptoms in patients with schizophrenia, Schizophr. Res., 2010, vol. 121, no. 1–3, pp. 90–100.

    Article  Google Scholar 

  162. Kale, U., Levels of interaction and proximity: Content analysis of video-based classroom cases, Internet High. Educ., 2008, vol. 11, no. 2, pp. 119–128.

    Article  Google Scholar 

  163. Richmond, V.P., Mccroskey, J.C., and Mottet, T., Handbook of Instructional Communication: Rhetorical and Relational Perspectives, Routledge, 2015.

    Book  Google Scholar 

  164. Pérez, P., Roose, P., Cardinale, Y., Dalmau, M., Masson, D., and Couture, N., Mobile proxemic application development for smart environments, Proc. 18th Int. Conf. on Advances in Mobile Computing and Multimedia, Chang Mai, 2020, pp. 94–103.

  165. Kivrak, H., Cakmak, F., Kose, H., and Yavuz, S., Social navigation framework for assistive robots in human inhabited unknown environments, Int. J. Eng. Sci. Technol., 2021, vol. 24, no. 2, pp. 284–298.

    Google Scholar 

  166. Maniscalco, U., Storniolo, P., and Messina, A., Bidirectional multi-modal signs of checking human-robot engagement and interaction, Int. J. Soc. Robot., 2022, vol. 14, no. 5, pp. 1295–1309.

    Article  Google Scholar 

  167. Philpott, J.S., The relative contribution to meaning of verbal and nonverbal channels of communication: a meta-analysis, Unpublished Master’s Thesis, Lincoln: Univ. of Nebraska, 1983.

  168. Mehrabain, A., Some referants and measures of non-verbal behaviour, Behav. Res. Meth. Instrum., 1969, vol. 1, pp. 213–217.

    Google Scholar 

  169. Girolami, M., Mavilia, F., and Delmastro, F., Sensing social interactions through BLE beacons and commercial mobile devices, Pervasive Mobile Comput., 2020, vol. 67, p. 101198.

  170. Martínez-Maldonado, R., Yan, L., Deppeler, J., Phillips, M., and Gašević, D., Classroom analytics: Telling stories about learning spaces using sensor data, in Hybrid Learning Spaces, Cham: Springer Int. Publ., 2022, pp. 185–203.

    Google Scholar 

  171. Miller, P.W., Nonverbal Communication. What Research Says to the Teacher, National Association Education Publ., 1988.

    Google Scholar 

  172. Miller, P.W., Body Language in the Classroom, Tech.: Connect. Educ. Careers, 2005, vol. 80, no. 8, pp. 28–30.

    Google Scholar 

  173. Wang, Y., Lee, L.H., Braud, T., and Hui, P., Re-shaping Post-COVID-19 teaching and learning: a blueprint of virtual-physical blended classrooms in the metaverse era, Proc. 42nd IEEE Int. Conf. on Distributed Computing Systems Workshops (ICDCSW), Bologna, 2022, pp. 241–247.

  174. Driscoll, M.P., Psychology of Learning for Instruction, Needham, MA: Allyn and Bacon, 2000.

    Google Scholar 

  175. Dhelim, S., Ning, H., Farha, F., Chen, L., Atzori, L., and Daneshmand, M., IoT-enabled social relationships meet artificial social intelligence, IEEE Internet Things J., 2021, vol. 8, no. 24, pp. 17817–17828.

    Article  Google Scholar 

  176. Chin, C., Classroom interaction in science: Teacher questioning and feedback to students’ responses, Int. J. Sci. Educ., 2006, vol. 28, no. 11, pp. 1315–1346.

    Article  Google Scholar 

  177. Reigeluth, C.M. and Moore, J., Cognitive education and the cognitive domain, in Instructional-Design Theories and Models: A New Paradigm of Instructional Theory, Lawrence Erlbaum Associates, 1999, pp. 51–68.

    Google Scholar 

  178. Teräs, M., Suoranta, J., Teräs, H., and Curcher, M., Post-Covid-19 education and education technology “solutionism”: a seller’s market, Postdigital Sci. Educ., 2020, vol. 2, no. 3, pp. 863–878.

    Article  Google Scholar 

  179. Fredricks, J.A., Blumenfeld, P.C., and Paris, A.H., School engagement: Potential of the concept, state of the evidence, Rev. Educ. Res., 2004, vol. 74, no. 1, pp. 59–109.

    Article  Google Scholar 

  180. Nigam, A., Pasricha, R., Singh, T., and Churi, P., A systematic review on AI-based proctoring systems: Past, present, and future, Educ. Inf. Technol., 2021, vol. 26, no. 5, pp. 6421–6445.

    Article  Google Scholar 

  181. Silvola, A., Naykki, P., Kaveri, A., and Muukkonen, H., Expectations for supporting student engagement with learning analytics: An academic path perspective, Comput. Educ., 2021, vol. 168, p. 104192.

  182. Reeve, J. and Tseng, C.-M., Agency as a fourth aspect of students’ engagement during learning activities, Contemp. Educ. Psychol., 2011, vol. 36, no. 4, pp. 257–267.

    Article  Google Scholar 

  183. Mach, K.J., Lemos, M.C., Meadow, A.M., Wyborn, C., Klenk, N., Arnott, J.C., and Wong-Parodi, G., Actionable knowledge and the art of engagement, Curr. Opin. Environ. Sustain., 2020, vol. 42, pp. 30–37.

    Article  Google Scholar 

  184. Koedinger, K.R., Anderson, J.R., Hadley, W.H., and Mark, M.A., Int. J. Artif. Intell. Educ., 1997, vol. 8, pp. 30–43.

    Google Scholar 

  185. Guo, L., Wang, D., Gu, F., Li, Y., Wang, Y., and Zhou, R., Evolution and trends in intelligent tutoring systems research: A multidisciplinary and scientometric view, Asia Pac. Educ. Rev., 2021, vol. 22, no. 3, pp. 441–461.

    Article  Google Scholar 

  186. D’mello, S.K. and Graesser, A., Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features, User Model. User-adapt. Interact., 2010, vol. 20, no. 2, pp. 147–187.

    Article  Google Scholar 

  187. Behera, A., Matthew, P., Keidel, A., Vangorp, P., Fang, H., and Canning, S., Associating facial expressions and upper-body gestures with learning tasks for enhancing intelligent tutoring systems, Int. J. Artif. Intell. Educ., 2020, vol. 30, pp. 236–270.

    Article  Google Scholar 

  188. Joseph, E., Engagement tracing: using response times to model student disengagement, Artif. Intellig. Educ.: Support. Learn. Intellig. Soc. Inf. Technol., 2005, vol. 125, p. 88.

    Google Scholar 

  189. Li, S., Lajoie, S.P., Zheng, J., Wu, H., and Cheng, H., Automated detection of cognitive engagement to inform the art of staying engaged in problem-solving, Comput. Educ., 2021, vol. 163, p. 104114.

  190. Chaouachi, M., Pierre, C., Jraidi, I., and Frasson, C., Affect and mental engagement: Towards adaptability for intelligent, Proc. 23rd Int. FLAIRS Conf., Daytona Beach, FL, 2010.

  191. Goldberg, B.S., Sottilare, R.A., and Brawner, K.W., andHolden, H.K., Predicting learner engagement during welldefined and ill-defined computer-based intercultural interac tions, in Proc. Int. Conf. on Affective Computing and Intelligent Interaction, Springer, 2011, pp. 538–547.

  192. Xiao, X. and Wang, J., Undertanding and detecting divided attention in mobile mooc learning, Proc. CHI Conf. on Human Factors in Computing Systems, Denver, 2017, pp. 2411–2415.

  193. Alqahtani, F., Katsigiannis, S., and Ramzan, N., Using wearable physiological sensors for affect-aware intelligent tutoring systems, IEEE Sens. J., 2020, vol. 21, no. 3, pp. 3366–3378.

    Article  Google Scholar 

  194. Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., and Movellan, J.R., The faces of engagement: Automatic recognition of student engagementfrom facial expressions, IEEE Trans. Affect. Comput., 2014, vol. 5, no. 1, pp. 86–98.

    Article  Google Scholar 

  195. Beal, C.R., Walles, R., Arroyo, I., and Woolf, B.P., On-line tutoring for math achievement testing: A controlled evaluation, J. Interact. Online Learn., 2007, vol. 6, no. 1, pp. 43–55.

    Google Scholar 

  196. Kaur, A., Mustafa, A., Mehta, L., and Dhall, A., Prediction and localization of student engagement in the wild, Proc. IEEE Conf. on Digital Image Computing: Techniques and Applications (DICTA), Canberra, 2018, pp. 1–8.

  197. Baltrušaitis, T., Robinson, P., and Morency, L.-P., Openface: An open-source facial behavior analysis toolkit, Proc. IEEE Winter Conf. on Applications of Computer Vision (WACV), Lake Placid, NY, 2016, pp. 1–10.

  198. Zhu, B., Lan, X., Guo, X., Barner, K.E., and Boncelet, C., Multi-rate attention based gru model for engagement prediction, Proc. Int. Conf. on Multimodal Interaction, Utrecht, 2020, pp. 841–848.

  199. Li, Y.Y. and Hung, Y.P., Feature fusion of face and body for engagement intensity detection, Proc. IEEE Int. Conf. on Image Processing (ICIP), Taipei, 2019, pp. 3312–3316.

  200. Thong Huynh, V., Kim, S.H., Lee, G.S., and Yang, H.J., Engagement intensity prediction withfacial behavior features, Proc. Int. Conf. on Multimodal Interaction, Suzhou, 2019, pp. 567–571.

  201. Demochkina, P. and Savchenko, A., Efficient algorithms for video-based engagement prediction for a MOOC course, Proc. IEEE Int. Russian Automation Conf. (RusAutoCon), Sochi, 2022, pp. 672–676.

  202. Wu, S., Simulation of classroom student behavior recognition based on PSO-kNN algorithm and emotional image processing, J. Intell. Fuzzy Syst., 2021, vol. 40, no. 4, pp. 7273–7283.

    Article  Google Scholar 

  203. Chakraborty, S., Mondal, R., Singh, P.K., Sarkar, R., and Bhattacharjee, D., Transfer learning with fine tuning for human action recognition from still images, Multimedia Tools Appl., 2021, vol. 80, pp. 20547–20578.

    Article  Google Scholar 

  204. Nadeem, A., Jalal, A., and Kim, K., Automatic human posture estimation for sport activity recognition with robust body parts detection and entropy markov model, Multimedia Tools Appl., 2021, vol. 80, pp. 21465–21498.

    Article  Google Scholar 

  205. Akhter, I., Jalal, A., and Kim, K., Pose estimation and detection for event recognition using Sense-Aware features and Adaboost classifier, Proc. IEEE Int. Bhurban Conf. on Applied Sciences and Technologies (IBCAST), Islamabad, 2021, pp. 500–505.

  206. Irvine, N., Nugent, C., Zhang, S., Wang, H., and Ng, W.W., Neural network ensembles for sensor-based human activity recognition within smart environments, Sensors (Basel), 2019, vol. 20, no. 1, p. 216.

    Article  Google Scholar 

  207. Ghadi, Y.Y., Akhter, I., Alsuhibany, S.A., al Shloul, T., Jalal, A., and Kim, K., Multiple events detection using context-intelligence features, Intellig. Automat. Soft Comput., 2022, vol. 34, no. 3.

  208. Mohmed, G., Lotfi, A., and Pourabdollah, A., Employing a deep convolutional neural network for human activity recognition based on binary ambient sensor data, Proc. 13th ACM Int. Conf. on Pervasive Technologies Related to Assistive Environments, Corfu, 2020, pp. 1–7.

  209. Ahmad, Z. and Khan, N.M., Multidomain multimodal fusion for human action recognition using inertial sensors, Proc. 5th IEEE Int. Conf. on Multimedia Big Data (BigMM), Singapore, 2019, pp. 429–434.

  210. Wang, M., Yan, Z., Wang, T., Cai, P., Gao, S., Zeng, Y., and Chen, X., Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors, Nat. Electron., 2020, vol. 3, no. 9, pp. 563–570.

    Article  Google Scholar 

  211. Dewan, M., Murshed, M., and Lin, F., Engagement detection in online learning: A review, Smart Learn. Environ., 2019, vol. 6, no. 1, pp. 1–20.

    Article  Google Scholar 

  212. Du, Y., Crespo, R.G., and Martinez, O.S., Human emotion recognition for enhanced performance evaluation in e-learning, Progr. Artif. Intellig., 2023, vol. 12, no. 2, pp. 199–211.

    Article  Google Scholar 

  213. Wang, L. and Yoon, K.J., Knowledge distillation and student-teacher learning for visual intelligence: a review and new outlooks, IEEE Trans. Pattern Anal. Mach. Intell., 2021, vol. 44, no. 6, pp. 3048–3068.

    Article  Google Scholar 

  214. Baran, E. and AlZoubi, D., Human-centered design as a frame for transition to remote teaching during the COVID-19 pandemic, J. Technol. Teach. Educ., 2020, vol. 28, no. 2, pp. 365–372.

    Google Scholar 

  215. Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X., Fsd50k: an open dataset of human-labeled sound events, IEEE/ACM Trans. Audio Speech Lang. Process., 2021, vol. 30, pp. 829–852.

    Article  Google Scholar 

  216. Zheng, R., Jiang, F., and Shen, R., Intelligent student behavior analysis system for real classrooms, Proc. ICASSP IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 9244–9248.

  217. Pabba, C. and Kumar, P., An intelligent system for monitoring students’ engagement in large classroom teaching through facial expression recognition, Expert Syst., 2022, vol. 39, no. 1, p. e12839.

  218. Ovur, S.E., Su, H., Qi, W., De Momi, E., and Ferrigno, G., Novel adaptive sensor fusion methodology for hand pose estimation with multileap motion, IEEE Trans. Instrum. Meas., 2021, vol. 70, pp. 1–8.

    Article  Google Scholar 

  219. Xu, J., Yu, Z., Ni, B., Yang, J., Yang, X., and Zhang, W., Deep kinematics analysis for monocular 3d human pose estimation, Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2020, pp. 899–908.

  220. Goldberg, P., Sümer, Ö., Stürmer, K., Wagner, W., Göllner, R., Gerjets, P., and Trautwein, U., Attentive or not? Toward a machine learning approach to assessing students’ visible engagement in classroom instruction, Educ. Psychol. Rev., 2021, vol. 33, pp. 27–49.

    Article  Google Scholar 

  221. Shou, T., Borchers, C., Karumbaiah, S., and Aleven, V., Optimizing parameters for accurate position data mining in diverse classrooms layouts, Proc. 16th Int. Conf. on Educational Data Mining, Bengaluru, 2023.

  222. Bigalke, A., Hansen, L., Diesel, J., Hennigs, C., Rostalski, P., and Heinrich, M.P., Anatomy-guided domain adaptation for 3D in-bed human pose estimation, Med. Image Anal., 2023, vol. 89, p. 102887.

  223. Luo, C., Zhang, J., Yu, J., Chen, C.W., and Wang, S., Real-time head pose estimation and face modeling from a depth image, IEEE Trans. Multimed., 2019, vol. 21, no. 10, pp. 2473–2481.

    Article  Google Scholar 

  224. Abowd, G.D., Atkeson, C.G., Feinstein, A., Hmelo, C., Kooper, R., Long, S., Sawhney, N., and Tani, M., Teaching and learning as multimedia authoring: the classroom 2000 project, Proc. 4th ACM Int. Conf. on Multimedia, Boston, MA, 1997, pp. 187–198.

  225. Zhang, L. and Lin, S., Research on the design and application of intelligence classroom teaching model with rain classroom digital support, in Proc. Int. Conf. on Modern Educational Technology and Innovation and Entrepreneurship (ICMETIE 2020), Atlantis Press, 2020, pp. 368–373.

  226. Mady, M.A. and Baadel, S., Technology-Enabled Learning (TEL): YouTube as a ubiquitous learning aid, J. Inf. Knowledge Manag., 2020, vol. 19, no. 01, p. 2040007.

  227. Augusto, J.C., Ambient intelligence: Opportunities and consequences of its use in smart classrooms, Innov. Teach. Learn. Inf. Comput. Sci., 2009, vol. 8, no. 2, pp. 53–63.

    Google Scholar 

  228. Abdellatif, I., Towards a novel approach for designing smart classrooms, Proc. IEEE 2nd Int. Conf. on Information and Computer Technologies (ICICT), Kahului, 2019, pp. 280–284.

  229. Jaiswal, S., Parmar, A., Singh, H., and Rathee, G., Smart Classroom Automation, Jaypee Univ. of Information Technology, 2018.

    Google Scholar 

  230. Basilaia, G. and Kvavadze, D., Transition to online education in schools during a SARS-CoV-2 coronavirus (COVID-19) pandemic in Georgia, Pedagogical Res., 2020, vol. 5, no. 4, pp. 1–9.

    Article  Google Scholar 

  231. Magnani, A., Human action recognition and monitoring in ambient assisted living environments, PhD Thesis, Alma, 2020.

  232. Cebrian, G., Palau, R., and Mogas, J., The Smart Classroom as a means to the development of ESD methodologies, Sustainability, 2020, vol. 12, no. 7, p. 3010.

    Article  Google Scholar 

  233. Chen, L., Hoey, J., Nugent, C.D., Cook, D.J., and Yu, Z., Sensor-based activity recognition, IEEE Trans. Syst. Man Cybern. C, 2012, vol. 42, no. 6, pp. 790–808.

    Article  Google Scholar 

  234. Logan, B., Healey, J., Philipose, M., Tapia, E.M., and Intille, S., A long-term evaluation of sensing modalities for activity recognition, in Proc. Int. Conf. on Ubiquitous Computing, Springer, 2007, pp. 483–500.

  235. Tapia, E.M., Intille, S.S., Haskell, W., Larson, K., Wright, J., King, A., and Friedman, R., Real-time recognition of physical activities and their intensities using wireless accelerometers and a heart rate monitor, Proc. 11th IEEE Int. Symp. on Wearable Computers, Boston, MA, 2007, pp. 37–40.

  236. Stikic, M., Huynh, T., Van Laerhoven, K., and Schiele, B., ADL recognition based on the combination of RFID and accelerometer sensing, Proc. 2nd IEEE Int. Conf. on Pervasive Computing Technologies for Healthcare, Tampere, 2008, pp. 258–263.

  237. Roy, N., Misra, A., and Cook, D., Ambient and smartphone sensor assisted ADL recognition in multi-inhabitant smart environments, J. Ambient Intell. Humaniz. Comput., 2016, vol. 7, no. 1, pp. 1–19.

    Article  Google Scholar 

  238. Diethe, T., Twomey, N., Kull, M., Flach, P., and Craddock, I., Probabilistic sensor fusion for ambient assisted living, 2017. arXiv:1702.01209.

  239. Hermanis, A., Cacurs, R., Nesenbergs, K., Greitans, M., Syundyukov, E., and Selavo, L., Wearable sensor system for human biomechanics monitoring, Proc. Int. Conf. on Embedded Wireless Systems and Networks, Graz, 2016, pp. 247–248.

  240. Jung, S., Hong, S., Kim, J., Lee, S., Hyeon, T., Lee, M., and Kim, D.-H., Wearable fall detector using integrated sensors and energy devices, Sci. Rep., 2015, vol. 5, p. 17081.

    Article  Google Scholar 

  241. Um, T.T., Babakeshizadeh, V., and Kulíc, D., Exercise motion classification from large-scale wearable sensor data using convolutional neural networks, Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Vancouver, 2017, pp. 2385–2390.

  242. Wang, Z., He, S.Y., and Leung, Y., Applying mobile phone data to travel behaviour research: A literature review, Travel Behav. Soc., 2018, vol. 11, pp. 141–155.

    Article  Google Scholar 

  243. Dimitriadou, E. and Lanitis, A., A critical evaluation, challenges, and future perspectives of using artificial intelligence and emerging technologies in smart classrooms, Smart Learn. Environ., 2023, vol. 10, no. 1, pp. 1–26.

    Article  Google Scholar 

  244. Kwet, M. and Prinsloo, P., The “smart” classroom: A new frontier in the age of the smart university, Teach. High. Educ., 2020, vol. 25, no. 4, pp. 510–526.

    Article  Google Scholar 

  245. Chen, L. and Nugent, C.D., Human Activity Recognition and Behavior Analysis, Springer Int. Publ., 2019.

    Book  Google Scholar 

  246. Candamo, J., Shreve, M., Goldgof, D.B., Sapper, D.B., and Kasturi, R., Understanding transit scenes: A survey on human behavior-recognition algorithms, IEEE Trans. Intell. Transp. Syst., 2009, vol. 11, no. 1, pp. 206–224.

    Article  Google Scholar 

Download references

Funding

This work was supported by ongoing institutional funding. No additional grants to carry out or direct this particular research were obtained.

Author information

Authors and Affiliations

  1. Facultad de Estadística e Informática, Universidad Veracruzana, Xalapa, Veracruz, México

    M. L. Córdoba-Tlaxcalteco & E. Benítez-Guerrero

Authors
  1. M. L. Córdoba-Tlaxcalteco
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. E. Benítez-Guerrero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to M. L. Córdoba-Tlaxcalteco or E. Benítez-Guerrero.

Ethics declarations

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s Note.

Pleiades Publishing remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Córdoba-Tlaxcalteco, M.L., Benítez-Guerrero, E. Human Event Recognition in Smart Classrooms Using Computer Vision: A Systematic Literature Review. Program Comput Soft 49, 625–642 (2023). https://doi.org/10.1134/S0361768823080066

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0361768823080066

Keywords:

Search

Navigation