M. Kalaiyarasi
Skip Abstract Section
Abstract
The goal of emotion detection is to find and recognise emotions in text, speech, gestures, facial expressions, and more. This paper proposes an effective multimodal emotion recognition system based on facial expressions, sentence-level text, and voice. Using public datasets, we examine face expression image classification and feature extraction. The Tri-modal fusion is used to integrate the findings and to provide the final emotion. The proposed method has been verified in classroom students, and the feelings correlate with their performance. This method categorizes students' expressions into seven emotions: happy, surprise, sad, fear, disgust, anger, and contempt. Compared to the unimodal models, the suggested multimodal network design may reach up to 65% accuracy. The proposed method can detect negative feelings such as boredom or loss of interest in the learning environment.
- Sharmeen, M., Saleem, A., Siddeeq, Y., Mohammed, A., Sadeeq, M., Subhi, R.M.: Multimodal Emotion Recognition using Deep Learning. Journal of Applied Science and Technology Trends. 02,52 –58 (2021).Google Scholar
- Xie, B., Sidulova, M., Park, C.H.: Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion. Sensors. 21, 4913(2021).Google ScholarCross Ref
- Perveen, N., Roy, D., Chalavadi, K. M.: Facial Expression Recognition in Videos Using Dynamic Kernels. IEEE Transactions on Image Processing. 29, 8316-8325(2020).Google ScholarCross Ref
- Ho, N.H., Yang, H.J., Kim, S.H., Lee, G.: Multimodal approach of speech emotion recognition using multi-level multihead fusion attention based recurrent neural network. IEEE Access. 8, 61672–61686 (2020).Google ScholarCross Ref
- Sumanth Ratna Kandavalli, Anthuvan Stephen Edberk, Dinesh Kumar Rajendran, Varthini Rajagopal; A Progressive Review on Wire Arc Additive Manufacturing: Mechanical Properties, Metallurgical and Defect Analysis, Advances in Additive Manufacturing Processes (2021) 1: 178. https://doi.org/10.2174/9789815036336121010014Google ScholarCross Ref
- Wu, M., Su, W., Chen, L., Pedrycz, W., Hirota, K.: Two-stage Fuzzy Fusion based-Convolution Neural Network for Dynamic Emotion Recognition. IEEE Transactions on Affective Computing. (2020).Google Scholar
- Mohan, K., Seal, A., Krejcar, O., Yazidi, A.: Facial Expression Recognition using Local Gravitational Force Descriptor based Deep Convolution Neural Networks. IEEE Transactions on Instrumentation and Measurement. (2020).Google Scholar
- Priyalakshmi, B., and Parikshit Verma. "Navigational tool for the blind." American Institute of Physics Conference Series. Vol. 2460. No. 1. 2022.Google Scholar
- Hu, L., Li, W., Yang, J., Fortino, G., Chen, M.: A Sustainable Multimodal Multi-layer Emotion-aware Service at the Edge. IEEE Transactions on Sustainable Computing. (2019).Google Scholar
- Li, J., Qiu, S., Du, C., Wang, Y., He, H.: Domain adaptation for EEG emotion recognition based on latent representation similarity. IEEE Trans. Cogn. Develop. Syst. 12, 344-353 (2020).Google ScholarCross Ref
- Bao, L.Q., Qiu, J.L., Tang, H., Zheng, W.L., Lu, B.L.: Investigating sex differences in classification of five emotions from EEG and eye movement signals. IEEE Eng. Med. Biol. Soc. (EMBC), Berlin, Germany. 6746-6749 (2019).Google Scholar
- Chen, D.W., Miao, R., Yang, W.Q., Liang, Y., Chen, H.H., Huang, L., Deng, C.J., Han, N., A feature extraction method based on differential entropy and linear discriminant analysis for emotion recognition. Sensors. 19, 1631-1647 (2019).Google ScholarCross Ref
- Huang, Y., Yang, J., Liu, S., Pan, J.: Combining facial expressions and electroencephalography to enhance emotion recognition. Future Internet. 11, 105-121 (2019).Google ScholarCross Ref
- Santamaria-Granados, L., Munoz-Organero, M., Ramirez-González, G., Abdulhay, E., Arunkumar, N.: Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS). IEEE Access. 7, 2169-3536 (2019).Google ScholarCross Ref
- Cimtay, Y., Ekmekcioglu, E., Caglar-Ozhan, S.: Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access. 8,168865-168878 (2020).Google ScholarCross Ref
- Chen, J., Lv, Y., Xu, R., Xu, C.: Automatic social signal analysis: Facial expression recognition using difference convolution neural network. Journal of Parallel and Distributed Computing. 131,97-102 (2019).Google ScholarDigital Library
- Mahmood, M.R., Abdulrazzaq, M.B., Zeebaree, S., Ibrahim, A.K., Zebari, R.R., Dino, H.I.: Classification techniques' performance evaluation for facial expression recognition. Indonesian Journal of Electrical Engineering and Computer Science. 21, 176-1184 (2021).Google Scholar
- Obaid, K. B., Zeebaree, S., Ahmed, O.M.: Deep Learning Models Based on Image Classification: A Review. International Journal of Science and Business. 4, 75-81 (2020).Google Scholar
- Varunkumar, K. A., Prabakaran, M., Kaurav, A., Chakkaravarthy, S. S., Thiyagarajan, S., & Venkatesh, P. (2014). Various Database Attacks and its Prevention Techniques. International Journal of Engineering Trends and Technology, 9(11), 532-536.Google Scholar
- Gorantla, Venkata Ashok K., et al. "An intelligent optimization framework to predict the vulnerable range of tumor cells using Internet of things." 2023 IEEE 2nd International Conference on Industrial Electronics: Developments & Applications (ICIDeA). IEEE, 2023.Google Scholar
- Domínguez-Jiménez, J.A., Campo-Landines, K.C., Martínez-Santos, J.C., Delahoz, E.J., Contreras-Ortiz, S.H.: A machine learning model for emotion recognition from physiological signals. Biomedical Signal Processing and Control. 55, 101646 (2019).Google ScholarCross Ref
- Priyadarshini, R., & Tamilselvan, L. (2018). Semantic clustering approach for documents in distributed system framework with multi-node setup. International Journal of Networking and Virtual Organisations, 19(2-4), 321-340.Google ScholarCross Ref
- Baijun, X., Mariia, S., Chung Hyuk, P.: Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion. Sensors. 21,4913 (2021).Google ScholarCross Ref
- Brett, M., Chloe, C.B., Susan, N.: Using Emotion as Information in Future-Oriented Cognition: Individual Differences in the Context of State Negative Affect. PersIndivid Dif. 121-126 (2016).Google Scholar
- Hanifah, H., Unayah, N.: Mencegah Dan MenanggulangiPenyalahgunaanNapza. Informasi. 33-46 (2011).Google Scholar
- Shumaila, K., Qaisara, P., Imran Yousuf, M., Ghafoor Chaudhry, A.: Effects of Depression on Students' Academic Performance. Sci.int.(Lahore). 27(2), (2015).Google Scholar
- Rachana Y. Patil, Yogesh H. Patil, Renu Kachhoria & Savita Lonare. A provably secure data sharing scheme for smart gas distribution grid using fog computing. International Journal of Information Technology (2022). https://doi.org/10.1007/s41870-022-01043-3Google ScholarCross Ref
- Mohamed, S., Sofia Jonathan, G.: Effect of facial expressions on student's comprehension recognition in virtual. SpringerPlus. 1-9 (2013).Google Scholar
- Peter, L., Hyman, H., Robert, R., John, S.: Adolescent Psychopathology: I. Prevalence and Incidence of Depression and Other DSM-III-R Disorders in High School Students. Journal of Abnormal Psychology. 102, 133-144 (1993).Google ScholarCross Ref
- Putra, W.B., Arifin, F.: Real-Time Emotion Recognition System to monitor Student's Mood in a Classroom. Journal of Physics: Conference Series. 1413, 012021(2019).Google ScholarCross Ref
Index Terms
- Student's Emotion Recognition using Multimodality and Deep Learning
Index terms have been assigned to the content through auto-classification.
Recommendations
Emotion Recognition in Conversations Using Brain and Physiological Signals
IUI '22: Proceedings of the 27th International Conference on Intelligent User InterfacesEmotions are complicated psycho-physiological processes that are related to numerous external and internal changes in the body. They play an essential role in human-human interaction and can be important for human-machine interfaces. Automatically ...
Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects
AbstractEmotion recognition has recently attracted extensive interest due to its significant applications to human–computer interaction. The expression of human emotion depends on various verbal and non-verbal languages like audio, visual, text, etc. ...
Deep learning based multimodal emotion recognition using model-level fusion of audio–visual modalities
AbstractEmotion identification based on multimodal data (e.g., audio, video, text, etc.) is one of the most demanding and important research fields, with various uses. In this context, this research work has conducted a rigorous exploration of ...
Highlights- Deep learning-based feature extractor networks for video and audio data are proposed.
Comments