Abstract
Emotion recognition is vital for augmenting human–computer interactions by integrating emotional contextual information for enhanced communication. Hence, the study presents an intelligent emotion detection system developed utilizing hybrid stacked gated recurrent units (GRU)-recurrent neural network (RNN) deep learning architecture. Integration of GRU with RNN allows the system to make use of both models’ capabilities, making it better at capturing complex emotional patterns and temporal correlations. The EEG signals are investigated in time, frequency, and time–frequency domains, meticulously curated to capture intricate multi-domain patterns. Then, the SMOTE-Tomek method ensures a uniform class distribution, while the PCA technique optimizes features by minimizing data redundancy. A comprehensive experimentation including the well-established emotion datasets: DEAP and AMIGOS, assesses the efficacy of the hybrid stacked GRU and RNN architecture in contrast to 1D convolution neural network, RNN and GRU models. Moreover, the “Hyperopt” technique fine-tunes the model’s hyperparameter, improving the average accuracy by about 3.73%. Hence, results revealed that the hybrid GRU-RNN model demonstrates the most optimal performance with the highest classification accuracies of 99.77% ± 0.13, 99.54% ± 0.16, 99.82% ± 0.14, and 99.68% ± 0.13 for the 3D VAD and liking parameter, respectively. Furthermore, the model’s generalizability is examined using the cross-subject and database analysis on the DEAP and AMIGOS datasets, exhibiting a classification with an average accuracy of about 99.75% ± 0.10 and 99.97% ± 0.03. Obtained results when compared with the existing methods in literature demonstrate superior performance, highlighting potential in emotion recognition.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The data that support the findings of this study are available from Koelstra et al. (2012), but restrictions apply to the availability of these data, which were used under licence for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of authors of the DEAP dataset. Weblink: https://www.eecs.qmul.ac.uk/mmv/datasets/deap/
References
Arnau-González P, Arevalillo-Herráez M, Ramzan N (2017) Fusing highly dimensional energy and connectivity features to identify affective states from EEG signals. Neurocomputing 244:81–89. https://doi.org/10.1016/j.neucom.2017.03.027
Bergstra J, Yamins D, Cox D (2013) Hyperopt: a python library for optimizing the hyperparameters of machine learning algorithms. In: Proceedings of the 12th python in science conference
Bernico M (2018) Deep learning quick reference useful hacks for training and optimizing deep neural networks with TensorFlow and Keras
Chen JX, Zhang PW, Mao ZJ, Huang YF, Jiang DM, Zhang YN (2019) Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access 7:44317–44328. https://doi.org/10.1109/ACCESS.2019.2908285
Cimtay Y, Ekmekcioglu E (2020) Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset eeg emotion recognition. Sensors 20:2034. https://doi.org/10.3390/s20072034
Combrisson E, Jerbi K (2015) Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods 250:5454. https://doi.org/10.1016/j.jneumeth.2015.01.010
Cui H, Liu A, Zhang X, Chen X, Wang K, Chen X (2020) EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl Based Syst 205:106243. https://doi.org/10.1016/j.knosys.2020.106243
Cui S, Yin Y, Wang D, Li Z, Wang Y (2021) A stacking-based ensemble learning method for earthquake casualty prediction. Appl Soft Comput 101:107038. https://doi.org/10.1016/j.asoc.2020.107038
Dastider AG, Sadik F, Fattah SA (2021) An integrated autoencoder-based hybrid CNN-LSTM model for COVID-19 severity prediction from lung ultrasound. Comput Biol Med 132:104296. https://doi.org/10.1016/j.compbiomed.2021.104296
Dey R, Salem FM (2017) Gate-variants of gated recurrent unit (GRU) Neural Networks
Du X, Ma C, Zhang G, Li J, Lai YK, Zhao G, Deng X, Liu YJ, Wang H (2020) An efficient LSTM network for emotion recognition from multichannel EEG signals. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.3013711
Faraji P, Khodabakhshi MB (2023) CollectiveNet-AltSpec: a collective concurrent CNN architecture of alternate specifications for EEG media perception and emotion tracing aided by multi-domain feature-augmentation. Neural Netw 167:502–516. https://doi.org/10.1016/j.neunet.2023.08.031
Gabriel KR (1971) The biplot graphic display of matrices with application to principal component analysis. Biometrika 58:453–467. https://doi.org/10.1093/biomet/58.3.453
Gannouni S, Aledaily A, Belwafi K, Aboalsamh H (2020) Adaptive emotion detection using the valence-arousal-dominance model and EEG brain rhythmic activity changes in relevant brain lobes. IEEE Access 8:67444–67455. https://doi.org/10.1109/ACCESS.2020.2986504
Gao Z, Wang X, Yang Y, Li Y, Ma K, Chen G (2021) A channel-fused dense convolutional network for EEG-based emotion recognition. IEEE Trans Cogn Dev Syst 13:945–954. https://doi.org/10.1109/TCDS.2020.2976112
Gu T, Wang Z, Xu X, Li D, Yang H, Du W (2022) Frame-level teacher-student learning with data privacy for EEG emotion recognition. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2022.3168935
Houssein EH, Hammad A, Ali AA (2022) Human emotion recognition from EEG-based brain–computer interface using machine learning: a comprehensive review. Neural Comput Appl 34:12527–12557
Huang D, Chen S, Liu C, Zheng L, Tian Z, Jiang D (2021) Differences first in asymmetric brain: a bi-hemisphere discrepancy convolutional neural network for EEG emotion recognition. Neurocomputing 448:140–151. https://doi.org/10.1016/j.neucom.2021.03.105
Islam MR, Moni MA, Islam MM, Rashed-Al-Mahfuz M, Islam MS, Hasan MK, Hossain MS, Ahmad M, Uddin S, Azad A, Alyami SA, Ahad MAR, Lio P (2021) Emotion recognition from EEG signal focusing on deep learning and shallow learning techniques. IEEE Access 9:94601–94624. https://doi.org/10.1109/ACCESS.2021.3091487
Iyer A, Das SS, Teotia R, Maheshwari S, Sharma RR (2022) CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-12310-7
Jaouedi N, Boujnah N, Bouhlel MS (2020) A new hybrid deep learning model for human action recognition. J King Saud Univ Comput Inf Sci 32:447–453. https://doi.org/10.1016/j.jksuci.2019.09.004
Katsogiannis-Meimarakis G, Koutrika G (2021) A deep dive into deep learning approaches for Text-to-SQL systems. In: Proceedings of the ACM SIGMOD international conference on management of data
Khateeb M, Anwar SM, Alnowami MR (2021) Multi-domain feature fusion for emotion classification using DEAP dataset. IEEE Access. https://doi.org/10.1109/access.2021.3051281
Koelstra S, Mühl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput. https://doi.org/10.1109/t-affc.2011.15
Li R, Ren C, Li C, Zhao N, Lu D, Zhang X (2022) SSTD: a novel spatio-temporal demographic network for EEG-based emotion recognition. IEEE Trans Comput Soc Syst. https://doi.org/10.1109/TCSS.2022.3188891
Liang Z, Oba S, Ishii S (2019) An unsupervised EEG decoding system for human emotion recognition. Neural Netw 116:257–268. https://doi.org/10.1016/j.neunet.2019.04.003
Meng M, Hu J, Gao Y, Kong W, Luo Z (2022) A deep subdomain associate adaptation network for cross-session and cross-subject EEG emotion recognition. Biomed Signal Process Control 78:103873. https://doi.org/10.1016/j.bspc.2022.103873
Miranda-Correa JA, Abadi MK, Sebe N, Patras I (2021) AMIGOS: a dataset for affect, personality and mood research on individuals and groups. IEEE Trans Affect Comput 12:479–493. https://doi.org/10.1109/TAFFC.2018.2884461
Nandini D, Yadav J, Rani A, Singh V (2023a) Design of subject independent 3D VAD emotion detection system using EEG signals and machine learning algorithms. Biomed Signal Process Control 85:104894. https://doi.org/10.1016/j.bspc.2023.104894
Nandini D, Yadav J, Rani A, Singh V, Kravchenko OV, Rathee N (2023b) Improved patient-independent seizure detection using hybrid feature extraction approach with atomic function-based wavelets. Iran J Sci Technol Trans Electr Eng. https://doi.org/10.1007/s40998-023-00644-3
Pandey P, Seeja KR (2022a) Subject independent emotion recognition from EEG using VMD and deep learning. J King Saud Univ Comput Inf Sci 34:1730–1738. https://doi.org/10.1016/j.jksuci.2019.11.003
Pandey P, Seeja KR (2022b) A one-dimensional CNN model for subject independent emotion recognition using EEG signals. In: Khanna A, Gupta D, Bhattacharyya S, Hassanien AE, Anand S, Jaiswal A (eds) International Conference on Innovative Computing and Communications: Proceedings of ICICC 2021, Volume 2. Springer Singapore, Singapore, pp 509–515. https://doi.org/10.1007/978-981-16-2597-8_43
Prakeash KB, Kanagachidambaresan GR (2021) Pattern Recognition and Machine Learning. In: Prakash KB, Kanagachidambaresan GR (eds) Programming with TensorFlow Solution for Edge Computing Applications. Springer International Publishing, Cham, pp 105–144
Russell JA, Mehrabian A (1977) Evidence for a three-factor theory of emotions. J Res Person 11(3):273–294. https://doi.org/10.1016/0092-6566(77)90037-X
Sakalle A, Tomar P, Bhardwaj H, Acharya D, Bhardwaj A (2021) A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system. Expert Syst Appl 173:114516. https://doi.org/10.1016/j.eswa.2020.114516
Santamaria-Granados L, Munoz-Organero M, Ramirez-Gonzalez G, Abdulhay E, Arunkumar N (2019) Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS). IEEE Access 7:2883213. https://doi.org/10.1109/ACCESS.2018.2883213
Sharma R, Pachori RB, Sircar P (2020) Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomed Signal Process Control 58:101867. https://doi.org/10.1016/j.bspc.2020.101867
Song T, Zheng W, Song P, Cui Z (2020) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11:532–541. https://doi.org/10.1109/TAFFC.2018.2817622
Swana EF, Doorsamy W, Bokoro P (2022) Tomek link and SMOTE approaches for machine fault classification with an imbalanced dataset. Sensors 22:3246. https://doi.org/10.3390/s22093246
Tinker N, Yan W, Tinker NA (2015) Biplot analysis of multi-environment trial data: principles and applications. Canad J Plant Sci 6:623–645
Tuncer T, Dogan S, Subasi A (2022) LEDPatNet19: automated emotion recognition model based on nonlinear led pattern feature extraction function using EEG signals. Cogn Neurodyn. https://doi.org/10.1007/s11571-021-09748-0
Umer M, Imtiaz Z, Ullah S, Mehmood A, Choi GS, On BW (2020) Fake news stance detection using deep learning architecture (CNN-LSTM). IEEE Access 8:156695–156706. https://doi.org/10.1109/ACCESS.2020.3019735
Verma GK, Tiwary US (2017) Affect representation and recognition in 3D continuous valence–arousal–dominance space. Multimed Tools Appl 76:2159–2183. https://doi.org/10.1007/s11042-015-3119-y
Xiong Z, Cui Y, Liu Z, Zhao Y, Hu M, Hu J (2020) Evaluating explorative prediction power of machine learning algorithms for materials discovery using k-fold forward cross-validation. Comput Mater Sci 171:109203. https://doi.org/10.1016/j.commatsci.2019.109203
Xu G, Guo W, Wang Y (2023) Subject-independent EEG emotion recognition with hybrid spatio-temporal GRU-Conv architecture. Med Biol Eng Comput 61:456. https://doi.org/10.1007/s11517-022-02686-x
Zeng C, Ma C, Wang K, Cui Z (2022) Parking occupancy prediction method based on multi factors and stacked GRU-LSTM. IEEE Access 10:47361–47370. https://doi.org/10.1109/ACCESS.2022.3171330
Zhang Y, Cheng C, Zhang Y (2021) Multimodal emotion recognition using a hierarchical fusion convolutional neural network. IEEE Access 9:7943–7951. https://doi.org/10.1109/ACCESS.2021.3049516
Zhang A, Lipton ZC, Li M, Smola AJ (2020) Dive into deep learning release 0.14.3
Zhao J, Mao X, Chen L (2019) Speech emotion recognition using deep 1D and 2D CNN LSTM networks. Biomed Signal Process Control 47:312–323. https://doi.org/10.1016/j.bspc.2018.08.035
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Contributions
Durgesh Nandini formulated and investigated the problem, conceptualized the methodology, performed software analysis, and interpretation, and wrote the original manuscript. Dr. Jyoti Yadav formulated the problem, conceptualized the methodology, and formally analysed, interpreted, and supervised manuscript writing and reviewing. Prof. Asha Rani and Prof. Vijander Singh validated the work and formally analysed and supervised manuscript writing and reviewing.
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Nandini, D., Yadav, J., Rani, A. et al. Enhancing Emotion Detection with Non-invasive Multi-Channel EEG and Hybrid Deep Learning Architecture. Iran J Sci Technol Trans Electr Eng (2024). https://doi.org/10.1007/s40998-024-00710-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s40998-024-00710-4