Abstract
This study mainly examined the role of the combination of three senses (i.e., auditory, visual, and tactile) and five senses (i.e., auditory, visual, tactile, olfactory, and gustatory) in the correlation between electrophysiological and electrodermal responses underlying second language (L2) sentence comprehension. Forty subjects did two acceptability judgment tasks, encompassing congruent and semantically/pragmatically incongruent sentences. The event-related potential (ERP) and galvanic skin response (GSR) data for both the target and final words of the sentences were collected and analyzed. The results revealed that there is an interaction between cognitive and emotional responses in both semantically and pragmatically incongruent sentences, yet the timing of the interaction is longer in sentences with pragmatic incongruity due to their complexity. Based on the ERP and GSR correlation results, it was further found that the five-sense combination approach improves L2 sentence comprehension and interest in learning materials yet reduces the level of excitement or arousal. While this approach might be beneficial for some learners, it might be detrimental for those in favor of stimulating learning environments.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akbari, M., & Pishghadam, R. (2022). Developing new software to analyze the emo-sensory load of language. Journal of Business Communication & Technology, 1(1), 1–13. https://doi.org/10.56632/bct.2022.1101
Allan, D. (1992). The Oxford quick placement test. Oxford University Press.
Bahari, N., & Dost Mohammadi, Y. (2023). Multisensory learning and vocabulary retention: An emotioncy-based study with the Stroop task. Journal of Cognition Emotion & Education, 1(1), 58–76.
Bardi, A., Guerra, V. M., & Ramdeny, G. S. D. (2009). Openness and ambiguity intolerance: Their differential relations to well-being in the context of an academic life transition. Personality and Individual Differences, 47(3), 219–223. https://doi.org/10.1016/j.paid.2009.03.003
Bauer, A. R., Debener, S., & Nobre, A. C. (2020). Synchronisation of neural oscillations and cross-modal influences. Trends in Cognitive Sciences, 24(6), 481–495. https://doi.org/10.1016/j.tics.2020.03.003
Beckage, N. M., & Colunga, E. (2016). Language networks as models of cognition: Understanding cognition through language. In A. Mehler, A. Lücking, S. Banisch, P. Blanchard, & B. Job (Eds.), Towards a theoretical framework for analyzing complex linguistic networks (pp. 3–28). Springer. https://doi.org/10.1007/978-3-662-47238-5_1
Borsipour, B. (2016). Emotioncy and willingness to read: A case of Iranian EFL learners. Master’s thesis. Ferdowsi University of Mashhad, Mashhad, Iran.
Boustani, N. (2023). How does the brain support theory of mind in children? An EEG/ERP review. Journal of Cognition Emotion & Education, 1(1), 17–28. https://doi.org/10.22034/cee.2023.171194
Boustani, N., Pishghadam, R., & Shayesteh, S. (2021). Multisensory input modulates P200 and L2 sentence comprehension: A 1-week consolidation phase. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2021.746813
Buhr, K., & Dugas, M. J. (2002). The intolerance of uncertainty scale: Psychometric properties of the English version. Behaviour Research and Therapy, 40(8), 931–945. https://doi.org/10.1016/S0005-7967(01)00092-4
DeLong, K. A., Urbach, T. P., Groppe, D. M., & Kutas, M. (2011). Overlapping dual ERP responses to low cloze probability sentence continuations. Psychophysiology, 48(9), 1203–1207. https://doi.org/10.1111/j.1469-8986.2011.01199.x
Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron, 57(1), 11–23. https://doi.org/10.1016/j.neuron.2007.12.013
Dubovi, I. (2022). Cognitive and emotional engagement while learning with VR: The perspective of multimodal methodology. Computers and Education, 183, 104495. https://doi.org/10.1016/j.compedu.2022.104495
Dudai, Y. (2004). The neurobiology of consolidations, or, how stable is the Engram? Annual Review of Psychology, 55, 51–86. https://doi.org/10.1146/annurev.psych.55.090902.142050
Ebrahimi, S., Tabatabaeian, M., & Al Abdwani, T. (2022). Enhancing the communicative skills of normal and mentally-challenged learners through emo-sensory textbooks. Journal of Business Communication & Technology, 1(2), 1–12. https://doi.org/10.56632/bct.2022.1201
Ezzyat, Y., Inhoff, M. C., & Davachi, L. (2018). Differentiation of human medial prefrontal cortex activity underlies long-term resistance to forgetting in memory. Journal of Neuroscience, 38(48), 10244–10254. https://doi.org/10.1523/JNEUROSCI.2290-17.2018
Faghihi, U., Poirier, P., & Larue, O. (2011). Emotional cognitive architectures. In S. D’Mello, A. Graesser, B. Schuller, & J. C. Martin (Eds.), Affective computing and intelligent interaction (pp. 487–496). Springer. https://doi.org/10.1007/978-3-642-24600-5_52
Federmeier, K. D., Wlotko, E. W., De Ochoa-Dewald, E., & Kutas, M. (2007). Multiple effects of sentential constraint on word processing. Brain Research, 1146, 75–84.
Golshaie, R., Mokhtari, B., & Haghbin, F. (2021). A psycholinguistic study of the effect of grammatical aspect on the interpretation of ambiguous subject pronouns and discourse relations in Persian. Journal of Language Researches, 12(1), 193–218. https://doi.org/10.22059/jolr.2021.314515.666673
Greenspan, S. I. (1992). Infancy and early childhood: The practice of clinical assessment and intervention with emotional and developmental challenges. International Universities Press.
Greenspan, S. I., & Benderly, B. L. (1997). The growth of the mind and the endangered origins of intelligence. Perseus Publishing.
Greenspan, S. I., & Shanker, S. G. (2004). The first idea: How symbols, language and intelligence evolved from our primate ancestors to modern humans. Da Capo Press.
Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304(5669), 438–441. https://doi.org/10.1126/science.1095455
Hald, L. A., Bastiaansen, M. C., & Hagoort, P. (2006). EEG theta and gamma responses to semantic violations in online sentence processing. Brain and Language, 96(1), 90–105. https://doi.org/10.1016/j.bandl.2005.06.007
Harris, C. L. (2004). Bilingual speakers in the lab: Psychophysiological measures of emotional reactivity. Journal of Multilingual and Multicultural Development, 25(2–3), 223–247. https://doi.org/10.1080/01434630408666530
Herbert, C., Junghofer, M., & Kissler, J. (2008). Event related potentials to emotional adjectives during reading. Psychophysiology, 45(3), 487–498. https://doi.org/10.1111/j.1469-8986.2007.00638.x
Hess, U., Kappas, A., McHugo, G. J., Lanzetta, J. T., & Kleck, R. E. (1992). The facilitative effect of facial expression on the self-generation of emotion. International Journal of Psychophysiology, 12, 251–265. https://doi.org/10.1016/0167-8760(92)90064-I
Holler, J., & Levinson, S. C. (2019). Multimodal language processing in human communication. Trends in Cognitive Sciences, 23(8), 639–652. https://doi.org/10.1016/j.tics.2019.05.006
Huang, Y. (2017). Introduction: What is pragmatics? In Y. Huang (Ed.), The Oxford handbook of pragmatics (pp. 1–18). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199697960.013.33
Jankowiak, K., & Korpal, P. (2018). On modality effects in bilingual emotional language processing: Evidence from galvanic skin response. Journal of Psycholinguistic Research, 47(3), 663–677. https://doi.org/10.1007/s10936-017-9552-5
Kos, M., Van den Brink, D., & Hagoort, P. (2012). Individual variation in the late positive complex to semantic anomalies. Frontiers in Psychology, 3, 318. https://doi.org/10.3389/fpsyg.2012.00318
Kutas, M., & Federmeier, K. D. (2000). Electrophysiology reveals semantic memory use in language comprehension. Trends in Cognitive Sciences, 4(12), 463–470. https://doi.org/10.1016/S1364-6613(00)01560-6
Kutas, M., & Federmeier, K. D. (2011). Thirty years and counting: Finding meaning in the N400 component of the event-related brain potential (ERP). Annual Review of Psychology, 62, 621–647. https://doi.org/10.1146/annurev.psych.093008.131123
Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(4427), 203–205. https://doi.org/10.1126/science.7350657
Lim, C. L., Barry, R. J., Gordon, E., Sawant, A., Rennie, C., & Yiannikas, C. (1996). The relationship between quantified EEG and skin conductance level. International Journal of Psychophysiology, 21(2–3), 151–162. https://doi.org/10.1016/0167-8760(95)00049-6
Löbner, S. (2013). Understanding semantics. Routledge.
Marinis, T. (2010). Using on-line processing methods in language acquisition research. In S. Unsworth & E. Blom (Eds.), Experimental methods in language acquisition research (pp. 139–162). John Benjamins.
McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3), 419–457. https://doi.org/10.1037/0033-295X.102.3.419
Mehler, B., Reimer, B., & Coughlin, J. F. (2012). Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task: An on-road study across three age groups. Human Factors: The Journal of Human Factors and Ergonomics Society, 54(3), 396–412.
Mertens, G., & Morriss, J. (2021). Intolerance of uncertainty and threat reversal: A conceptual replication of Morriss et al. (2019). Behaviour Research and Therapy, 137, 103799.
Myréen, S. (2017). Evaluating the role of multisensory elements in foreign language acquisition. International Journal of Educational and Pedagogical Sciences, 11(2), 375–378.
Northoff, G., Heinzel, A., De Greck, M., Bermpohl, F., Dobrowolny, H., & Panksepp, J. (2006). Self-referential processing in our brain-a meta-analysis of imaging studies on the self. NeuroImage, 31(1), 440–457. https://doi.org/10.1016/j.neuroimage.2005.12.002
Nourbakhsh, N., Wang, Y., Chen, F., & Calvo, R. A. (2012). Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks. In Proceedings of the 24th Australian computer–human interaction conference (pp. 420–423). OZCHI. https://doi.org/10.1145/2414536.2414602
Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia, 9(1), 97–113. https://doi.org/10.1016/0028-3932(71)90067-4
Paul, T., Bhattacharyya, C., Sen, P., Prasad, R., & Shaw, S. (2020). Human emotion recognition using GSR and EEG. International Journal of Scientific and Research Publications, 10(5), 394–400. https://doi.org/10.29322/IJSRP.10.05.2020.p10146
Payne, B., & Federmeier, R. (2018). Contextual constraints on lexico-semantic processing in aging: Evidence from single‐word event‐related brain potentials. Brain Research, 1687, 117–128. https://doi.org/10.1016/j.brainres.2018.02.021
Payne, B. R., Stites, M. C., & Federmeier, K. D. (2019). Event-related brain potentials reveal how multiple aspects of semantic processing unfold across parafoveal and foveal vision during sentence reading. Psychophysiology, 56(9), e13432. https://doi.org/10.1111/psyp.13432
Pishghadam, R. (2016). Emotioncy, extraversion, and anxiety in willingness to communicate in English. In W. A. Lokman, F. M. Fazidah, I. Salahuddin, & I. A. W. Mohd (Eds.), Proceedings of the 5th international conference on language, education, and innovation (pp. 1–5). Infobase Creation Sdn Bhd.
Pishghadam, R., Adamson, B., & Shayesteh, S. (2013). Emotion-based language instruction (EBLI) as a new perspective in bilingual education. Multilingual Education, 3(9), 1–16. https://doi.org/10.1186/2191-5059-3-9
Pishghadam, R., Daneshvarfard, F., & Shayesteh, S. (2021). Oscillatory neuronal dynamics during L2 sentence comprehension: The effects of sensory enrichment and semantic incongruency. Language Cognition and Neuroscience. https://doi.org/10.1080/23273798.2021.1886312
Pishghadam, R., Ebrahimi, S., & Derakhshan, A. (2020). Cultuling analysis: A new methodology for discovering cultural memes. International Journal of Society Culture & Language, 8(2), 17–34.
Pishghadam, R., Ebrahimi, S., Miri, M. A., & Shayesteh, S. (2021). Sapioemotionality as a new attribute in socio-cultural studies. International Journal of Society Culture & Language, 9(3), 16–27.
Pishghadam, R., Jajarmi, H., & Shayesteh, S. (2016). Conceptualizing sensory relativism in light of emotioncy: A movement beyond linguistic relativism. International Journal of Society Culture & Language, 4(2), 11–21.
Pishghadam, R., Jajarmi, H., & Shayesteh, S. (2021). Sense combinations influence the neural mechanism of L2 comprehension in semantically violated sentences: Insights from emotioncy. Journal of Neurolinguistics, 58, 100962. https://doi.org/10.1016/j.jneuroling.2020.100962
Pop-Jordanova, N., & Pop-Jordanov, J. (2020). Electrodermal activity and stress assessment. Prilozi, 41(2), 5–15. https://doi.org/10.2478/prilozi-2020-0028
Rugg, M. D., & Curran, T. (2007). Event-related potentials and recognition memory. Trends in Cognitive Sciences, 11(6), 251–257. https://doi.org/10.1016/j.tics.2007.04.004
Rushby, J. A., & Barry, R. J. (2009). Single-trial event-related potentials to significant stimuli. International Journal of Psychophysiology, 74(2), 120–131. https://doi.org/10.1016/j.ijpsycho.2009.08.003
Sanford, A. J., Leuthold, H., Bohan, J., & Sanford, A. J. (2011). Anomalies at the borderline of awareness: An ERP study. Journal of Cognitive Neuroscience, 23(3), 514–523. https://doi.org/10.1162/jocn.2009.2137
Severens, E., & Hartsuiker, R. J. (2009). Is there a lexical bias effect in comprehension monitoring? Language and Cognitive Processes, 24(6), 910–927. https://doi.org/10.1080/01690960902775517
Sharma, M., Sharma, S., & Kacker, M. (2016). A brief introduction and review on galvanic skin response. International Journal of Medical Research Professionals, 2(6), 13–17. https://doi.org/10.21276/ijmrp.2016.2.6.003
Shayesteh, S., Pishghadam, R., & Khodaverdi, A. (2020). FN400 and LPC responses to different degrees of sensory involvement: A study of sentence comprehension. Advances in Cognitive Psychology, 16(1), 45–58.
Siever, D., & Collura, T. (2017). Audio-visual entrainment: Physiological mechanisms and clinical outcomes. In J. R. Evans & R. P. Turner (Eds.), Rhythmic stimulation procedures in neuromodulation (pp. 51–95). Academic Press. https://doi.org/10.1016/B978-0-12-803726-3.00003-1
Stuss, D. T., Picton, T. W., & Cerri, A. M. (1988). Electrophysiological manifestations of typicality judgment. Brain and Language, 33(2), 260–272.
Szewczyk, J. M., & Schriefers, H. (2011). Is animacy special? ERP correlates of semantic violations and animacy violations in sentence processing. Brain Research, 1368, 208–221. https://doi.org/10.1016/j.brainres.2010.10.070
Tabatabaee Farani, S., Pishghadam, R., & Moghimi, S. (2020). Sensory emotion in words: Evidence from an ERP study in light of the emotioncy model. Basic and Clinical Neuroscience. https://doi.org/10.32598/bcn.2021.1870.1
Tanner, D., Grey, S., & van Hell, J. G. (2017). Dissociating retrieval interference and reanalysis in the P600 during sentence comprehension. Psychophysiology, 54(2), 248–259. https://doi.org/10.1111/psyp.12788
Tomasello, R., Kim, C., Dreyer, F. R., Grisoni, L., & Pulvermüller, F. (2019). Neurophysiological evidence for rapid processing of verbal and gestural information in understanding communicative actions. Scientific Reports, 9(1), 1–17. https://doi.org/10.1038/s41598-019-52158-w
Tsai, J. L., Levenson, R. W., & Carstensen, L. L. (2000). Autonomic, subjective and expressive responses to emotional films in older and younger Chinese americans and European Americans. Cultural Diversity and Ethnic Minority Psychology, 15, 684–693. https://doi.org/10.1037/0882-7974.15.4.684
Tyng, C. M., Amin, H. U., Saad, M. N., & Malik, A. S. (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8, 1454. https://doi.org/10.3389/fpsyg.2017.01454
Van Berkum, J. J., Brown, C. M., Zwitserlood, P., Kooijman, V., & Hagoort, P. (2005). Anticipating upcoming words in discourse: Evidence from ERPs and reading times. Journal of Experimental Psychology: Learning Memory and Cognition, 31(3), 443–467. https://doi.org/10.1037/0278-7393.31.3.443
Van De Meerendonk, N., Kolk, H. H., Vissers, C. T. W., & Chwilla, D. J. (2010). Monitoring in language perception: Mild and strong conflicts elicit different ERP patterns. Journal of Cognitive Neuroscience, 22(1), 67–82. https://doi.org/10.1162/jocn.2008.21170
Van Petten, C., & Luka, B. J. (2012). Prediction during language comprehension: Benefits, costs, and ERP components. International Journal of Psychophysiology, 83(2), 176–190. https://doi.org/10.1016/j.ijpsycho.2011.09.015
Villarejo, M. V., Zapirain, B. G., & Zorrilla, A. M. (2012). A stress sensor based on galvanic skin response (GSR) controlled by Zigbee. Sensors, 12(5), 6075–6101. https://doi.org/10.3390/s120506075
Wechsler, D. (1981). Wechsler adult intelligence scale: WAIS-R manual. Harcourt Brace Jovanovich [for] the Psychological Corporation.
Wirebring, L. K., Wiklund-Hörnqvist, C., Eriksson, J., Andersson, M., Jonsson, B., & Nyberg, L. (2015). Lesser neural pattern similarity across repeated tests is associated with better long-term memory retention. Journal of Neuroscience, 35(26), 9595–9602. https://doi.org/10.1523/JNEUROSCI.3550-14.2015
Zhang, D., He, W., Wang, T., Luo, W., Zhu, X., Gu, R., Hi, H., & Luo, Y. (2014). Three stages of emotional word processing: An ERP study with rapid serial visual presentation. Social Cognitive and Affective Neuroscience, 9(12), 1897–1903. https://doi.org/10.1093/scan/nst188
Zimmer, H. (2002). Habituation and recovery of a slow negative wave of the event-related brain potential. International Journal of Psychophysiology, 43(3), 225–235. https://doi.org/10.1016/S0167-8760(01)00185-4
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original article was revised due to an error in authors list.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Pishghadam, R., Shayesteh, S., Daneshvarfard, F. et al. Cognition-Emotion Interaction during L2 Sentence Comprehension: The Correlation of ERP and GSR Responses to Sense Combinations. J Psycholinguist Res 53, 7 (2024). https://doi.org/10.1007/s10936-024-10039-y
Accepted:
Published:
DOI: https://doi.org/10.1007/s10936-024-10039-y