Hostname: page-component-848d4c4894-m9kch Total loading time: 0 Render date: 2024-05-06T22:05:10.530Z Has data issue: false hasContentIssue false
  • English
  • Français

A review of physiological measures for mental workload assessment in aviation

A state-of-the-art review of mental workload physiological assessment methods in human-machine interaction analysis

Part of: The AJ Special Rotorcraft Collection

Published online by Cambridge University Press:  25 October 2023

G. Luzzani Open the ORCID record for G. Luzzani [Opens in a new window] ,
I. Buraioli Open the ORCID record for I. Buraioli [Opens in a new window] ,
D. Demarchi Open the ORCID record for D. Demarchi [Opens in a new window]  and
G. Guglieri Open the ORCID record for G. Guglieri [Opens in a new window]
G. Luzzani*
Affiliation:
Department of Mechanical and Aerospace Engineering, Politecnico di Torino, Turin, Italy
I. Buraioli
Affiliation:
Department of Electronics and Telecommunication, Politecnico di Torino, Turin, Italy
D. Demarchi
Affiliation:
Department of Electronics and Telecommunication, Politecnico di Torino, Turin, Italy
G. Guglieri
Affiliation:
Department of Mechanical and Aerospace Engineering, Politecnico di Torino, Turin, Italy
*
Corresponding author: G. Luzzani; Email: gabriele.luzzani@polito.it
Get access Rights & Permissions [Opens in a new window]

Abstract

The relevant growth of human-machine interaction (HMI) systems in recent years is leading to the necessity of being constantly aware of the cognitive workload level of an operator, especially in a safety-critical context such as aviation. Since the confusion in the definition of this concept, this paper clarifies this terminology and also highlights its relationship with stress. Thus, we analysed the state-of-the-art of cognitive workload evaluations, showing three up-to-date methodologies: subjective, behavioural and physiological. In particular, the physiological approach is increasingly gaining attention in the literature due to today’s exponential growth of biomedical sensors. Therefore, a review of the most adopted physiological signals in the workload evaluation is provided, focusing on the aeronautical field. We conclude by highlighting the necessity of a multimodal approach for mental workload assessment as a result of this analysis.

Keywords

Human factorsmental workloadstressphysiological signalshuman-machine interaction
Type
Survey Paper
Information
The Aeronautical Journal , Volume 128 , Issue 1323 , May 2024 , pp. 928 - 949
Check for updates
Copyright
© The Author(s), 2023. Published by Cambridge University Press on behalf of Royal Aeronautical Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Young, M.S., Brookhuis, C., Wickens, C.D. and Hancock, P.A. State of science: mental workload in ergonomics, Ergonomics, 2014, 58, (1), pp 1–17.CrossRefGoogle ScholarPubMed
Debie, E., Rojas, R.F., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S. and Garratt, M. Multimodal fusion for objective assessment of cognitive workload, IEEE Trans. Cybern., 2021, 51, (3), pp 1542–1555.CrossRefGoogle ScholarPubMed
De Filippis, L., Gaia, E., Guglieri, G., Re, M. and Ricco, C. Cognitive based design of a human machine interface for telenavigation of a space rover, J. Aerospace Technol. Manag., 2014, 6, pp 415–430.CrossRefGoogle Scholar
De Filippis, L., Guglieri, G., Ricco, C. and Sartori, D. Remote control station design and testing for tele-operated space-missions, Int. J. Aerospace Sci., 2013, 2, (3), pp 92–105.Google Scholar
ICAO, 2021 Safety Report, International Civil Aviation Organization, 2021.Google Scholar
Shappell, S., Detwiler, C., Holcomb, K., Hackworth, C., Boquet, A. and Wiegmann, D. Human Error and Commercial Aviation Accidents: A Comprehensive, Fine-Grained Analysis Using HFACS, 2006, Federal Aviation Administration.Google Scholar
Martins, A.P.G. A review of important cognitive concepts in aviation, Aviation, 2016, 20, (2), pp 65–84.CrossRefGoogle Scholar
Matessa, M., Strybel, T., Vu, K., Battiste, V. and Schnell, T. Concept of Operations for RCO SPO, 2017, NASA.Google Scholar
Honecker, F. and Schulte, A. Automated online determination of pilot activity under uncertainty by using evidential reasoning, in Engineering Psychology and Cognitive Ergonomics: Cognition and Design, vol. 10276, 2017, pp 231–250.CrossRefGoogle Scholar
Aerospace Technology Institute, The Single Pilot Commercial Aircraft, 2019, INSIGHT.Google Scholar
Hancock, G.M., Longo, L., Young, M.S. and Hancock, P.A. Mental workload, in Handbook of Human Factors and Ergonomics, 5th ed, vol. 7, John Wiley & Sons, Inc, 2021, pp. 203–226. CrossRefGoogle Scholar
Van Acker, B., Parmentier, D., Vlerick, P. and Saldien, J. Understanding mental workload: from a clarifying concept analysis toward an implementable framework, Cognit. Technol. Work, 2018, 20, pp. 351–365.CrossRefGoogle Scholar
Tao, D., Tan, H., Wang, H., Zhang, X., Qu, X. and Zhang, T. A systematic review of physiological measures of mental workload, Int. J. Environ. Res. Public Health, 2019, 16, (15), article number: 2716.CrossRefGoogle ScholarPubMed
Wickens, C.D. Multiple resources and performance prediction, Theoret. Issues Ergon. Sci., 2002, 3, (2), pp 159–177.CrossRefGoogle Scholar
McKendrick, R., Feest, B., Harwood, A. and Falcone, B. Theories and methods for labeling cognitive workload: classification and transfer learning, Front. Hum. Neurosci., 2019, 13, (259), pp 1–20.CrossRefGoogle ScholarPubMed
Fink, G. Stress: definition and history, in Stress Science: Neuroendocrinology, 2010, pp 3–9.Google Scholar
Hart, S.G. and Staveland, L.E. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research, in Human Mental Workload. Advances in Psychology, vol. 52, 1988, pp 139–183.CrossRefGoogle Scholar
Ayres, P., Lee, J.Y., Paas, F. and van Merriënboer, J. The validity of physiological measures to identify differences in intrinsic cognitive load, Front. Psychol., 2021, 12, pp 1–16.CrossRefGoogle ScholarPubMed
McKendrick, R.D. and Cherry, E. A deeper look at the NASA TLX and where it falls short, Proc. Hum. Factors Ergon. Soc. 2018 Ann. Meet., 2018, 62, (1), pp 44–48.Google Scholar
Cooper, G.H. and Harper, R.P. The Use of Pilot Rating in the Evaluation of Aircraft Handling Qualities, 1969, NASA.Google Scholar
Casali, J. and Wierwillie, W. A comparison of rating scale, secondary-task, physiological, and primary-task workload estimation techniques in a simulated flight task emphasizing communications load, Hum. Factors, 1983, 25, (6), pp 623–641.CrossRefGoogle Scholar
Brand, Y. and Schulte, A. Workload-adaptive and task-specific support for cockpit crews: design and evaluation of an adaptive associate system, Hum.-Intell. Syst. Integr., 2021, 3, pp 187–199.CrossRefGoogle Scholar
Brand, Y. and Schulte, A. Model-based prediction of workload for adaptive associate systems, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017.CrossRefGoogle Scholar
Wickens, C.D. Multiple resources and performance prediction, Theoret. Issues Ergon. Sci., 2002, 3, (2), pp 159–177.CrossRefGoogle Scholar
Wickens, C.D. Multiple resources and mental workload, Hum. Factors, 2008, 50, (3), pp 449–455.CrossRefGoogle ScholarPubMed
Horrey, W. and Wickens, C.D. Multiple resource modeling of task interference in vehicle control, hazard awareness and in-vehicle task performance, Driving Assessment Conference 2003, 2003.Google Scholar
Chen, J., Zhang, Q. and Hou, B. An assessment method of pilot workload in manned/unmanned-aerial-vehicles team, 2017 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), October 2017, pp 1–5.CrossRefGoogle Scholar
Moray, N. Mental workload: its theory and measurement, Proceedings of the NATO Symposium on Theory and Measurement of Mental Workload, 1979.CrossRefGoogle Scholar
Wilson, G. An analysis of mental workload in pilots during flight using multiple psychophysiological measures, Int. J. Aviat. Psychol., 2002, 12, (1), pp 3–18.CrossRefGoogle Scholar
Wan, H., Zhuang, L., Pan, Y., Gao, F., Tu, J., Zhang, B. and Wang, P. Biomedical sensors, in Biomedical Information Technology, 2020, pp 51–79.CrossRefGoogle Scholar
Charles, R.L. and Nixon, J. Measuring mental workload using physiological measures: a systematic review, Appl. Ergon., 2019, 74, pp 221–232.CrossRefGoogle ScholarPubMed
Kramer, A.F. Physiological metrics of mental workload: a review of recent progress, in Multiple-Task Performance, Taylor & Francis, 1991, pp 279–328.Google Scholar
Hughes, A.M., Hancock, G.M., Marlow, S.L., Stowers, K. and Salas, E. Cardiac measures of cognitive workload: a meta-analysis, Hum. Factors, 2019, 61, (3), pp 393–414.CrossRefGoogle ScholarPubMed
Parak, J. and Havlik, J. ECG Signal Processing and Heart Rate Frequency Detection Methods, Technical Computing 2011, 2011.Google Scholar
Park, J., Seok Seok, H., Kim, S. and Shin, H. Photoplethysmogram analysis and applications: an integrative review, Front. Physiol., 2022, 12, pp 1–23.CrossRefGoogle Scholar
Yoo, K.S. and Lee, W.H. Mental stress assessment based on photoplethysmography, 2011 IEEE 15th International Symposium on Consumer Electronics, 2011.CrossRefGoogle Scholar
Miyake, S., Yamada, S., Shoji, T., Takae, Y., Kuge, N. and Yamamura, T. Physiological responses to workload change. A test/retest examination, Appl. Ergon., 2009, 40, (6), pp 987–996.CrossRefGoogle Scholar
Gable, T.M., Kun, A.L., Walker, B.N. and Winton, R.J. Comparing heart rate and pupil size as objective measures of workload in the driving context: initial look, Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI ’15), 2015.CrossRefGoogle Scholar
Zhang, B., Morere, Y., Sieler, L. and Lenglet, C. Stress recognition from heterogeneous data, J. Image Graphics, 2016, 4, (2), pp 116–121.CrossRefGoogle Scholar
Katsis, C.D., Ganiatas, G. and Fotiadis, D.I. An integrated telemedicine platform for the assessment of affective physiological states, Diagn. Pathol., 2006, 1, (16), article number: 1.CrossRefGoogle ScholarPubMed
Liao, W., Zhang, W., Zhu, Z. and Ji, Q. A real-time human stress monitoring system using dynamic bayesian network, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) - Workshops, 2005.Google Scholar
Giorgi, A., Ronca, V., Vozzi, A., Sciaraffa, N., di Florio, A., Tamborra, L., Simonetti, I., Aricò, P., Di Flumeri, G., Rossi, D. and Borghini, G. Wearable technologies for mental workload, stress, and emotional state assessment during working-like tasks: a comparison with laboratory technologies, Sensors, 2021, 21, (7), article number: 2332.CrossRefGoogle ScholarPubMed
Pongsakornsathien, N., Lim, Y., Gardi, A., Hilton, S. Planke, L., Sabatini, R., Kistan, T. and Ezer, N. Sensor networks for aerospace human-machine systems, Sensors, 2019, 19, (16), article number: 3465.CrossRefGoogle ScholarPubMed
Lim, Y., Gardi, A., Sabatini, R., Ramasamy, S., Kistan, T., Ezer, N., Vince, J. and Bolia, R. Avionics human-machine interfaces and interactions for manned and unmanned aircraft, Progr. Aerospace Sci., 2018, 102, pp 1–46.CrossRefGoogle Scholar
Planke, L.J., Lim, Y., Gardi, A., Sabatini, R., Trevor, K. and Ezer, N. A cyber-physical-human system for one-to-many UAS operations: cognitive load analysis, Sensors, 2020, 20, (19), article number: 5467.CrossRefGoogle ScholarPubMed
Wang, Z., Zheng, L., Lu, Y. and Fu, S. Physiological indices of pilots’ abilities under varying task demands, Aerospace Med. Hum. Perform., 2016, 87, (4), pp 375–381.CrossRefGoogle ScholarPubMed
Mansikka, H., Virtanen, K., Harris, D. and Simola, P. Fighter pilots’ heart rate, heart rate variation and performance during an instrument flight rules proficiency test, Appl. Ergon., 2016, 56, pp 213–219.CrossRefGoogle ScholarPubMed
Wei, Z., Zhuang, D., Wanyan, X., Liu, C. and Zhuang, H. A model for discrimination and prediction of mental workload of aircraft cockpit display interface, Chin. J. Aeronaut., 2014, 27, (5), pp 1070–1077.CrossRefGoogle Scholar
Gentili, R.J., Rietschel, J.C., Jaquess, K.J., Lo, L.C., Prevost, M., Miller, M.W., Mohler, J.M., Oh, H., Tan, Y.Y. and Hatfield, B.D. Brain biomarkers based assessment of cognitive workload in pilots under various task demands, Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2014, pp 5860–5863.CrossRefGoogle Scholar
Martin, P., Calhoun, P., Schnell, T. and Thompson, C. Objective Measures of Pilot Workload, 2019.Google Scholar
Lai, M.L., Tsai, M.L., Yang, F.Y., Hsu, C.Y., Liu, T.C., Lee, S.W.Y., Lee, M.H., Chiou, G.L., Liang, J.C. and Tsai, C.C. A review of using eye-tracking technology in exploring learning from 2000 to 2012, Edu. Res. Rev., 2013, 10, pp 90–115.Google Scholar
Mahanama, B., Jayawardana, Y., Rengarajan, S., Jayawardena, G., Chukoskie, L., Snider, J. and Jayarathna, S. Eye movement and pupil measures. A review, Front. Comput. Sci., 2022, 3, article number: 733531.CrossRefGoogle Scholar
Salvucci, D.D. and Goldberg, J.H. Identifying fixations and saccades in eye-tracking protocols, Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (ETRA’00), 2000.CrossRefGoogle Scholar
Barea, R., Boquete, S., Ortega, S., Lopez, E. and Rodríguez-Ascariz, J.M. EOG-based eye movements codification for human computer interaction, Expert Syst. Appl., 2012, 39, (3), pp 2677–2683.CrossRefGoogle Scholar
Belkhiria, C. and Peysakhovich, V. EOG metrics for cognitive workload detection, 25th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems, vol. 192, 2021, pp 1875–1884.CrossRefGoogle Scholar
Vanneste, P., Raes, A., Morton, J., Bombeke, K., Van Acker, B.B., Larmuseau, C., Depaepe, F. and Van den Noortgate, W. Towards measuring cognitive load through multimodal physiological data, Cognit. Technol. Work, 2021, 23, pp 567–585.CrossRefGoogle Scholar
Dilli Babu, M., Jeevitha, S., Gowdham, P., Kamalpreet, S., Abhay, P. and Pradipta, B. Estimating pilots’ cognitive load from ocular parameters through simulation and in-flight studies, J. Eye Mov. Res., 2019, 12, (3), article number: 3.Google Scholar
Posada-Quintero, H.F. and Chon, K.H. Innovations in electrodermal activity data collection and signal processing: a systematic review, Sensors, 2020, 20, (2).CrossRefGoogle ScholarPubMed
Cacioppo, J., Tassinary, L. and Berntson, G. Handbook of Psychophysiology, 2007.Google Scholar
Greco, A., Valenza, G., Lanata, A., Scilingo, E.P. and Citi, L. cvxEDA: a convex optimization approach to electrodermal activity processing, IEEE Trans. Biomed. Eng., 2016, 63, (4), pp 797–804.Google ScholarPubMed
Setz, C., Arnrich, B., Schumm, J., Marca, R., Tröster, G. and Ehlert, U. Discriminating stress from cognitive load using wearable EDA device, IEEE Trans. Inf. Technol. Biomed., 2010, 14, pp 410–417.CrossRefGoogle ScholarPubMed
Ghaderyan, P., Abbasi, A. and Ebrahimi, A. Time-varying singular value decomposition analysis of electrodermal activity: a novel method of cognitive load estimation, Measurement, 2018, 126, pp 102–109.CrossRefGoogle Scholar
Ghani, U., Signal, N., Niazi, I.K. and Taylor, D. ERP based measures of cognitive workload: a review, Neurosci. Biobehav. Rev., 2020, 118, pp 18–26.CrossRefGoogle ScholarPubMed
Kumar, S.J. and Bhuvaneswari, P. Analysis of Electroencephalography (EEG) signals and its categorization - a study, Proc. Eng., 2012, 38, pp 2525–2536.CrossRefGoogle Scholar
Handy, T.C. Event-Related Potentials: A Methods Handbook, MIT Press, Cambridge, MA, 2005.Google Scholar
Welch, T.L. and Pasternak, J.J. Chapter 56 - recent advances in neuroanesthesiology, in Essentials of Neuroanesthesia, Academic Press, 2017, pp 897–905.CrossRefGoogle Scholar
Rebsamen, B., Kwok, K. and Penney, T.B. Evaluation of cognitive workload from EEG durting a mental arithmetic task, Proc. Hum. Factors Ergon. Soc. 55th Ann. Meet., 2011, 55, (1), pp 1342–1345.CrossRefGoogle Scholar
Svinkunaite, L., Horschig, J.M. and Floor-Westerdijk, M.J. Employing cardiac and respiratory features extracted from fNIRS signals for mental workload classification, SPIE BiOS, 2021.CrossRefGoogle Scholar
Hajra, S.G., Liu, C., Law, A. Neural responses to spontaneous blinking capture differences in working memory load: assessing blink related oscillations with N-back task, International Neuroergonomics Conference, 2019.Google Scholar
Herff, C., Heger, D., Fortmann, O., Hennrich, J., Putze, F. and Schultz, T. Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS, Front. Hum. Neurosci., 2014, 7, article number: 935.CrossRefGoogle ScholarPubMed
Mohanavelu, K., Poonguzhali, S., Adalarasu, K., Ravi, D., Vijayakumar, C., Vinutha, S., Ramachandran, K. and Srinivasan, J. Dynamic cognitive workload assessment for fighter pilots in simulated fighter aircraft environment using EEG, Biomed. Signal Process. Control, 2020, 61, article number: 102018.Google Scholar
Grassmann, M., Vlemincx, E., von Leupoldt, A., Mittelstädt, J.M. and Van den Bergh, O. Respiratory changes in response to cognitive load: a systematic review, Neural Plast., 2016, 2016, article number: 8146809.CrossRefGoogle Scholar
Monaco, V. and Stefanini, C. Assessing the tidal volume through wearables: a scoping review, Sensors, 2021, 21, (12).CrossRefGoogle ScholarPubMed
Romine, W., Scroeder, N., Graft, J., Yang, F., Sadeghi, R., Zabihimayvan, M., Kadariya, D. and Banerjee, T. Using machine learning to train a wearable device for measuring students’ cognitive load during problem-solving activities based on electrodermal activity, body temperature, and heart rate: development of a cognitive load tracker for both personal and classroom use, Sensors, 2020, 20, (4883).CrossRefGoogle Scholar
Raez, M.B., Hussain, M.S. and Mohd-Yasin, F. Techniques of EMG signal analysis: detection, processing, classification and applications, Biol. Proced. Online, 2006, 8, (163), pp 11–35.CrossRefGoogle ScholarPubMed
Salomone, M., Burle, B., Fabre, L. and Berberian, B. An electromyographic analysis of the effects of cognitive fatigue on online and anticipatory action control, Front. Hum. Neurosci., 2021, 14, article number: 615046.CrossRefGoogle ScholarPubMed
Massaroni, C., Nicolò, A., Lo Presti, D., Sacchetti, M., Silvestri, S. and Schena, E. Contact-based methods for measuring respiratory rate, Sensors, 2019, 19, (4), article number: 908.CrossRefGoogle ScholarPubMed
Al-khalidi, F., Saatchi, R., Burke, D. and Elphick, H. Respiration rate monitoring methods: a review, Pediatr. Pulonol., 2011, 46, pp 523–529.CrossRefGoogle ScholarPubMed
Bach, A.J.E., Stewart, I.B., Minnett, G.M. and Costello, J.T. Does the technique employed for skin temperature assessment alter outcomes? A systematic review, Physiol. Meas., 2015, 36, (9), pp 27–51 CrossRefGoogle ScholarPubMed