Abstract

In recent years, emotion recognition has received significant attention, presenting a plethora of opportunities for application in diverse fields such as human–computer interaction, psychology, and neuroscience, to name a few. Although unimodal emotion recognition methods offer certain benefits, they have limited ability to encompass the full spectrum of human emotional expression. In contrast, Multimodal Emotion Recognition (MER) delivers a more holistic and detailed insight into an individual's emotional state. However, existing multimodal data collection approaches utilizing contact-based devices hinder the effective deployment of this technology. We address this issue by examining the potential of contactless data collection techniques for MER. In our tertiary review study, we highlight the unaddressed gaps in the existing body of literature on MER. Through our rigorous analysis of MER studies, we identify the modalities, specific cues, open datasets with contactless cues, and unique modality combinations. This further leads us to the formulation of a comparative schema for mapping the MER requirements of a given scenario to a specific modality combination. Subsequently, we discuss the implementation of Contactless Multimodal Emotion Recognition (CMER) systems in diverse use cases with the help of the comparative schema which serves as an evaluation blueprint. Furthermore, this paper also explores ethical and privacy considerations concerning the employment of contactless MER and proposes the key principles for addressing ethical and privacy concerns. The paper further investigates the current challenges and future prospects in the field, offering recommendations for future research and development in CMER. Our study serves as a resource for researchers and practitioners in the field of emotion recognition, as well as those intrigued by the broader outcomes of this rapidly progressing technology.

中文翻译：

---

探索多模态情感识别中的非接触式技术：深入了解不同的应用、挑战、解决方案和前景

摘要

近年来，情绪识别受到了极大的关注，在人机交互、心理学和神经科学等不同领域提供了大量的应用机会。尽管单模态情感识别方法具有一定的优势，但它们涵盖人类情感表达全部范围的能力有限。相比之下，多模态情绪识别 (MER) 可以更全面、更详细地洞察个人的情绪状态。然而，现有的利用基于接触的设备的多模式数据收集方法阻碍了该技术的有效部署。我们通过研究 MER 非接触式数据收集技术的潜力来解决这个问题。在我们的三级综述研究中，我们强调了现有 MER 文献中尚未解决的空白。通过对 MER 研究的严格分析，我们确定了模式、特定线索、具有非接触式线索的开放数据集以及独特的模式组合。这进一步引导我们制定一个比较模式，用于将给定场景的 MER 要求映射到特定的模态组合。随后，我们借助作为评估蓝图的比较模式，讨论非接触式多模态情感识别（CMER）系统在不同用例中的实施。此外，本文还探讨了有关使用非接触式 MER 的道德和隐私考虑因素，并提出了解决道德和隐私问题的关键原则。本文进一步探讨了该领域当前的挑战和未来前景，为 CMER 未来的研究和发展提供了建议。我们的研究为情感识别领域的研究人员和从业者以及那些对这种快速发展的技术的更广泛成果感兴趣的人提供了资源。