Objective. Recent studies have shown that integrating inertial measurement unit (IMU) signals with surface electromyographic (sEMG) can greatly improve hand gesture recognition (HGR) performance in applications such as prosthetic control and rehabilitation training. However, current deep learning models for multimodal HGR encounter difficulties in invasive modal fusion, complex feature extraction from heterogeneous signals, and limited inter-subject model generalization. To address these challenges, this study aims to develop an end-to-end and inter-subject transferable model that utilizes non-invasively fused sEMG and acceleration (ACC) data. Approach. The proposed non-invasive modal fusion-transformer (NIMFT) model utilizes 1D-convolutional neural networks-based patch embedding for local information extraction and employs a multi-head cross-attention (MCA) mechanism to non-invasively integrate sEMG and ACC signals, stabilizing the variability induced by sEMG. The proposed architecture undergoes detailed ablation studies after hyperparameter tuning. Transfer learning is employed by fine-tuning a pre-trained model on new subject and a comparative analysis is performed between the fine-tuning and subject-specific model. Additionally, the performance of NIMFT is compared to state-of-the-art fusion models. Main results. The NIMFT model achieved recognition accuracies of 93.91%, 91.02%, and 95.56% on the three action sets in the Ninapro DB2 dataset. The proposed embedding method and MCA outperformed the traditional invasive modal fusion transformer by 2.01% (embedding) and 1.23% (fusion), respectively. In comparison to subject-specific models, the fine-tuning model exhibited the highest average accuracy improvement of 2.26%, achieving a final accuracy of 96.13%. Moreover, the NIMFT model demonstrated superiority in terms of accuracy, recall, precision, and F1-score compared to the latest modal fusion models with similar model scale. Significance. The NIMFT is a novel end-to-end HGR model, utilizes a non-invasive MCA mechanism to integrate long-range intermodal information effectively. Compared to recent modal fusion models, it demonstrates superior performance in inter-subject experiments and offers higher training efficiency and accuracy levels through transfer learning than subject-specific approaches.

中文翻译：

---

用于端到端手势识别的可转移非侵入模态融合变压器（NIMFT）

客观的。最近的研究表明，将惯性测量单元（IMU）信号与表面肌电图（sEMG）集成可以极大地提高假肢控制和康复训练等应用中的手势识别（HGR）性能。然而，当前的多模态 HGR 深度学习模型在侵入式模态融合、从异构信号中提取复杂特征以及有限的主体间模型泛化方面遇到了困难。为了应对这些挑战，本研究旨在开发一种端到端和受试者间可转移模型，该模型利用非侵入性融合表面肌电图和加速度（ACC）数据。方法。所提出的非侵入性模态融合变换器（NIMFT）模型利用基于一维卷积神经网络的补丁嵌入进行局部信息提取，并采用多头交叉注意（MCA）机制来非侵入性地集成sEMG和ACC信号，稳定 sEMG 引起的变异性。所提出的架构在超参数调整后进行了详细的消融研究。通过对新主题的预训练模型进行微调来采用迁移学习，并在微调模型和特定主题模型之间进行比较分析。此外，还将 NIMFT 的性能与最先进的融合模型进行了比较。主要结果。NIMFT 模型在 Ninapro DB2 数据集中的三个动作集上实现了 93.91%、91.02% 和 95.56% 的识别准确率。所提出的嵌入方法和 MCA 的性能分别比传统的侵入式模态融合变压器高 2.01%（嵌入）和 1.23%（融合）。与特定主题模型相比，微调模型的平均准确率最高提高了 2.26%，最终准确率达到 96.13%。此外，与具有相似模型规模的最新模态融合模型相比，NIMFT 模型在准确性、召回率、精度和 F1 分数方面表现出优越性。意义。NIMFT是一种新颖的端到端HGR模型，利用非侵入性MCA机制有效地集成远程多式联运信息。与最近的模态融合模型相比，它在受试者间实验中表现出优越的性能，并通过迁移学习提供比特定于受试者的方法更高的训练效率和准确性水平。