Accurate and rapid wood species identification is vital for wood utilization and trade. This goal is achievable with the fast development of deep learning (DL). Several studies have been published related to this topic; however, they were limited by their generalization performance in practical applications. Therefore, this study proposed a DL multimodal fusion framework to bridge this gap. The study utilized a state-of-the-art convolutional neural network (CNN) to simultaneously extract both short-wavelength near-infrared (NIR) spectra and RGB image feature, fully leveraging the advantages of both data types. Using portable devices for collecting spectra and image data enhances the feasibility of onsite rapid identification. In particular, a two-branch CNN framework was developed to extract spectra and image features. For NIR spectra feature extraction, 1 dimensional NIR (1D NIR) spectra were innovatively encoded as 2 dimensional (2D) images using the Gramian angular difference field (GADF) method. This representation enhances better data alignment with CNN operations, facilitating more robust discriminative feature extraction. Moreover, wood’s spectral and image features were fused at the full connection layer for species identification. In the experimental phase conducted on 16 difficult-to-distinguish wood samples from the Lauraceae family, all achieved identification metrics results exceed 99 %. The findings illustrate that the proposed multimodal fusion framework effectively extracts and fully integrates the wood’s features, thereby, improving wood species identification.

中文翻译：

---

使用近红外光谱 GADF 和 RGB 图像进行木材树种识别的深度学习多模态融合框架

准确、快速的木材树种鉴定对于木材利用和贸易至关重要。随着深度学习（DL）的快速发展，这个目标是可以实现的。与该主题相关的多项研究已发表；然而，它们在实际应用中受到泛化性能的限制。因此，本研究提出了一种深度学习多模态融合框架来弥补这一差距。该研究利用最先进的卷积神经网络（CNN）同时提取短波长近红外（NIR）光谱和RGB图像特征，充分利用两种数据类型的优势。使用便携式设备采集光谱和图像数据增强了现场快速识别的可行性。特别是，开发了一个两分支 CNN 框架来提取光谱和图像特征。对于近红外光谱特征提取，使用格拉米角差场 (GADF) 方法创新性地将一维近红外 (1D NIR) 光谱编码为二维 (2D) 图像。这种表示增强了与 CNN 操作的更好的数据对齐，从而促进更稳健的判别性特征提取。此外，木材的光谱和图像特征在全连接层融合，用于物种识别。在对樟科16种难以区分的木材样品进行的实验阶段，均取得了99%以上的鉴定指标结果。研究结果表明，所提出的多模态融合框架有效地提取并充分整合了木材的特征，从而提高了木材树种的识别。