Abstract
Accurate and rapid wood species identification is vital for wood utilization and trade. This goal is achievable with the fast development of deep learning (DL). Several studies have been published related to this topic; however, they were limited by their generalization performance in practical applications. Therefore, this study proposed a DL multimodal fusion framework to bridge this gap. The study utilized a state-of-the-art convolutional neural network (CNN) to simultaneously extract both short-wavelength near-infrared (NIR) spectra and RGB image feature, fully leveraging the advantages of both data types. Using portable devices for collecting spectra and image data enhances the feasibility of onsite rapid identification. In particular, a two-branch CNN framework was developed to extract spectra and image features. For NIR spectra feature extraction, 1 dimensional NIR (1D NIR) spectra were innovatively encoded as 2 dimensional (2D) images using the Gramian angular difference field (GADF) method. This representation enhances better data alignment with CNN operations, facilitating more robust discriminative feature extraction. Moreover, wood’s spectral and image features were fused at the full connection layer for species identification. In the experimental phase conducted on 16 difficult-to-distinguish wood samples from the Lauraceae family, all achieved identification metrics results exceed 99 %. The findings illustrate that the proposed multimodal fusion framework effectively extracts and fully integrates the wood’s features, thereby, improving wood species identification.
Funding source: China National Natural Science Funds
Award Identifier / Grant number: 31770766, 31370711
Funding source: Fundamental Research Funds for Central Public Welfare Research Institutes
Award Identifier / Grant number: CAFYBB2021ZJ001
Acknowledgments
The authors gratefully acknowledge the xylarium of Southwest Forestry University for supporting the wood specimens used in this study. Additionally, the authors thank Beijing Great Technology Co., Ltd. for supporting the handheld NIR spectrometer.
-
Research ethics: Not involved.
-
Author contributions: Xi Pan: conceptualization, data curation, investigation, methodology, software, writing (original draft), data curation, investigation, validation; Zhiming Yu: formal analysis, resources, writing-review and editing; Zhong Yang: funding acquisition, supervision, writing-review and editing.
-
Competing interests: The authors declare no conflict of interest.
-
Research funding: This work was supported by the China National Natural Science Funds [grant nos. 31770766 and 31370711]; the Fundamental Research Funds for Central Public Welfare Research Institutes [grant no. CAFYBB2021ZJ001].
-
Data availability: The datasets generated and/or analysed during the current study are not publicly available as they are part of a larger dataset.
References
Barmpoutis, P., Dimitropoulos, K., Barboutis, I., Grammalidis, N., and Lefakis, P. (2018). Wood species recognition through multidimensional texture analysis. Comput. Electron. Agric. 144: 241–248, https://doi.org/10.1016/j.compag.2017.12.011.Search in Google Scholar
Bergo, M.C., Pastore, T.C., Coradin, V.T., Wiedenhoeft, A.C., and Braga, J.W. (2016). NIRS identification of Swietenia macrophylla is robust across specimens from 27 countries. IAWA J. 37: 420–430, https://doi.org/10.1163/22941932-20160144.Search in Google Scholar
Braga, J.W., Pastore, T.C., Coradin, V.T., Camargos, J.A., and da Silva, A.R. (2011). The use of near infrared spectroscopy to identify solid wood specimens of Swietenia macrophylla (cites appendix II). IAWA J. 32: 285–296, https://doi.org/10.1163/22941932-90000058.Search in Google Scholar
Han, J., Zhao, P., and Wang, C. (2021). Wood species recognition through FGLAM textural and spectral feature fusion. Wood Sci. Technol. 55: 535–552, https://doi.org/10.1007/s00226-021-01262-0.Search in Google Scholar
He, T., Jiao, L., Wiedenhoeft, A.C., and Yin, Y. (2019). Machine learning approaches outperform distance- and tree-based methods for DNA barcoding of Pterocarpus wood. Planta 249: 1617–1625, https://doi.org/10.1007/s00425-019-03116-3.Search in Google Scholar PubMed
He, T., Lu, Y., Jiao, L., Zhang, Y., Jiang, X., and Yin, Y. (2020). Developing deep learning models to automate rosewood tree species identification for CITES designation and implementation. Holzforschung 74: 1123–1133, https://doi.org/10.1515/hf-2020-0006.Search in Google Scholar
He, T., Mu, S., Zhou, H., and Hu, J. (2021). Wood species identification based on an ensemble of deep convolution neural networks. Wood Res. 66: 01–14, https://doi.org/10.37763/66.1.0114.Search in Google Scholar
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, NV, USA.10.1109/CVPR.2016.90Search in Google Scholar
Hwang, S. and Sugiyama, J. (2021). Computer vision-based wood identification and its expansion and contribution potentials in wood science: a review. Plant Methods 17: 47, https://doi.org/10.1186/s13007-021-00746-1.Search in Google Scholar PubMed PubMed Central
Hwang, S., Kobayashi, K., Zhai, S., and Sugiyama, J. (2018). Automated identification of Lauraceae by scale-invariant feature transform. J. Wood Sci. 64: 69–77, https://doi.org/10.1007/s10086-017-1680-x.Search in Google Scholar
Kanayama, H., Ma, T., Tsuchikawa, S., and Inagaki, T. (2019). Cognitive spectroscopy for wood species identification: near infrared hyperspectral imaging combined with convolutional neural network. Analyst 144, https://doi.org/10.1039/C9AN01180C.Search in Google Scholar
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature 521: 436–444, https://doi.org/10.1038/nature14539.Search in Google Scholar PubMed
Lens, F., Liang, C., Guo, Y., Tang, X., Jahanbanifard, M., Da Silva, F.S., Ceccantini, G., and Verbeek, F. (2020). Computer-assisted timber identification based on features extracted from microscopic wood sections. IAWA J. 41: 660–680, https://doi.org/10.1163/22941932-bja10029.Search in Google Scholar
Liu, Z., Ci, X., Li, L., Li, H., Conran, J., and Li, J. (2017). DNA barcoding evaluation and implications for phylogenetic relationships in Lauraceae from China. PLoS One 12: e0175788, https://doi.org/10.1371/journal.pone.0175788.Search in Google Scholar PubMed PubMed Central
Liu, Z., Ma, H., Zhang, X., Ci, X., Li, L., Hu, J., Zhang, C., Xiao, J., Li, H., Conran, J., et al.. (2022). Do taxon-specific DNA barcodes improve species discrimination relative to universal barcodes in Lauraceae? Bot. J. Linn. Soc. 199: 741–753, https://doi.org/10.1093/botlinnean/boab089.Search in Google Scholar
Pan, X., Li, K., Chen, Z., and Yang, Z. (2021). Identifying wood based on near-infrared spectra and four gray-level co-occurrence matrix texture features. Forests 12: 1527, https://doi.org/10.3390/f12111527.Search in Google Scholar
Pan, X., Qiu, J., and Yang, Z. (2023). Identification of softwood species using convolutional neural networks and raw near-infrared spectroscopy. Wood Mater. Sci. Eng. 18: 1388–1348, https://doi.org/10.1080/17480272.2022.2130822.Search in Google Scholar
Passos, D. and Mishra, P. (2022). Perspectives on deep learning for near-infrared spectral data modelling. NIR News 33: 9–12, https://doi.org/10.1177/09603360221142821.Search in Google Scholar
Pastore, T.C., Braga, J.W., Coradin, V.T., Magalhães, W.L., Okino, E.Y., Camargos, J.A., de Muñiz, G.I., Bressan, O.A., and Davrieux, F. (2011). Near infrared spectroscopy (NIRS) as a potential tool for monitoring trade of similar woods: discrimination of true mahogany, cedar, andiroba, and curupixá. Holzforschung 65: 73–80, https://doi.org/10.1515/hf.2011.010.Search in Google Scholar
Ravindran, P., Thompson, B., Soares, R., and Wiedenhoeft, A. (2020). The XyloTron: flexible, open-source, image-based macroscopic field identification of wood products. Front. Plant Sci. 11: 1015, https://doi.org/10.3389/fpls.2020.01015.Search in Google Scholar PubMed PubMed Central
Rocha, H.S., Braga, J.W., Kunze, D.C., Coradin, V.T., and Pastore, T.C. (2021). Identification of mahogany sliced veneer using handheld near-infrared spectroscopy device and multivariate data analysis. IAWA J. 42: 336–347, https://doi.org/10.1163/22941932-bja10054.Search in Google Scholar
Schwanninger, M., Rodrigues, J., and Fackler, K. (2011). A review of band assignments in near infrared spectra of wood and wood components. J. Near Infrared Spectrosc. 19: 287–308, https://doi.org/10.1255/jnirs.955.Search in Google Scholar
Sermanet, P., Chintala, S., and LeCun, Y. (2012). Convolutional neural networks applied to house numbers digit classification. In: 2012 21st International Conference on Pattern Recognition (ICPR 2012). IEEE Computer Society, Tsukuba, pp. 3288–3291.Search in Google Scholar
Silva, D.C., Pastore, T.C., Soares, L.F., de Barros, F.A., Bergo, M.C., Coradin, V.T., Gontijo, A.B., Sosa, M.H., Chacón, C.B., and Braga, J.W. (2018). Determination of the country of origin of true mahogany (Swietenia macrophylla King) wood in five Latin American countries using handheld NIR devices and multivariate data analysis. Holzforschung 72: 521–530, https://doi.org/10.1515/hf-2017-0160.Search in Google Scholar
Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, San Diego.Search in Google Scholar
Snel, F.A., Braga, J.W., da Silva, D., Wiedenhoeft, A.C., Costa, A., Soares, R., Coradin, V.T., and Pastore, T.C. (2018). Potential field-deployable NIRS identification of seven Dalbergia species listed by CITES. Wood Sci. Technol. 52: 1411–1427, https://doi.org/10.1007/s00226-018-1027-9.Search in Google Scholar
Sohi, A., Avramidis, S., and Mansfield, S. (2017). Near-infrared spectroscopic separation of green chain sub-alpine fir lumber from a spruce-pine-fir mix. BioResources 12: 3720–3727, https://doi.org/10.15376/biores.12.2.3720-3727.Search in Google Scholar
Sun, J., Wu, J., Wang, X., Gu, J., and Gao, Z. (2015). Comparative wood anatomy of 56 species of Lauraceae from Yunnan, China. Braz. J. Bot. 38: 645–656, https://doi.org/10.1007/s40415-015-0172-8.Search in Google Scholar
Sun, Y., Lin, Q., He, X., Zhao, Y., Dai, F., Qiu, J., and Cao, Y. (2021). Wood species recognition with small data: a deep learning approach. Int. J. Comput. Intell. Syst. 14: 1451, https://doi.org/10.2991/ijcis.d.210423.001.Search in Google Scholar
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, pp. 2818–2826.10.1109/CVPR.2016.308Search in Google Scholar
Wang, C. and Zhao, P. (2021). Classification of wood species using spectral and texture features of transverse section. Eur. J. Wood Prod. 79: 1283–1296, https://doi.org/10.1007/s00107-021-01728-9.Search in Google Scholar
Wang, C., Zhao, P., Li, Z., and Li, X. (2022). Comparison of VIS/NIR spectral curves plus RGB images with hyperspectral images for the identification of Pterocarpus species. Holzforschung 76: 579–591, https://doi.org/10.1515/hf-2021-0194.Search in Google Scholar
Wang, Z. and Oates, T. (2015). Imaging time-series to improve classification and imputation. In: Proceedings of the 24th International Conference on Artificial Intelligence. Buenos Aires, pp. 3939–3945.Search in Google Scholar
Wang, Y., Zhang, W., Gao, R., Jin, Z., and Wang, X. (2021). Recent advances in the application of deep learning methods to forestry. Wood Sci. Technol. 55: 1171–1202, https://doi.org/10.1007/s00226-021-01309-2.Search in Google Scholar
Yang, S., Kwon, O., Park, Y., Chung, H., Kim, H., Park, S., Choi, I., and Yeo, H. (2020). Application of neural networks for classifying softwood species using near infrared spectroscopy. J. Near Infrared Spectrosc. 28: 298–307, https://doi.org/10.1177/0967033520939320.Search in Google Scholar
© 2023 Walter de Gruyter GmbH, Berlin/Boston
