Skip to main content
SpringerLink
Log in

Dynamic penetration prediction based on continuous video learning

  • Research Paper
  • Published:
Welding in the World Aims and scope Submit manuscript

Abstract

Online penetration monitoring of complex grooves remains challenging due to steel plates’ groove instability and welding heat distortion. Penetration is an accumulation process of material deposition. Temporal signals, such as video, can provide a more comprehensive characterization of the melt pool state. A deep learning method based on continuous video is designed to monitor groove welding penetration in-process. The proposed Fast Video-feature Extraction Net (FVENet) consists of a video extraction module and a multi-feature screening module. The efficient network can quickly extract high-dimensional data features in complex arc environments and achieve accurate results for backside melt width prediction. The feature extraction process of the network is explored by visualizing the results of different network layers. Experimental results indicate that the mean squared error (MSE) of FVENet reaches 0.0634 mm, outperforming other mainstream deep learning frameworks. The inference time under video input reaches 100 FPS. The network structure designed in this paper has the potential to become a universal template for processing melt pool images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Nomura K, Fukushima K, Matsumura T, Asai S (2021) Burn-through prediction and weld depth estimation by deep learning model monitoring the molten pool in gas metal arc welding with gap fluctuation. J Manuf Process 61:. https://doi.org/10.1016/j.jmapro.2020.10.019

  2. Jeon I, Liu P, Sohn H (2023) Real-time melt pool depth estimation and control during metal-directed energy deposition for porosity reduction. International Journal of Advanced Manufacturing Technology. https://doi.org/10.1007/s00170-023-11689-3

  3. Sweeney NE, Parke S, Lines D, et al (2023) In-process phased array ultrasonic weld pool monitoring. NDT and E International 137:. https://doi.org/10.1016/j.ndteint.2023.102850

  4. Lin R, Wang H ping, Lu F, et al (2017) Numerical study of keyhole dynamics and keyhole-induced porosity formation in remote laser welding of Al alloys. Int J Heat Mass Transf 108:. https://doi.org/10.1016/j.ijheatmasstransfer.2016.12.019

  5. Wu J, Huang C, Li Z, et al (2023) An in situ surface defect detection method based on improved you only look once algorithm for wire and arc additive manufacturing. Rapid Prototyp J 29:. https://doi.org/10.1108/RPJ-06-2022-0211

  6. Dai P, Wang Y, Li S, et al (2020) FEM analysis of residual stress induced by repair welding in SUS304 stainless steel pipe butt-welded joint. J Manuf Process 58:. https://doi.org/10.1016/j.jmapro.2020.09.006

  7. Zhang K, Yan M, Huang T, et al (2019) 3D reconstruction of complex spatial weld seam for autonomous welding by laser structured light scanning. J Manuf Process 39:. https://doi.org/10.1016/j.jmapro.2019.02.010

  8. Yu R, Han J, Bai L, Zhao Z (2021) Identification of butt welded joint penetration based on infrared thermal imaging. Journal of Materials Research and Technology 12:. https://doi.org/10.1016/j.jmrt.2021.03.075

  9. Chen C, Xiao R, Chen H, et al (2020) Arc sound model for pulsed GTAW and recognition of different penetration states. International Journal of Advanced Manufacturing Technology 108:. https://doi.org/10.1007/s00170-020-05462-z

  10. Zhang T, Xu C, Cheng J, et al (2023) Research of surface oxidation defects in copper alloy wire arc additive manufacturing based on time-frequency analysis and deep learning method. Journal of Materials Research and Technology 25:. https://doi.org/10.1016/j.jmrt.2023.05.227

  11. Li S, Jiang P, Gao Y, et al (2023) A penetration depth monitoring method for Al-Cu laser lap welding based on spectral signals. J Mater Process Technol 317:. https://doi.org/10.1016/j.jmatprotec.2023.117972

  12. Liu T, Zheng P, Bao J (2023) Deep learning-based welding image recognition: a comprehensive review. J Manuf Syst 68. https://doi.org/10.1016/j.jmsy.2023.05.026

  13. Liang R, Yu R, Luo Y, Zhang YM (2019) Machine learning of weld joint penetration from weld pool surface using support vector regression. J Manuf Process 41:. https://doi.org/10.1016/j.jmapro.2019.01.039

  14. Lei Z, Shen J, Wang Q, Chen Y (2019) Real-time weld geometry prediction based on multi-information using neural network optimized by PCA and GA during thin-plate laser welding. J Manuf Process 43:. https://doi.org/10.1016/j.jmapro.2019.05.013

  15. Wu D, Hu M, Huang Y, et al (2021) In situ monitoring and penetration prediction of plasma arc welding based on welder intelligence-enhanced deep random forest fusion. J Manuf Process 66:. https://doi.org/10.1016/j.jmapro.2021.04.007

  16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.90

  17. Jia Q, He J, Li F, Hua X (2023) Penetration depth predicition of thin plate plasma arc lap welding based on machine learning. Cailiao Kexue yu Gongyi/Material Science and Technology 31:. https://doi.org/10.11951/j.issn.1005-0299.20220247

  18. Wang Z, Chen H, Zhong Q, et al (2022) Recognition of penetration state in GTAW based on vision transformer using weld pool image. International Journal of Advanced Manufacturing Technology 119:. https://doi.org/10.1007/s00170-021-08538-6

  19. Liao S, Huang C, Liang Y, et al (2022) Solder joint defect inspection method based on ConvNeXt-YOLOX. IEEE Trans Compon Packaging Manuf Technol 12:. https://doi.org/10.1109/TCPMT.2022.3224997

  20. Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Advances in Neural Information Processing Systems. https://doi.org/10.48550/arXiv.1706.03762

  21. Liu Z, Mao H, Wu CY, et al (2022) A ConvNet for the 2020s. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR52688.2022.01167

  22. Yu R, Kershaw J, Wang P, Zhang YM (2022) How to accurately monitor the weld penetration from dynamic weld pool serial images using CNN-LSTM deep learning model? IEEE Robot Autom Lett 7:. https://doi.org/10.1109/LRA.2022.3173659

  23. Jiao W, Wang Q, Cheng Y, et al (2020) Prediction of weld penetration using dynamic weld pool arc images. researchgate.netW Jiao, Q Wang, Y Cheng, R Yu, Y ZhangWeld J, 2020•researchgate.net. https://doi.org/10.29391/2020.99.027

  24. Liu T, Wang J, Huang X, et al (2022) 3DSMDA-Net: an improved 3DCNN with separable structure and multi-dimensional attention for welding status recognition. J Manuf Syst 62:. https://doi.org/10.1016/j.jmsy.2021.01.017

  25. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. https://doi.org/10.1109/CVPR.2017.243

  26. Liu Z, Lin Y, Cao Y, et al (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE International Conference on Computer Vision. https://doi.org/10.48550/arXiv.2103.14030

  27. Dosovitskiy A, Beyer L, Kolesnikov A, et al (2021) An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR 2021 - 9th International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2010.11929

  28. Gao P, Wu Z, Wang Y, et al (2023) Method for monitoring and controlling penetration of complex groove welding based on online multi-modal data. J Intell Manuf. https://doi.org/10.1007/s10845-023-02107-2

Download references

Funding

This work was supported by the National Natural Science Foundation of China (grant nos. 62101265, U23A20283, 62271263), the China Postdoctoral Science Foundation (2021M691592), and the Fundamental Research Funds for the Central Universities (No. 30922010705).

Author information

Authors and Affiliations

  1. Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing, 210094, China

    Zhuang Zhao, Peng Gao, Jun Lu & Lianfa Bai

Authors
  1. Zhuang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lianfa Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhuang Zhao.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Recommended for publication by Commission I - Additive Manufacturing, Surfacing, and Thermal Cutting

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Z., Gao, P., Lu, J. et al. Dynamic penetration prediction based on continuous video learning. Weld World 68, 867–877 (2024). https://doi.org/10.1007/s40194-024-01745-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40194-024-01745-1

Keywords

Search

Navigation