Skip to main content
SpringerLink
Log in

Natural Image Matting with Attended Global Context

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Image matting is to estimate the opacity of foreground objects from an image. A few deep learning based methods have been proposed for image matting and perform well in capturing spatially close information. However, these methods fail to capture global contextual information, which has been proved essential in improving matting performance. This is because a matting image may be up to several megapixels, which is too big for a learning-based network to capture global contextual information due to the limit size of a receptive field. Although uniformly downsampling the matting image can alleviate this problem, it may result in the degradation of matting performance. To solve this problem, we introduce a natural image matting with the attended global context method to extract global contextual information from the whole image, and to condense them into a suitable size for learning-based network. Specifically, we first leverage a deformable sampling layer to obtain condensed foreground and background attended images respectively. Then, we utilize a contextual attention layer to extract information related to unknown regions from condensed foreground and background images generated by a deformable sampling layer. Besides, our network predicts a background as well as the alpha matte to obtain more purified foreground, which contributes to better qualitative performance in composition. Comprehensive experiments show that our method achieves competitive performance on both Composition-1k and the alphamatting.com benchmark quantitatively and qualitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Sun J, Jia J Y, Tang C K, Shum H Y. Poisson matting. ACM Trans. Graphics, 2004, 23(3): 315–321. https://doi.org/10.1145/1015706.1015721.

    Article  Google Scholar 

  2. Levin A, Lischinski D, Weiss Y. A closed-form solution to natural image matting. IEEE Trans. Pattern Analysis and Machine Intelligence, 2008, 30(2): 228–242. https://doi.org/10.1109/TPAMI.2007.1177.

  3. Chuang Y Y, Curless B, Salesin D H et al. A Bayesian approach to digital matting. In Proc. the 2001 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), Dec. 2001, pp.264–271. https://doi.org/10.1109/CVPR.2001.990970.

  4. Aksoy Y, Aydin T O, Pollefeys M. Designing effective inter-pixel information flow for natural image matting. In Proc. the 2017 IEEE CVPR, Jul. 2017, pp.228–236. https://doi.org/10.1109/CVPR.2017.32.

  5. Wang J, Cohen M F. Optimized color sampling for robust matting. In Proc. the 2007 IEEE CVPR, Jun. 2007. https://doi.org/10.1109/CVPR.2007.383006.

  6. Varnousfaderani E S, Rajan D. Weighted color and texture sample selection for image matting. IEEE Trans. Image Processing, 2013, 22(11): 4260–4270. https://doi.org/10.1109/TIP.2013.2271549.

    Article  MathSciNet  MATH  Google Scholar 

  7. Chen X, He F Z, Yu H P. A matting method based on full feature coverage. Multimedia Tools and Applications, 2019, 78(9): 11173–11201. https://doi.org/10.1007/s11042-018-6690-1.

    Article  Google Scholar 

  8. He K M, Rhemann C, Rother C, Tang X O, Sun J. A global sampling method for alpha matting. In Proc. the 24th IEEE CVPR, Jun. 2011, pp.2049–2056. https://doi.org/10.1109/CVPR.2011.5995495.

  9. Gastal E S L, Oliveira M M. Shared sampling for real-time alpha matting. Computer Graphics Forum, 2010, 29(2): 575–584. https://doi.org/10.1111/j.1467-8659.2009.01627.x.

    Article  Google Scholar 

  10. Shahrian E, Rajan D, Price B, Cohen S. Improving image matting using comprehensive sampling sets. In Proc. the 2013 IEEE CVPR, Jun. 2013, pp.636–643. https://doi.org/10.1109/CVPR.2013.88.

  11. Cho D, Tai Y W, Kweon I. Natural image matting using deep convolutional neural networks. In Proc. the 14th European Conference on Computer Vision, Oct. 2016, pp.626–643. https://doi.org/10.1007/978-3-319-46475-6_39.

  12. Lutz S, Amplianitis K, Smolic A. AlphaGAN: Generative adversarial networks for natural image matting. In Proc. the 2018 British Machine Vision Conference, Sept. 2018. https://doi.org/10.48550/arXiv.1807.10088.

  13. Chen Q, Ge T Z, Xu Y Y et al. Semantic human matting. In Proc. the 26th ACM Int. Conf. Multimedia, Oct. 2018, pp.618–626. https://doi.org/10.1145/3240508.3240610.

  14. Chen G Y, Han K, Wong K Y K. TOM-NET: Learning transparent object matting from a single image. In Proc. the 2018 IEEE/CVF Conf. CVPR, Jun. 2018, pp.9233–9241. https://doi.org/10.1109/CVPR.2018.00962.

  15. Xu N, Price B, Cohen S, Huang T. Deep image matting. In Proc. the 2017 IEEE CVPR, Jul. 2017, pp.311–320. https://doi.org/10.1109/CVPR.2017.41.

  16. Wang Y, Niu Y, Duan P Y, Lin J W, Zheng Y J. Deep propagation based image matting. In Proc. the 27th International Joint Conference on Artificial Intelligence, Jul. 2018, pp.999–1006. https://doi.org/10.24963/ijcai.2018/139.

  17. Zhang Y K, Gong L X, Fan L B et al. A late fusion CNN for digital matting. In Proc. the 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Jun. 2019, pp.7461–7470. https://doi.org/10.1109/CVPR.2019.00765.

  18. Cai S F, Zhang X S, Fan H Q et al. Disentangled image matting. In Proc. the 2019 IEEE/CVF Int. Conf. Computer Vision, Oct. 27–Nov. 2, 2019, pp.8818–8827. https://doi.org/10.1109/ICCV.2019.00891.

  19. Hou Q Q, Liu F. Context-aware image matting for simultaneous foreground and alpha estimation. In Proc. the 2019 IEEE/CVF Int. Conf. Computer Vision, Oct. 27–Nov. 2, 2019, pp.4129–4138. https://doi.org/10.1109/ICCV.2019.00423.

  20. Lu H, Dai Y T, Shen C H, Xu S C. Indices matter: Learning to index for deep image matting. In Proc. the 2019 IEEE/CVF Int. Conf. Computer Vision, Oct. 27–Nov. 2, 2019, pp.3265–3274. https://doi.org/10.1109/ICCV.2019.00336.

  21. Tang J W, Aksoy Y, Öztireli C, Gross M, Aydin T O. Learning-based sampling for natural image matting. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.3055–3063. https://doi.org/10.1109/CVPR.2019.00317.

  22. Chen Q, Li D, Tang C K. KNN matting. In Proc. the 2012 IEEE CVPR, Jun. 2012, pp.869–876. https://doi.org/10.1109/CVPR.2012.6247760.

  23. Zheng Y J, Kambhamettu C. Learning based digital matting. In Proc. the 12th IEEE International Conference on Computer Vision, Sept. 29–Oct. 2, 2009, pp.889–896. https://doi.org/10.1109/ICCV.2009.5459326.

  24. Li Y Y, Lu H T. Natural image matting via guided contextual attention. In Proc. the 34th AAAI Conference on Artificial Intelligence, Apr. 2020, pp.11450–11457. https://doi.org/10.1609/aaai.v34i07.6809.

  25. Yu J H, Lin Z, Yang J M et al. Generative image inpainting with contextual attention. In Proc. the 2018 IEEE/CVF Conf. CVPR, Jun. 2018, pp.5505–5514. https://doi.org/10.1109/CVPR.2018.00577.

  26. Rhemann C, Rother C, Wang J et al. A perceptually motivated online benchmark for image matting. In Proc. the 2009 IEEE CVPR, Jun. 2009, pp.1826–1833. https://doi.org/10.1109/CVPR.2009.5206503.

  27. Lee P, Wu Y. Nonlocal matting. In Proc. the 24th IEEE Conf. Computer Vision and Pattern Recognition, Jun. 2011, pp.2193–2200. https://doi.org/10.1109/CVPR.2011.5995665.

  28. Shen X Y, Tao X, Gao H Y, Zhou C, Jia J Y. Deep automatic portrait matting. In Proc. the 14th European Conference on Computer Vision, Oct. 2016, pp.92–107. https://doi.org/10.1007/978-3-319-46448-0_6.

  29. Chen L C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. arXiv: 1706.05587, 2017. http://arxiv.org/abs/1706.05587, May 2023.

  30. Wang X L, Girshick R, Gupta A, He K M. Non-local neural networks. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp.7794–7803. https://doi.org/10.1109/CVPR.2018.00813.

  31. Luo W J, Li Y J, Urtasun R, Zemel R S. Understanding the effective receptive field in deep convolutional neural networks. In Proc. the 30th International Conference on Neural Information Processing Systems, Dec. 2016, pp.4905–4913. https://doi.org/10.5555/3157382.3157645.

  32. Recasens A, Kellnhofer P, Stent S et al. Learning to zoom: A saliency-based sampling layer for neural networks. In Proc. the 15th European Conf. Computer Vision, Sept. 2018, pp.52–67. https://doi.org/10.1007/978-3-030-01240-3_4.

  33. Zhou F F, Tian Y J, Qi Z Q. Attention transfer network for nature image matting. IEEE Trans. Circuits and Systems for Video Technology, 2021, 31(6): 2192–2205. https://doi.org/10.1109/TCSVT.2020.3024213.

    Article  Google Scholar 

  34. Dai Y T, Lu H, Shen C H. Learning affinity-aware up-sampling for deep image matting. In Proc. the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2021, pp.6841–6850. https://doi.org/10.1109/CVPR46437.2021.00677.

  35. Yu H C, Xu N, Huang Z L, Zhou Y Q, Shi H. High-resolution deep image matting. In Proc. the 35th AAAI Conference on Artificial Intelligence, May 2021, pp.3217–3224. https://doi.org/10.1609/aaai.v35i4.16432.

  36. Kingma D P, Ba J. Adam: A method for stochastic optimization. In Proc. the 3rd International Conference on Learning Representations, May 2015.

  37. Sun Y N, Tang C K, Tai Y W. Semantic image matting. In Proc. the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2021, pp.11120–11129. https://doi.org/10.1109/CVPR46437.2021.01097.

Download references

Author information

Authors and Affiliations

  1. MoE Key Lab of Artificial Intelligence, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Yi-Yi Zhang, Li Niu, Jian-Fu Zhang, Wei-Jie Zhao & Li-Qing Zhang

  2. Department of Intelligent Media, Institute of Scientific and Industrial Research, Osaka University, Osaka, 565-0871, Japan

    Yi-Yi Zhang, Yasushi Makihara & Yasushi Yagi

Authors
  1. Yi-Yi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yasushi Makihara
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian-Fu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei-Jie Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yasushi Yagi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Li-Qing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi-Yi Zhang.

Supplementary Information

ESM 1

(PDF 334 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, YY., Niu, L., Makihara, Y. et al. Natural Image Matting with Attended Global Context. J. Comput. Sci. Technol. 38, 659–673 (2023). https://doi.org/10.1007/s11390-022-1690-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-022-1690-z

Keywords

Search

Navigation