Yao Zeng
Skip Abstract Section
Abstract
We propose an end-to-end generative adversarial network that allows for controllable ink wash painting generation from sketches by specifying the colors via color hints. To the best of our knowledge, this is the first study for interactive Chinese ink wash painting colorization from sketches. To help our network understand the ink style and artistic conception, we introduced an ink style prediction mechanism for our discriminator, which enables the discriminator to accurately predict the style with the help of a pre-trained style encoder. We also designed our generator to receive multi-scale feature information from the feature pyramid network for detail reconstruction of ink wash painting. Experimental results and user study show that ink wash paintings generated by our network have higher realism and richer artistic conception than existing image generation methods.
- Elad Aharoni-Mack, Yakov Shambik, and Dani Lischinski. 2017. Pigment-based recoloring of watercolor paintings. In Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. 1–11.Google Scholar
Digital Library
- Mikołaj Bińkowski, Sutherland Danica J, Arbel Michael, and Gretton Arthur. 2018. Demystifying mmd gans. arXiv preprint arXiv:1801.01401(2018), 13–28.Google Scholar
- Caroline Chan, Durand Fredo, and Isola Phillip. 2022. Learning to generate line drawings that convey geometry and semantics. arXiv preprint arXiv:2203.12691(2022), 2054–2068.Google Scholar
- Nelson S-H Chu and Chiew-Lan Tai. 2005. Moxi: real-time ink dispersion in absorbent paper. ACM Transactions on Graphics (TOG) 24, 3 (2005), 504–511.Google ScholarDigital Library
- Jia Deng, Wei Dong, Socher Richard, Lijia Li, Kai Li, and Feifei Li. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.Google ScholarCross Ref
- Lixing Dong, Shufang Lu, and Xiaogang Jin. 2014. Real-time image-based Chinese ink painting rendering. Multimedia tools and applications 69, 3 (2014), 605–620.Google Scholar
- Zhi Dou, Ning Wang, Baopu Li, Zhihui Wang, Haojie Li, and Bin Liu. 2021. Dual Color Space Guided Sketch Colorization. IEEE Transactions on Image Processing 30 (2021), 7292–7304.Google ScholarCross Ref
- Feifei Fu, Jiancheng Lv, Chenwei Tang, and Mao Li. 2021. Multi-style Chinese art painting generation of flowers. IET Image Processing 15, 3 (2021), 746–762.Google ScholarCross Ref
- Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. 2020. Sketchycoco: Image generation from freehand scene sketches. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5174–5183.Google ScholarCross Ref
- Bin He, Feng Gao, Daiqian Ma, Boxin Shi, and Lingyu Duan. 2018. Chipgan: A generative adversarial network for chinese ink wash painting style transfer. In Proceedings of the 26th ACM international conference on Multimedia. 1172–1180.Google ScholarDigital Library
- Xun Huang, Mingyu Liu, Belongie Serge, and Kautz Jan. 2018. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV). 172–189.Google ScholarDigital Library
- Xun Huang and Belongie Serge. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision. 1501–1510.Google ScholarCross Ref
- Johnson Justin, Alahi Alexandre, and Feifei Li. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision. Springer, 694–711.Google Scholar
- Simonyanb Karen and Zisserman Andrew. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014), 567–587.Google Scholar
- Jintae Lee. 2001. Diffusion rendering of black ink paintings using new paper and ink models. Computers & Graphics 25, 2 (2001), 295–308.Google ScholarCross Ref
- Junsoo Lee, Eungyeup Kim, Yunsung Lee, Dongjun Kim, Jaehyuk Chang, and Jaegul Choo. 2020. Reference-based sketch image colorization using augmented-self reference and dense semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5801–5810.Google ScholarCross Ref
- Bo Li, Caiming Xiong, Tianfu Wu, Yu Zhou, Lun Zhang, and Rufeng Chu. 2018. Neural abstract style transfer for chinese traditional painting. In Asian Conference on Computer Vision. Springer, 212–227.Google Scholar
- Chengze Li, Xueting Liu, and Tientsin Wong. 2017. Deep extraction of manga structural lines. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.Google ScholarDigital Library
- Daoyu Lin, Yang Wang, Guangluan Xu, Jun Li, and Kun Fu. 2018. Transform a simple sketch to a chinese painting by a multiscale deep neural network. Algorithms 11, 1 (2018), 4.Google ScholarCross Ref
- Tsungyi Lin, Dollár Piotr, Girshick Ross, Kaiming He, Hariharan Bharath, and Belongie Serge. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2117–2125.Google ScholarCross Ref
- Thomas Lindemeier, Jörg Marvin Gülzow, and Oliver Deussen. 2018. Painterly rendering using limited paint color palettes. In Proceedings of the Conference on Vision, Modeling, and Visualization. 135–145.Google ScholarDigital Library
- Bingchen Liu, Kunpeng Song, Yizhe Zhu, and Elgammal Ahmed. 2020. Sketch-to-art: Synthesizing stylized art images from sketches. In Proceedings of the Asian Conference on Computer Vision. 125–149.Google Scholar
- Heusel Martin, Ramsauer Hubert, Unterthiner Thomas, Nessler Bernhard, and Hochreiter Sepp. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017), 678–689.Google Scholar
- Taesung Park, Mingyu Liu, Tingchun Wang, and Junyan Zhu. 2019. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2337–2346.Google ScholarCross Ref
- Isola Phillip, Junyan Zhu, Tinghui Zhou, and Efros Alexei A. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.Google Scholar
- Shaham Tamar Rott, Gharbi Michaël, Richard Zhang, Shechtman Eli, and Michaeli Tomer. 2021. Spatially-adaptive pixelwise networks for fast image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14882–14891.Google Scholar
- Hicsonmez Samet, Samet Nermin, Akbas Emre, and Duygulu Pinar. 2021. Adversarial Segmentation Loss for Sketch Colorization. In 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2403–2407.Google Scholar
- Meijun Sun, Jizhou Sun, and Bin Yun. 2005. Physical modeling of” Xuan” paper in the simulation of Chinese ink-wash drawing. In International Conference on Computer Graphics, Imaging and Visualization (CGIV’05). IEEE, 317–322.Google ScholarDigital Library
- Chungming Wang and Renjie Wang. 2007. Image-based color ink diffusion rendering. IEEE Transactions on Visualization and Computer Graphics 13, 2(2007), 235–246.Google ScholarDigital Library
- Tingchun Wang, Mingyu Liu, Junyan Zhu, Andrew Tao, Kautz Jan, and Catanzaro Bryan. 2018. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8798–8807.Google ScholarCross Ref
- Yuan Wang, Weibo Zhang, and Peng Chen. 2019. Chinastyle: A mask-aware generative adversarial network for chinese traditional image translation. In SIGGRAPH Asia 2019 Technical Briefs. 5–8.Google Scholar
- Der-Lor Way, Shengwen Huang, and Zenchung Shih. 2003. Physical-based Model of Ink Diffusion in Chinese Paintings. In WSCG. 215–226.Google Scholar
- Ning Xie, Laga Hamid, Saito Sugru, and Nakajima Masayuki. 2010. IR2s: interactive real photo to Sumi-e. In Proceedings of the 8th international symposium on non-photorealistic animation and rendering. 63–71.Google ScholarDigital Library
- Tianchen Xu, Lijie Yang, and Enhua Wu. 2012. Stroke-based real-time ink wash painting style rendering for geometric models. In SIGGRAPH Asia 2012 Technical Briefs. 1–4.Google Scholar
- Alice Xue. 2021. End-to-end chinese landscape painting creation using generative adversarial networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3863–3871.Google ScholarCross Ref
- Jinhui Yu, Guoming Luo, and Qunsheng Peng. 2003. Image-based synthesis of Chinese landscape painting. Journal of Computer Science and Technology 18, 1 (2003), 22–28.Google ScholarDigital Library
- Jiajing Zhang, Jinhui Yu, Yongwei Miu, and Ren Peng. 2021. Self-Adaptive Computational Aesthetic Evaluation of Chinese Ink Paintings Based on Deep Learning. Journal of Computer-Aided Design & Computer Graphics/Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao 33, 9 (2021), 16–25. (in Chinese).Google Scholar
- Lvmin Zhang, Yi Ji, Xin Lin, and Chunping Liu. 2017. Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier gan. In 2017 4th IAPR Asian conference on pattern recognition (ACPR). IEEE, 506–511.Google ScholarCross Ref
- Lvmin Zhang, Chengze Li, Tientsin Wong, Yi Ji, and Chunping Liu. 2018. Two-stage sketch colorization. ACM Transactions on Graphics (TOG)(2018), 1–14.Google Scholar
- Richard Zhang, Isola Phillip, Efros Alexei A, Shechtman Eli, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586–595.Google ScholarCross Ref
- Mingtian Zhao and Songchun Zhu. 2013. Abstract painting with interactive control of perceptual entropy. ACM Transactions on Applied Perception (TAP) 10, 1 (2013), 1–21.Google ScholarDigital Library
- Chengyu Zheng and Yuan Zhang. 2018. Two-stage color ink painting style transfer via convolution neural network. In 2018 15th International Symposium on Pervasive Systems, Algorithms and Networks (I-SPAN). IEEE, 193–200.Google ScholarCross Ref
- Le Zhou, Qiufeng Wang, Kaizhu Huang, and Chenghung Lo. 2019. An interactive and generative approach for chinese shanshui painting document. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 819–824.Google ScholarCross Ref
- Junyan Zhu, Taesung Park, Phillip Isola, and Efros Alexei A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223–2232.Google ScholarCross Ref
Index Terms
- Color Hint-guided Ink Wash Painting Colorization with Ink Style Prediction Mechanism
Recommendations
Simulation of Chinese Ink-Wash Painting Based on Landscapes and Trees
PMA '06: Proceedings of the 2006 International Symposium on Plant Growth Modeling, Simulation, Visualization and ApplicationsIn digital times, digital art results from the combining of art and science, technology and aesthetics. For ink-wash landscape painting simulated with 3D techniques, the significance is to open a new window for creation and aesthetics of Chinese ...
Style-woven Attention Network for Zero-shot Ink Wash Painting Style Transfer
ICMR '22: Proceedings of the 2022 International Conference on Multimedia RetrievalTraditional Chinese painting is a unique form of artistic expression. Compared with western art painting, it pays more attention to the verve in visual effect, especially ink painting, which makes good use of lines and pays little attention to ...
Oriental Color Ink Rendering for Landscape
ICIS '05: Proceedings of the Fourth Annual ACIS International Conference on Computer and Information ScienceResearchers of oriental color ink painting have concentrated on physical models such as brushes; papers, and ink diffusion. These models can be efjective when painter depict a painting by tablet pen. Abstraction is main notion of oriental color ink ...
Comments