Abstract
When using Wasserstein GAN loss function for training generative adversarial networks (GAN), it is theoretically necessary to limit the discriminators’ expressive power (so-called discriminator normalization). Such limitation increases the stability of GAN training at the expense of a less expressive final model. Spectral normalization is one of the normalization algorithms that involves applying a fixed operation independently to each discriminator layer. However, the optimal strength of the discriminator limitation varies for different tasks, which requires a parameterized normalization method. This paper proposes modifications to the spectral normalization algorithm that allow changing the strength of the discriminator limitation. In addition to parameterization, the proposed methods can change the degree of limitation during training, unlike the original algorithm. The quality of the obtained models is explored for each of the proposed methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
For a normed space V, henceforward, ||⋅||V will denote the norm in this space.
GitHub repository with the used implementation of the training procedure and measuring metrics: https://github.com/TrickmanOff/GAN_project
An epoch in this context is one pass through the entire training set when performing generator training steps.
Further in the text, for brevity, this metric will be called “conditional average PRD-AUC,” when it is clear what partition of the set of conditions we are talking about.
REFERENCES
S. Agostinelli et al., “Geant4—a simulation toolkit,” Nucl. Instrum. Methods Phys. Res. Sect. A: Accel. Spectrom. Detect. Assoc. Equip. 506 (3), 250–303 (2003). https://doi.org/10.1016/S0168-9002(03)01368-8
V. Chekalina et al., “Generative models for fast calorimeter simulation: The LHCb case,” EPJ Web Conf. 214, 02034 (2019). https://doi.org/10.1051/epjconf/201921402034
A. Rogachev and F. Ratnikov, “GAN with an auxiliary regressor for the fast simulation of the electromagnetic calorimeter response,” J. Phys.: Conf. Ser. 2438, 012086 (2023). https://doi.org/10.1088/1742-6596/2438/1/012086
M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in Proceedings of the 34th International Conference on Machine Learning, Proc. Mach. Learn. Res. 70, 214–223 (2017). https://proceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf
T. Miyato et al., “Spectral normalization for generative adversarial networks,” in International Conference on Learning Representations (2018). https://arxiv.org/abs/1802.05957
I. J. Goodfellow et al., “Generative adversarial nets,” in Advances in Neural Information Processing Systems (2014), Vol. 2, pp. 2672–2680. https://arxiv.org/pdf/1406.2661.pdf
C. Villani, Optimal Transport: Old and New (Springer, Berlin, 2016).
S. Hirose et al., “ABCAS: Adaptive bound control of spectral norm as automatic stabilizer,” in 2023 IEEE International Conference on Consumer Electronics (2023), pp. 1–5. https://doi.org/10.1109/ICCE56470.2023.10043368
M. S. M. Sajjadi et al. “Assessing generative models via precision and recall,” in Neural Information Processing (2018), pp. 5234–5243. https://arxiv.org/abs/1806.00035
P. S. Kostenetskiy, R. A. Chulkevich, and V. I. Ko-zyrev, “HPC resources of the Higher School of Economics,” J. Phys.: Conf. Ser. 1740 (2021). https://doi.org/10.1088/1742-6596/1740/1/012050
ACKNOWLEDGEMENTS
This study was carried out using the supercomputer complex of the National Research University Higher School of Economics [10]; the authors are grateful for providing access to it.
Funding
This work was supported by ongoing institutional funding. No additional grants to carry out or direct this particular research were obtained.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
The authors of this work declare that they have no conflicts of interest.
Additional information
Publisher’s Note.
Pleiades Publishing remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
APPENDIX
APPENDIX
1.1 ARCHITECTURE OF MODELS USED
Architecture of the generator used.
Architecture of the discriminator used.
Rights and permissions
About this article
Cite this article
Egorov, E.A., Rogachev, A.I. Adaptive Spectral Normalization for Generative Models. Dokl. Math. 108 (Suppl 2), S205–S214 (2023). https://doi.org/10.1134/S1064562423701089
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S1064562423701089