Abstract

We propose to study neural networks’ loss surfaces by methods of topological data analysis. We suggest to apply barcodes of Morse complexes to explore topology of loss surfaces. An algorithm for calculations of the loss function’s barcodes of local minima is described. We have conducted experiments for calculating barcodes of local minima for benchmark functions and for loss surfaces of small neural networks. Our experiments confirm our two principal observations for neural networks’ loss surfaces. First, the barcodes of local minima are located in a small lower part of the range of values of neural networks’ loss function. Secondly, increase of the neural network’s depth and width lowers the barcodes of local minima. This has some natural implications for the neural network’s learning and for its generalization properties.

中文翻译：

---

条形码作为损失函数拓扑的总结

摘要

我们建议通过拓扑数据分析的方法来研究神经网络的损失面。我们建议应用莫尔斯复合体的条形码来探索损失表面的拓扑。描述了用于计算局部最小值的损失函数条形码的算法。我们进行了计算基准函数和小型神经网络损失表面的局部最小值条形码的实验。我们的实验证实了我们对神经网络损失表面的两个主要观察结果。首先，局部最小值的条形码位于神经网络损失函数值范围的一小部分。其次，神经网络深度和宽度的增加降低了局部最小值的条形码。这对于神经网络的学习及其泛化特性有一些自然的影响。