Barcodes as Summary of Loss Function Topology

Abstract

We propose to study neural networks’ loss surfaces by methods of topological data analysis. We suggest to apply barcodes of Morse complexes to explore topology of loss surfaces. An algorithm for calculations of the loss function’s barcodes of local minima is described. We have conducted experiments for calculating barcodes of local minima for benchmark functions and for loss surfaces of small neural networks. Our experiments confirm our two principal observations for neural networks’ loss surfaces. First, the barcodes of local minima are located in a small lower part of the range of values of neural networks’ loss function. Secondly, increase of the neural network’s depth and width lowers the barcodes of local minima. This has some natural implications for the neural network’s learning and for its generalization properties.

APPENDIX

1.1 GRADIENT MORSE COMPLEX

The gradient Morse complex $({{C}_{*}},{{\partial }_{*}})$, is defined as follows. For generic f  the critical points ${{p}_{\alpha }}$, $df{{{\text{|}}}_{{{{T}_{{{{p}_{\alpha }}}}}}}} = 0$, are isolated. Near each critical point ${{p}_{\alpha }}$ f can be written as $f = \sum\nolimits_{l = 1}^j - {{({{x}^{l}})}^{2}} + \sum\nolimits_{l = j}^n {{({{x}^{l}})}^{2}}$ in some local coordinates. The index of the critical point is defined as the dimension of the set of downward pointing directions at that point, or of the negative subspace of the Hessian:

$${\text{index}}({{p}_{\alpha }}) = j,$$

Then define

$${{C}_{j}} = {{ \oplus }_{{{\text{index}}({{p}_{\alpha }}) = j}}}[{{p}_{\alpha }},{\text{or}}(T_{{{{l}_{\alpha }}}}^{ - })],$$

where or is an orientation on a negative subspace ${{T}_{{{{p}_{\alpha }}}}} = T_{{{{p}_{\alpha }}}}^{ - } \oplus T_{{{{p}_{\alpha }}}}^{ + }$ of the Hessian ${{\partial }^{2}}f$.

Let

$$\begin{gathered} \mathcal{M}({{p}_{\alpha }},{{p}_{\beta }}) = \{ \gamma :\mathbb{R} \to {{M}^{n}}|\dot {\gamma } = - ({\text{gra}}{{{\text{d}}}_{g}}f)(\gamma (t)), \\ \mathop {\lim }\limits_{t \to - \infty } = {{p}_{\alpha }},\mathop {\lim }\limits_{t \to + \infty } = {{p}_{\beta }}{\text{\} /}}\mathbb{R}{\text{,}} \\ \end{gathered} $$

be the set of gradient trajectories connecting critical points ${{p}_{\alpha }}$ and ${{p}_{\beta }}$, where the natural action of $\mathbb{R}$ is by the shift $\gamma (t) \mapsto \gamma (t + \tau )$.

If ${\text{index}}({{p}_{\beta }}) = {\text{index}}({{p}_{\alpha }}) - 1$ then generically the set $\mathcal{M}({{p}_{\alpha }},{{p}_{\beta }})$ is finite. Let

$$\# \mathcal{M}(\left[ {{{p}_{\alpha }},{\text{or}}} \right],\left[ {{{p}_{\beta }},{\text{or}}} \right]),$$

denote in this case the number of the trajectories, counted with signs taking into account a choice of orientation, between critical points ${{p}_{\alpha }}$ and ${{p}_{\beta }}$.

The linear operator ${{\partial }_{j}}$ is defined by

$${{\partial }_{j}}\left[ {{{p}_{\alpha }},{\text{or}}} \right] = \sum\limits_{{\text{index}}({{p}_{\beta }}) = j - 1} \left[ {{{p}_{\beta }},{\text{or}}} \right]\# \mathcal{M}({{p}_{\alpha }},{{p}_{\beta }}).$$

The description of the critical points on manifold $\Theta$ with nonempty boundary $\partial \Theta$ is modified slightly in the following way. A connected component of sublevel set is born also at a local minimum of restriction of f to the boundary ${{\left. f \right|}_{{\partial \Theta }}}$, if ${\text{grad}}f$ is pointed inside manifold $\Theta$. The merging of two connected components can also happen at 1-saddle of ${{\left. f \right|}_{{\partial \Theta }}}$, if ${\text{grad}}f$ is pointed inside $\Theta$. When we speak about minima and 1-saddles, this also means such critical points of ${{\left. f \right|}_{{\partial \Theta }}}$. Similarly the set of generators of index $j$ chains in Morse complex includes index $j$ critical points of ${{\left. f \right|}_{{\partial \Theta }}}$ with ${\text{grad}}f$ pointed inside $\Theta$. The differential is also modified similarly to take into account trajectories involving such critical points.

1.2 PROOF OF THEOREM 3

Theorem. ([7], Section 2) Any $\mathbb{R}$-filtered chain complex ${{C}_{*}}$ over field $k$ can be brought by a linear transformation preserving the $\mathbb{R}$-filtration to “canonical form”, a canonically defined direct sum of indecomposable $\mathbb{R}$-filtered complexes of two types:

• 1-dimensional $\mathbb{R}$-filtered complex with trivial differential, $\partial \tilde {e}_{i}^{{(j)}} = 0$ , $\left\langle {\tilde {e}_{i}^{{(j)}}} \right\rangle = {{F}_{{ \leqslant r}}}$, $r \in \mathbb{R}$,

• 2-dimensional $\mathbb{R}$-filtered complex with trivial homology $\partial \tilde {e}_{{{{i}_{2}}}}^{{(j + 1)}} = \tilde {e}_{{{{i}_{1}}}}^{{(j)}}$, $\left\langle {\tilde {e}_{{{{i}_{1}}}}^{{(j)}}} \right\rangle = {{F}_{{ \leqslant {{s}_{1}}}}}$, $\left\langle {\tilde {e}_{{{{i}_{1}}}}^{{(j)}},\tilde {e}_{{{{i}_{2}}}}^{{(j + 1)}}} \right\rangle$ = ${{F}_{{ \leqslant {{s}_{2}}}}}$, ${{s}_{1}},{\kern 1pt} {{s}_{2}} \in \mathbb{R}$.

The resulting canonical form is unique.

Proof. ([7], Section 2). Let $\{ e_{i}^{{(n)}}\}$ be a basis in the vector spaces C_n compatible with the filtration, so that each subspace ${{F}_{r}}{{C}_{n}}$ is the span $\left\langle {e_{1}^{{(n)}}, \ldots ,e_{{{{i}_{r}}}}^{{(n)}}} \right\rangle$. Notice that the filtration defines the natural order on the set of basis elements.

Let $\partial e_{l}^{{(n)}}$ have the required form for $n = j$ and $l \leqslant i$, or $n < j$ and all l; i.e., either $\partial e_{l}^{{(n)}} = 0$ or $\partial e_{l}^{{(n)}} = e_{{m(l)}}^{{(n - 1)}}$, where $m(l) \ne m(l')$ for $l \ne l'$.

Let

$$\partial e_{{i + 1}}^{{(j)}} = \sum\limits_k e_{k}^{{(j - 1)}}{{\alpha }_{k}}.$$

Let’s move all the terms with $e_{k}^{{(j - 1)}} = \partial e_{q}^{j}$, $q \leqslant i$, from the right to the left side. We get

$$\partial \left( {e_{{i + 1}}^{{(j)}} - \sum\limits_{q \leqslant i} e_{q}^{{(j)}}{{\alpha }_{{k(q)}}}} \right) = \sum\limits_k e_{k}^{{(j - 1)}}{{\beta }_{k}}.$$

If ${{\beta }_{k}} = 0$ for all k, then define

$$\tilde {e}_{{i + 1}}^{{(j)}} = e_{{i + 1}}^{{(j)}} - \sum\limits_{q \leqslant i} e_{q}^{{(j)}}{{\alpha }_{{k(q)}}},$$

so that

$$\partial \tilde {e}_{{i + 1}}^{{(j)}} = 0,$$

and $\partial e_{l}^{{(n)}}$ has the required form for $l \leqslant i + 1$ and $n = j$, and for $n < j$ and all l.

Otherwise let ${{k}_{0}}$ be the maximal k with ${{\beta }_{k}} \ne 0$. Then

$$\partial \left( {e_{{i + 1}}^{{(j)}} - \sum\limits_{q \leqslant i} e_{q}^{{(j)}}{{\alpha }_{{k(q)}}}} \right) = e_{{{{k}_{0}}}}^{{(j - 1)}}{{\beta }_{{{{k}_{0}}}}} + \sum\limits_{k < {{k}_{0}}} e_{k}^{{(j - 1)}}{{\beta }_{k}},$$

${{\beta }_{{{{k}_{0}}}}} \ne 0.$ Define

$$\tilde {e}_{{i + 1}}^{{(j)}} = \left( {e_{{i + 1}}^{{(j)}} - \sum\limits_{q \leqslant i} e_{q}^{{(j)}}{{\alpha }_{{k(q)}}}} \right){\text{/}}{{\beta }_{{{{k}_{0}}}}},$$

$$\tilde {e}_{{{{k}_{0}}}}^{{(j - 1)}} = e_{{{{k}_{0}}}}^{{(j - 1)}} + \sum\limits_{k < {{k}_{0}}} e_{k}^{{(j - 1)}}{{\beta }_{k}}{\text{/}}{{\beta }_{{{{k}_{0}}}}}.$$

Then

$$\partial \tilde {e}_{{i + 1}}^{{(j)}} = \tilde {e}_{{{{k}_{0}}}}^{{(j - 1)}}$$

and for n = j and $l \leqslant i + 1$, or $n < j$ and all l, $\partial e_{l}^{{(n)}}$ has the required form. If the complex has been reduced to “canonical form” on subcomplex ${{ \oplus }_{{n \leqslant j}}}{{C}_{n}}$, then reduce similarly $\partial e_{1}^{{(j + 1)}}$ and so on.

Uniqueness of the canonical form follows essentially from the uniqueness at each previous step. Let $\left\{ {a_{i}^{{(j)}}} \right\}$, $\left\{ {b_{i}^{{(j)}} = \sum\nolimits_{k \leqslant i} a_{k}^{{(j)}}{{\alpha }_{k}}} \right\}$ be two bases of ${{C}_{*}}$ for two different canonical forms. Assume that for all indexes $p < j$ and all $n$, and $p = j$ and $n \leqslant i$ the canonical forms agree. Let $\partial a_{{i + 1}}^{{(j)}} = a_{m}^{{(j - 1)}}$ and $\partial b_{{i + 1}}^{{(j)}} = b_{l}^{{(j - 1)}}$ with $m > l$, $a_{m}^{{(j - 1)}}$ is not in the filtration subspace corresponding to $b_{l}^{{(j - 1)}}$.

It follows that

$$\partial \left( {\sum\limits_{k \leqslant i + 1} a_{k}^{{(j)}}{{\alpha }_{k}}} \right) = \sum\limits_{n \leqslant l} a_{n}^{{(j - 1)}}{{\beta }_{n}},$$

where ${{\alpha }_{{i + 1}}} \ne 0$, ${{\beta }_{l}} \ne 0$. Therefore

$$\partial a_{{i + 1}}^{{(j)}} = \sum\limits_{n \leqslant l} a_{n}^{{(j - 1)}}{{\beta }_{n}}{\text{/}}{{\alpha }_{{i + 1}}} - \sum\limits_{k \leqslant i} \partial a_{k}^{{(j)}}{{\alpha }_{k}}{\text{/}}{{\alpha }_{{i + 1}}}.$$

On the other hand $\partial a_{{i + 1}}^{{(j)}} = a_{m}^{{(j - 1)}}$, with $m > l$, and $\partial a_{k}^{{(j)}}$ for $k \leqslant i$ are either zero or some basis elements $a_{n}^{{(j - 1)}}$ different from $a_{m}^{{(j - 1)}}$. This gives a contradiction.

Similarly if $\partial b_{{i + 1}}^{{(j)}} = 0$, then

$$\partial a_{{i + 1}}^{{(j)}} = - \sum\limits_{k \leqslant i} \partial a_{k}^{{(j)}}{{\alpha }_{k}}{\text{/}}{{\alpha }_{{i + 1}}},$$

which again gives a contradiction by the same arguments. This proves the uniqueness of the canonical form.

Remark 2. The barcode of $\mathbb{R}$-filtered chain complex consists of segments representing the indecomposable $\mathbb{R}$-filtered chain complexes, see Definition 2 in Section 2. There is the standard lexicographic order on a set of such segments. The direct sum from the theorem statement is the standard “direct sum over a set” vector space.