Stability of accuracy for the training of DNNs via the uniform doubling condition

Abstract

We study the stability of accuracy during the training of deep neural networks (DNNs). In this context, the training of a DNN is performed via the minimization of a cross-entropy loss function, and the performance metric is accuracy (the proportion of objects that are classified correctly). While training results in a decrease of loss, the accuracy does not necessarily increase during the process and may sometimes even decrease. The goal of achieving stability of accuracy is to ensure that if accuracy is high at some initial time, it remains high throughout training. A recent result by Berlyand, Jabin, and Safsten introduces a doubling condition on the training data, which ensures the stability of accuracy during training for DNNs using the absolute value activation function. For training data in $\mathbb {R}^n$, this doubling condition is formulated using slabs in $\mathbb {R}^n$ and depends on the choice of the slabs. The goal of this paper is twofold. First, to make the doubling condition uniform, that is, independent of the choice of slabs. This leads to sufficient conditions for stability in terms of training data only. In other words, for a training set T that satisfies the uniform doubling condition, there exists a family of DNNs such that a DNN from this family with high accuracy on the training set at some training time $t_0$ will have high accuracy for all time $t>t_0$. Moreover, establishing uniformity is necessary for the numerical implementation of the doubling condition. We demonstrate how to numerically implement a simplified version of this uniform doubling condition on a dataset and apply it to achieve stability of accuracy using a few model examples. The second goal is to extend the original stability results from the absolute value activation function to a broader class of piecewise linear activation functions with finitely many critical points, such as the popular Leaky ReLU.

Appendix A

1.1 Proof of Lemma 6.2

Suppose $\mu (P_1) \ne \varnothing$. If either $(1+\sigma )\,\mu (\varepsilon \,\bar{S})\geqslant \delta$ or $(1+\sigma )\,\mu (\varepsilon \,\bar{S}')\geqslant \delta$, we are done.

If both $(1+\sigma )\,\mu (\varepsilon \,\bar{S})<\delta$ and $(1+\sigma )\,\mu (\varepsilon \,\bar{S}')<\delta$. Then

$$\begin{aligned} \mu (g^{-1}(\kappa \,\varepsilon \,S))&=\mu (S^1_{\kappa \,\varepsilon })+\mu (S^0_{\kappa \,\varepsilon }) +\mu (S^2_{\kappa \,\varepsilon })\\&\geqslant \frac{\mu (S^1_{\kappa \,\varepsilon })+\mu (S^1_{\varepsilon })}{2}+\mu (S^0_{\kappa \,\varepsilon }) +\frac{\mu (S^2_{\kappa \,\varepsilon }) +\mu (S^2_{\varepsilon }) }{2}\\&\geqslant \mu (S^1_{\varepsilon })+\mu (S^0_{\varepsilon }) +\mu (S^2_{\varepsilon })+\frac{\mu (S^1_{\kappa \,\varepsilon })+2\mu (S^0_{\kappa \,\varepsilon }) +\mu (S^2_{\kappa \,\varepsilon })- \mu (S^1_{\varepsilon })-2\,\mu (S^0_{\varepsilon }) -\mu (S^2_{\varepsilon })}{2}. \end{aligned}$$

Here we are using an averaging trick to deal with the complication that we only have one $S_\varepsilon ^0$ and $S_{\kappa \varepsilon }^0$. Applying now (42) and (43), we find that proving the doubling condition with $\sigma '=\sigma /2$. Meaning, for $\varepsilon =\ell _S \kappa ^i< \beta$ we have

$$\begin{aligned} \mu (g^{-1}(\kappa \,\varepsilon \,S))\geqslant \mu (g^{-1}(\varepsilon \,S)) +\frac{\sigma }{2}\,(\mu (\varepsilon \,\bar{S}) +\mu (\varepsilon \,\bar{S}')) \geqslant (1+\sigma /2)\, \mu (g^{-1}(\varepsilon \,S)) \end{aligned}$$

(1)

$$\begin{aligned} \mu (S_{\kappa \varepsilon })> \min \{\ \delta , (1+\frac{\sigma }{2})\mu (S_{\varepsilon }) \}\ \end{aligned}$$

(2)

The critical point of $\lambda _1$ at $x_1=0$ would then be the only complicated part of $\lambda _1$.$\square$

Fig. 28

Classification confidence for the toy DNN for the Example 2.1 at different stages of training

1.2 Proof of Lemma 6.5: upper bound on the loss

Proof

We know the DNN $\phi$ is of the form

$$\begin{aligned} \phi (s,\alpha )=\rho \circ X(s,\alpha )=\left( \frac{e^{X_1(s,\alpha )}}{\sum _{k=1}^{K}e^{X_k(s,\alpha )}},\cdots ,\frac{e^{X_K}}{\sum _{k=1}^{K}e^{X_k(s,\alpha )}}\right) . \end{aligned}$$

(3)

Thus, (63) gives

$$\begin{aligned} p_{i(s)}(s,\alpha )=\frac{e^{X_{i(s)}(s,\alpha )}}{\sum _{k=1}^K e^{X_k(s,\alpha )}}=\frac{1}{\sum _{k=1}^K e^{X_k(s,\alpha )-X_{i(s)}(s,\alpha )}}=\frac{1}{1+\sum _{k\ne i(s)} e^{X_k(s,\alpha )-X_{i(s)}(s,\alpha )}}. \end{aligned}$$

(4)

For each $k\ne i(s)$, $X_k(s,\alpha )-X_{i(s)}(s,\alpha )\leqslant \max _{k\ne i(s)} X_k(s,\alpha )- X_{i(s)}(s,\alpha )=-\delta X(s,\alpha )$. Using (6) and (64), we obtain the following estimates on $p_{i(s)}(s,\alpha )$:

$$\begin{aligned} \frac{1}{1+(K-1)e^{-\delta X(s,\alpha )}}\leqslant p_{i(s)}(s,\alpha )\leqslant \frac{1}{1+e^{-\delta X(s,\alpha )}}. \end{aligned}$$

(5)

Finally, (65) gives estimates on loss:

$$\begin{aligned} \sum _{s\in T}\mu (s)\log \left( 1+e^{-\delta X(s,\alpha )}\right) \leqslant \bar{L}(\alpha )\leqslant \sum _{s\in T}\mu (s)\log \left( 1+(K-1)e^{-\delta X(s,\alpha )}\right) . \end{aligned}$$

(6)

From (66) we get

$$\begin{aligned} \sum _{s\in G_{\eta ^*}^c(t)} \mu (s)\,\log (1+e^{-\eta ^*})\leqslant \bar{L}(t)\leqslant \bar{L}(t_0). \end{aligned}$$

(7)

$\square$