Neural networks are trained by minimizing a loss function that defines the discrepancy between the predicted model output and the target value. The selection of the loss function is crucial to achieve task-specific behaviour and highly influences the capability of the model. A variety of loss functions have been proposed for a wide range of tasks affecting training and model performance. For classification tasks, the cross entropy is the de-facto standard and usually the first choice. Here, we try to experimentally challenge the well-known loss functions, including cross entropy (CE) loss, by utilizing the genetic programming (GP) approach, a population-based evolutionary algorithm. GP constructs loss functions from a set of operators and leaf nodes and these functions are repeatedly recombined and mutated to find an optimal structure. Experiments were carried out on different small-sized datasets CIFAR-10, CIFAR-100 and Fashion-MNIST using an Inception model. The 5 best functions found were evaluated for different model architectures on a set of standard datasets ranging from 2 to 102 classes and very different sizes. One function, denoted as Next Generation Loss (NGL), clearly stood out showing same or better performance for all tested datasets compared to CE. To evaluate the NGL function on a large-scale dataset, we tested its performance on the Imagenet-1k dataset where it showed improved top-1 accuracy compared to models trained with identical settings and other losses. Finally, the NGL was trained on a segmentation downstream task for Pascal VOC 2012 and COCO-Stuff164k datasets improving the underlying model performance.

中文翻译：

---

用于图像分类的下一代损失函数

通过最小化定义预测模型输出和目标值之间差异的损失函数来训练神经网络。损失函数的选择对于实现特定于任务的行为至关重要，并且高度影响模型的能力。人们已经针对影响训练和模型性能的各种任务提出了各种损失函数。对于分类任务，交叉熵是事实上的标准，通常是首选。在这里，我们尝试利用遗传编程（GP）方法（一种基于群体的进化算法）对众所周知的损失函数进行实验挑战，包括交叉熵（CE）损失。 GP 从一组算子和叶节点构造损失函数，这些函数被反复重组和变异以找到最佳结构。使用 Inception 模型在不同的小型数据集 CIFAR-10、CIFAR-100 和 Fashion-MNIST 上进行了实验。我们在一组标准数据集上针对不同模型架构对找到的 5 个最佳函数进行了评估，数据集的类别范围从 2 到 102 个类别，大小也截然不同。一个被称为下一代损失 (NGL) 的函数明显突出，显示与 CE 相比，所有测试数据集具有相同或更好的性能。为了在大规模数据集上评估 NGL 函数，我们在 Imagenet-1k 数据集上测试了其性能，与使用相同设置和其他损失训练的模型相比，它显示出更高的 top-1 准确性。最后，NGL 接受了 Pascal VOC 2012 和 COCO-Stuff164k 数据集的分割下游任务的训练，提高了底层模型的性能。