Past decades have witnessed a great interest in the distinction and connection between neural network learning and kernel learning. Recent advancements have made theoretical progress in connecting infinite-wide neural networks and Gaussian processes. Two predominant approaches have emerged: the Neural Network Gaussian Process (NNGP) and the Neural Tangent Kernel (NTK). The former, rooted in Bayesian inference, represents a zero-order kernel, while the latter, grounded in the tangent space of gradient descents, is a first-order kernel. In this paper, we present the Unified Neural Kernel (UNK), which characterizes the learning dynamics of neural networks with gradient descents and parameter initialization. The proposed UNK kernel maintains the limiting properties of both NNGP and NTK, exhibiting behaviors akin to NTK with a finite learning step and converging to NNGP as the learning step approaches infinity. Besides, we also theoretically characterize the uniform tightness and learning convergence of the UNK kernel, providing comprehensive insights into this unified kernel. Experimental results underscore the effectiveness of our proposed method.

中文翻译：

---

神经网络学习的统一内核

在过去的几十年里，人们对神经网络学习和内核学习之间的区别和联系产生了极大的兴趣。最近的进展在连接无限宽神经网络和高斯过程方面取得了理论进展。出现了两种主要方法：神经网络高斯过程（NNGP）和神经正切核（NTK）。前者植根于贝叶斯推理，代表零阶核，而后者则植根于梯度下降的切线空间，是一阶核。在本文中，我们提出了统一神经核（UNK），它通过梯度下降和参数初始化来表征神经网络的学习动态。所提出的 UNK 内核保持了 NNGP 和 NTK 的限制属性，在有限的学习步骤中表现出类似于 NTK 的行为，并随着学习步骤接近无穷大而收敛到 NNGP。此外，我们还从理论上描述了 UNK 内核的均匀紧密性和学习收敛性，为这个统一内核提供了全面的见解。实验结果强调了我们提出的方法的有效性。