Neural networks offer good approximation to many tasks but consistently fail to reach perfect generalization, even when theoretical work shows that such perfect solutions can be expressed by certain architectures. Using the task of formal language learning, we focus on one simple formal language and show that the theoretically correct solution is in fact not an optimum of commonly used objectives -- even with regularization techniques that according to common wisdom should lead to simple weights and good generalization (L1, L2) or other meta-heuristics (early-stopping, dropout). However, replacing standard targets with the Minimum Description Length objective (MDL) results in the correct solution being an optimum.

中文翻译：

---

使用最小描述长度弥合神经网络形式语言学习中的经验与理论差距

神经网络对许多任务提供了很好的近似，但始终无法达到完美的泛化，即使理论工作表明这种完美的解决方案可以通过某些架构来表达。使用形式语言学习的任务，我们专注于一种简单的形式语言，并表明理论上正确的解决方案实际上并不是常用目标的最佳方案 - 即使使用根据常识应该导致简单权重和良好的正则化技术泛化（L1、L2）或其他元启发式（提前停止、退出）。但是，用最小描述长度目标 (MDL) 替换标准目标会导致正确的解决方案成为最佳解决方案。