SIAM/ASA Journal on Uncertainty Quantification, Volume 12, Issue 1, Page 69-100, March 2024.
Abstract. Variational inference is an approximation framework for Bayesian inference that seeks to improve quantified uncertainty in predictions by optimizing a simplified distribution over parameters to stand in for the full posterior. Capturing model variations that remain consistent with training data enables more robust predictions by reducing parameter sensitivity. This work introduces a fixed-point optimization for variational inference that is applicable when every feasible log density can be expressed as a linear combination of functions from a given basis. In such cases, the optimizer becomes a fixed-point of projective integral updates. When the basis spans univariate quadratics in each parameter, the feasible distributions are Gaussian mean-fields and the projective integral updates yield quasi-Newton variational Bayes (QNVB). Other bases and updates are also possible. Since these updates require high-dimensional integration, this work begins by proposing an efficient quasirandom sequence of quadratures for mean-field distributions. Each iterate of the sequence contains two evaluation points that combine to correctly integrate all univariate quadratic functions and, if the mean-field factors are symmetric, all univariate cubics. More importantly, averaging results over short subsequences achieves periodic exactness on a much larger space of multivariate polynomials of quadratic total degree. The corresponding variational updates require four loss evaluations with standard (not second-order) backpropagation to eliminate error terms from over half of all multivariate quadratic basis functions. This integration technique is motivated by first proposing stochastic blocked mean-field quadratures, which may be useful in other contexts. A PyTorch implementation of QNVB allows for better control over model uncertainty during training than competing methods. Experiments demonstrate superior generalizability for multiple learning problems and architectures.

中文翻译：

---

高维变分推理的投影积分更新

SIAM/ASA 不确定性量化杂志，第 12 卷，第 1 期，第 69-100 页，2024 年 3 月。
摘要。变分推理是贝叶斯推理的一种近似框架，旨在通过优化参数的简化分布来代替完整的后验，从而改善预测中的量化不确定性。捕获与训练数据保持一致的模型变化可以通过降低参数敏感性来实现更稳健的预测。这项工作引入了变分推理的定点优化，当每个可行的对数密度都可以表示为给定基础的函数的线性组合时，该优化适用。在这种情况下，优化器成为投影积分更新的定点。当每个参数的基跨越单变量二次方程时，可行分布是高斯平均场，并且投影积分更新产生拟牛顿变分贝叶斯 (QNVB)。其他基础和更新也是可能的。由于这些更新需要高维积分，因此这项工作首先提出平均场分布的有效准随机正交序列。序列的每次迭代都包含两个评估点，它们结合起来可以正确积分所有单变量二次函数，并且如果平均场因子对称，则可以正确积分所有单变量三次函数。更重要的是，对短子序列的结果进行平均可以在更大的二次总次数多元多项式空间上实现周期性精确性。相应的变分更新需要使用标准（非二阶）反向传播进行四次损失评估，以消除所有多元二次基函数一半以上的误差项。这种积分技术的动机是首先提出随机分块平均场求积，这在其他情况下可能很有用。与竞争方法相比，QNVB 的 PyTorch 实现可以更好地控制训练期间的模型不确定性。实验证明了对多种学习问题和架构的卓越通用性。