We present SIM-Sync , a certifiably optimal algorithm that estimates camera trajectory and 3D scene structure directly from multiview image keypoints . The key enabler of SIM-Sync is a pretrained depth prediction network. Given a graph with nodes representing monocular images taken at unknown camera poses and edges containing pairwise image keypoint correspondences, SIM-Sync first uses a pretrained depth prediction network to lift the 2D keypoints into 3D scaled point clouds, where the scaling of the per-image point cloud is unknown due to the scale ambiguity in monocular depth prediction. SIM-Sync then seeks to synchronize jointly the unknown camera poses and scaling factors (i.e., over the 3D similarity group) by minimizing the sum of the Euclidean distances between edge-wise scaled point clouds. The SIM-Sync formulation, despite being nonconvex, allows for the design of an efficient, certifiably optimal solver that is almost identical to the SE-Sync algorithm. Particularly, after solving the translations in closed-form, the remaining optimization over the rotations and scales can be written as a quadratically constrained quadratic program , for which we apply Shor's semidefinite relaxation. We demonstrate the empirical tightness and practical usefulness of SIM-Sync in both simulated and real experiments, and investigate the impact of graph structure and sparsity.

中文翻译：

---

SIM-Sync：从 3D 相似组上可证明的最佳同步到具有学习深度的场景重建

我们提出SIM 卡同步 ， A可验证的最佳算法，可估计相机轨迹和 3D 场景结构直接来自多视图图像关键点。的关键推动者SIM 卡同步是一个预训练深度预测网络。给定一个图，其中的节点表示在未知相机姿势下拍摄的单眼图像，并且边缘包含成对图像关键点对应关系，SIM 卡同步首先使用预训练的深度预测网络将 2D 关键点提升为 3D缩放点云，由于单目深度预测中的尺度模糊，每个图像点云的缩放是未知的。SIM 卡同步然后寻求通过最小化边缘缩放点云之间的欧几里德距离之和，联合同步未知相机姿态和缩放因子（即，在 3D 相似性组上）。这SIM 卡同步公式尽管是非凸的，但允许设计一个高效的、可证明的最佳求解器，该求解器几乎与SE同步算法。特别是，在以封闭形式求解平移后，旋转和缩放的剩余优化可以写为二次约束二次规划 ，我们应用 Shor 半定松弛。我们证明了经验的紧密性和实际的有用性SIM 卡同步在模拟和真实实验中，并研究图结构和稀疏性的影响。