We address the 3D shape assembly of multiple geometric pieces without overlaps, a scenario often encountered in 3D shape design, field archeology, and robotics. Existing methods depend on strong assumptions on the number of shape pieces and coherent geometry or semantics of shape pieces. Despite raising attention to 3D registration with complex or low overlapping patterns, few methods consider shape assembly with rare overlaps. To address this problem, we present a novel framework inspired by solving puzzles, named PuzzleNet, which conducts multi-task learning by leveraging both 3D alignment and boundary information. Specifically, we design an end-to-end neural network based on a point cloud transformer with two-way branches for estimating rigid transformation and predicting boundaries simultaneously. The framework is then naturally extended to reassemble multiple pieces into a full shape by using an iterative greedy approach based on the distance between each pair of candidate-matched pieces. To train and evaluate PuzzleNet, we construct two datasets, named DublinPuzzle and ModelPuzzle, based on a real-world urban scan dataset (DublinCity) and a synthetic CAD dataset (ModelNet40) respectively. Experiments demonstrate our effectiveness in solving 3D shape assembly for multiple pieces with arbitrary geometry and inconsistent semantics. Our method surpasses state-of-the-art algorithms by more than 10 times in rotation metrics and four times in translation metrics.

我们解决了多个几何部件不重叠的 3D 形状组装问题，这是 3D 形状设计、野外考古和机器人技术中经常遇到的场景。现有方法依赖于对形状块的数量以及形状块的连贯几何或语义的强有力的假设。尽管人们越来越关注具有复杂或低重叠图案的 3D 配准，但很少有方法考虑具有罕见重叠的形状组装。为了解决这个问题，我们提出了一个受解决难题启发的新颖框架，名为 PuzzleNet，它通过利用 3D 对齐和边界信息来进行多任务学习。具体来说，我们设计了一个基于点云变换器的端到端神经网络，具有双向分支，用于同时估计刚性变换和预测边界。然后，该框架自然地扩展为通过使用基于每对候选匹配块之间的距离的迭代贪婪方法将多个块重新组装成完整的形状。为了训练和评估 PuzzleNet，我们分别基于真实城市扫描数据集 (DublinCity) 和合成 CAD 数据集 (ModelNet40) 构建了两个数据集，名为 DublinPuzzle 和 ModelPuzzle。实验证明了我们在解决具有任意几何形状和不一致语义的多个部件的 3D 形状组装方面的有效性。我们的方法在旋转指标上超越最先进的算法 10 倍以上，在平移指标上超越最先进的算法 4 倍以上。我们分别基于真实城市扫描数据集 (DublinCity) 和合成 CAD 数据集 (ModelNet40) 构建了两个数据集，分别命名为 DublinPuzzle 和 ModelPuzzle。实验证明了我们在解决具有任意几何形状和不一致语义的多个部件的 3D 形状组装方面的有效性。我们的方法在旋转指标上超越最先进的算法 10 倍以上，在平移指标上超越最先进的算法 4 倍以上。我们分别基于真实城市扫描数据集 (DublinCity) 和合成 CAD 数据集 (ModelNet40) 构建了两个数据集，分别命名为 DublinPuzzle 和 ModelPuzzle。实验证明了我们在解决具有任意几何形状和不一致语义的多个部件的 3D 形状组装方面的有效性。我们的方法在旋转指标上超越最先进的算法 10 倍以上，在平移指标上超越最先进的算法 4 倍以上。