PART: Learning 3D Part Assembly and Retrieval with Transformers

Ruchao Bao1, Wenzheng Wu1, Chucheng Xiang1, Zhongyuan Liu2, Yuan Liu3, Jinxin Dong2, Ligang Liu1, Ziqi Wang3*
1USTC   2Tencent   3HKUST
SIGGRAPH Asia 2026
Paper (Coming Soon) Code (Coming Soon) 🤗 Data (Coming Soon)
PART teaser results.

Results of our PART framework. Given a monolithic target shape (left, visualized as a mesh and represented as a point cloud for model input), PART retrieves appropriate components from a candidate part library (middle) and reconstructs the target as a part assembly by predicting the 6-DoF pose of each selected component (right).

Abstract

3D assembly is fundamental to modern manufacturing and digital content creation. In this paper, we present PART, a unified transformer-based framework for 3D part retrieval and assembly: given a target shape and a part library, PART automatically selects the appropriate parts and predicts their 6-DoF poses to reconstruct the target. While prior work has achieved impressive progress on assembling a pre-defined set of parts, this more practical retrieval-based setting remains largely unexplored. The task faces three key challenges: (i) a combinatorially explosive search space that grows exponentially with library size; (ii) variable-length outputs, as different targets require different numbers of parts; and (iii) continuous 6-DoF pose estimation for part assembly. To address these, we formulate retrieval and assembly as a set prediction problem and design a novel transformer-based framework that retrieves parts and regresses their poses with variable-length output. Additionally, we exploit the duality between part pose estimation and target segmentation through joint training and a novel segmentation-enhanced optimization module. Finally, we curate a large-scale dataset of 80K+ shapes, and the results show that PART generalizes to scene layouts, image targets, and real-world scans. Code and data will be released.

Pipeline

PART pipeline overview.

Overview of our framework. Given a target point cloud and a set of candidate parts, we employ Point Transformer v3 layers followed by self- and cross-attention mechanisms to extract point-wise features $f^\mathcal{P}$ and part global features $f^\mathcal{C}$. These representations serve as conditions for the Transformer decoder layers, where a set of learnable instance queries interacts with the encoded features. The decoded instance features are then fed into parallel prediction heads to determine part assignments (classifying into candidate indices or empty class), estimate 6-DoF assembly poses, and generate segmentation masks. Finally, a segmentation-enhanced optimization module refines the assembly configuration to yield the precise result.

Results

BibTeX


      
BibTeX (Coming Soon)