StructRe: Rewriting for Structured Shape Modeling
Abstract
Man-made 3D shapes are naturally organized in parts and hierarchies; such structures provide important constraints for shape reconstruction and generation. Modeling shape structures is difficult, because there can be multiple hierarchies for a given shape, causing ambiguity, and across different categories the shape structures are correlated with semantics, limiting generalization. We present StructRe, a structure rewriting system, as a novel approach to structured shape modeling. Given a 3D object represented by points and components, StructRe can rewrite it upward into more concise structures, or downward into more detailed structures; by iterating the rewriting process, hierarchies are obtained. Such a localized rewriting process enables probabilistic modeling of ambiguous structures and robust generalization across object categories. We train StructRe on PartNet data and show its generalization to cross-category and multiple object hierarchies, and test its extension to ShapeNet. We also demonstrate the benefits of probabilistic and generalizable structure modeling for shape reconstruction, generation and editing tasks.
Introduction
In this paper, we propose a robust and generalizable approach for structured shape modeling, by using local and probabilistic structure rewriting. The rewriting system transits locally between two levels of part decompositions within a hierarchy, therefore getting rid of categorical templates and generalizing across categories. The rewriting system also allows probabilistic sampling of different transitions that are equally plausible, therefore resolving the ambiguity with diverse structures. Specifically, we first build a unified latent space of all parts and shapes across categories. Given the latent encoding, we learn the rewriting model that maps a set of input parts into another set of output parts, with iterative decoding at the output side to sample multiple solutions according to probability.
Structured reconstruction and completion
In this part, we show our model's performance on structured reconstruction on complete point cloud input (a) and structured completion on partial point cloud via virtual scanning (b). Compared with the baseline methods, our method can generate more regularized and high quality results via local rewriting.
Structured generation
For each input point cloud, we encode it into a root node and apply the downward rewriting process interleaved with shape code perturbation to generate three different shapes in rows.
Result analysis
In this part we show the generalization ability of the trained network through its performance on small categories (a) of PartNet and on multiple objects (b). We also show the effect of probabilistic sampling for modeling multiple structures (c).
ShapeNet extension
In this part, we show the model's entension to ShapeNet data, where it infers meaningful structures (a) and leaf nodes (b) in zero-shot and also show its capability to new data via few-shot tunning.
Applications
Modeling applications enabled by rewriting. (a) reconstruction plus variation. Given partial scans, our rewriting model parses them into structured hierarchies, and explores variations by downward rewriting plus shape perturbation. (b) part editing. By selecting parts at proper levels for objects of different categories, swapping them, and rewriting upward to regularize the editing, novel objects are obtained. (c) shape vectorization. After a shape is parsed into parts, nearest parts with compact polygonal meshes can be retrieved from a database to vectorize the shape.
Citation
@article{wang2023structre,
title={StructRe: Rewriting for Structured Shape Modeling},
author={Wang, Jiepeng and Pan, Hao and Liu, Yang and Tong, Xin and Komura, Taku and Wang, Wenping},
journal={arXiv preprint arXiv:2311.17510},
year={2023}
}
This page is Zotero translator friendly. Page last updated