ResemBrick: Brick Reconstruction from Photographs with Perceptual Fidelity and Buildability

How can we generate physically stable, hand-buildable brick models from casual photographs while maintaining high perceptual fidelity? Existing methods often treat voxelization and structural assembly as independent problems, leading to models that either look good but fall apart or are stable but fail to capture the original object's appearance.

ResemBrick introduces a two-stage pipeline that treats discretization as a budget-allocation problem.

1. Budgeted Occupancy Completion: A resolution-conditioned 3D U-Net decides which surface voxels to fill to maximize visual resemblance under a fixed voxel budget. This network is trained in two phases: first by distilling an oracle that optimizes for perceptual metrics, and second by refining against rendered quality using a straight-through estimator.

2. Buildability-Aware Assembly: The system converts the resulting voxel grid into bricks using a greedy placement algorithm. This stage incorporates support-aware scoring, color-matching constraints, and a look-ahead penalty to prevent fragile structures. A deterministic, provably terminating repair stage then resolves any remaining floating components, ensuring the final assembly is fully grounded.

ResemBrick outperforms existing brick-construction systems in both perceptual fidelity and structural stability. On held-out objects, it is the only system to consistently achieve zero floating or unstable bricks while maintaining the highest MS-SSIM and lowest LPIPS scores. The results demonstrate that treating discretization and assembly as a coupled, buildability-aware process is essential for high-quality, real-world brick reconstruction.

This work provides a robust framework for digital-to-physical content creation. By framing voxelization as a constrained allocation problem rather than a static preprocessing step, ResemBrick bridges the gap between high-fidelity 3D reconstruction and the physical limitations of modular assembly, making it easier to turn casual 3D captures into tangible, buildable models.