Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

This paper investigates whether a strong, open-source single-image-to-3D generator (TRELLIS) can be specialized for a specific asset class (furniture) using only public data and lightweight parameter-efficient adaptation. The researchers move beyond simple geometric proxies by using a de-biased, cross-model VLM-as-3D-judge to provide the training signal. The study is framed as a protocol-first contribution, where the primary challenge is hardening the judge to function reliably within an optimization loop.

To prevent the generator from learning the judge's quirks, the authors implement a strict protocol: they use a training judge (Qwen2.5-VL-7B) distinct from an evaluation judge (InternVL3-8B). They address three critical failure modes that emerge during optimization: image overload (fixed by using two-image comparisons), Gaussian-splat renders that hide defects (fixed by using mesh normal-map montages), and reference-free bias (mitigated by clear-gap calibration). This protocol ensures that the judge only provides a signal when a genuine quality gap exists.

The researchers find that independent samples from the base generator carry no learnable preference, meaning the signal must be engineered through quality-contrastive construction. Across six adaptation methods, the study shows that lightweight fine-tuning (LoRA) of the flow-transformer fails to improve performance because updates are washed out by the rectified-flow sampler. The only effective intervention is a conditioner-repair adapter, which targets the DINOv2 image features. While this approach shows a monotonic improvement as input degradation increases, it only reaches parity with the original base model, suggesting that exceeding a strong public-data base requires more than lightweight adaptation on public data.