Novel view synthesis at scale
Token disentanglement lets transformer models learn from diffusion-generated views and generalize to unseen scenes.
Token disentanglement
Tok-D is a transformer block for novel view synthesis that distinguishes source-image tokens from target-view tokens through layer-wise modulation. It addresses feature alignment that can make a reconstruction model sensitive to artifacts in synthetic views.
The goal is to learn a model that reconstructs new camera views across scenes, without optimizing a separate representation for every test scene.
How it works
- Generate additional camera-conditioned views with CAT3D, a pretrained multi-view diffusion model.
- Use separate modulation for source and target tokens to improve feature alignment when training with synthetic views.
- Train the view-synthesis transformer on real and generated observations, then evaluate reconstruction on both familiar and unseen scene distributions.
Results
In the published in-distribution comparison, Tok-D reaches 30.02 dB PSNR on RealEstate10K and 21.55 dB on DL3DV. These are gains of 1.13 dB and 1.64 dB over the reported LVSM baseline.
A separate synthetic-data experiment shows small cross-dataset PSNR gains for Tok-D when generated views are added, while LVSM's PSNR decreases under the same augmentation setting.
The first comparison is Table 1 at 256 × 256 resolution with real-data training. Table 2 studies synthetic augmentation separately. The paper reports training our models on eight H100 GPUs for 100,000 iterations.
Contributions
- Introduced source/target token disentanglement to reduce the effect of synthetic-view artifacts.
- Improved PSNR over LVSM by 1.13 dB on RealEstate10K and 1.64 dB on DL3DV in the published in-distribution, real-data comparison.
- Co-first author. The reported experiments use eight H100 GPUs.
Scaling Transformer-Based Novel View Synthesis with Token Disentanglement and Synthetic Data. ICCV 2025.