NVIDIA 2024 · Research draft
Physical-property assignment to 3D meshes
Use visual foundation models and multimodal reasoning to divide 3D objects into meaningful parts and estimate what those parts are made of.
Research intern
Parts and materials
Part comparisons
How it works
From rendered views to parts
Foundation-model features and region masks from multiple rendered views are refined into a consistent 3D feature field. Clustering and multimodal reasoning produce part groups and estimated material labels.
Research and invention
This work is documented in the SmartTagger research draft. I am a co-inventor on U.S. Patent Application 19/024,394, published as US20260038283A1 on February 5, 2026. The mass-estimation benchmark is reported in the draft, separately from the patent application.
Cross-view features
Results
These research-draft results compare mass estimates on ABO-500. The framework improves some error measures and worsens another.
| Metric | NeRF2- Physics | Ours |
|---|---|---|
| ADE | 10.846 | 9.752 |
| ALDE | 1.489 | 1.285 |
| APE | 0.790 | 0.834 |
| MnRE | 0.300 | 0.365 |
NVIDIA research draft, SmartTagger, Table 1
Lower ADE, ALDE and higher MnRE. APE does not improve.
Research-draft results improve ADE, ALDE and MnRE, but APE worsens. Material properties are inferred estimates. The patent is a published application.
Presentation
