Unite and Conquer
Independently trained diffusion models compose at inference time to satisfy multiple conditions without joint multimodal training.
Composing independent diffusion models
Unite and Conquer combines independently trained, modality-conditioned diffusion models into a single sampling process. A generalized product of experts provides a way to combine their information without jointly training a new model on every modality combination.
Each model contributes a conditional prediction. Reliability-weighted predictions with a null condition estimate the shared unconditional density used in the composition.
How it works
- Select diffusion models trained for the individual conditions, such as an object class and a text description.
- Combine their conditional and null-condition predictions using the product-of-experts formulation and reliability weights.
- Sample one image with the combined guidance so the result can express the conditions together.
Results
The method demonstrates multimodal synthesis without joint multimodal training or retraining for each new combination of conditions.
The slide pairs ImageNet classes with text-defined scenes. The composed samples retain the requested class while expressing scenes such as a starry night or a snowy forest.
Contributions
- Pioneered compositional multimodal generation by combining independent diffusion models through a generalized product of experts.
- Introduced controllable modality weights to balance the influence of different models.
- Demonstrated joint class-and-text constraints without training a new model for every modality combination.
Unite and Conquer: Plug & Play Multi-Modal Synthesis Using Diffusion Models. CVPR 2023.