MaxFusion
Variance-guided feature fusion combines off-the-shelf controllers for generation with multiple spatial conditions.
Variance-guided feature fusion
MaxFusion combines intermediate features from pretrained conditional diffusion models. It uses feature variance as a signal of conditioning intensity, allowing different spatial controls to contribute where they are informative.
The models already know how to respond to their individual conditions. Feature fusion lets a generation use several controls without training a new controller for every combination.
How it works
- Run off-the-shelf conditional models, including ControlNet or T2I-Adapter, for the requested spatial conditions.
- Measure variance in their aligned intermediate features to identify the strength of each condition across spatial locations.
- Fuse the conditional features and use them to guide text-to-image generation with multiple simultaneous controls.
Results
The paper demonstrates generation under both complementary and conflicting conditions without retraining the pretrained models.
The original examples combine controls such as pose, depth, edges and semantic layouts. They illustrate how spatial constraints from different models can influence the same output.
Contributions
- Composed features from pretrained conditional diffusion models without retraining.
- Used feature variance to guide the combination of simultaneous conditions.
- Handled complementary and conflicting controls with off-the-shelf ControlNet and T2I-Adapter models.
MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models. ECCV 2024.