Steered Diffusion
A general framework steers pretrained diffusion models during sampling for image-to-image and label-conditioned generation.
Energy-based diffusion steering
Steered Diffusion turns a pretrained diffusion generator into a conditional generator at inference time. A pretrained inverse model defines a loss for the requested condition, and that loss steers the denoising trajectory toward an image that satisfies it.
For example, an inverse model can estimate a semantic label or a degraded observation from the current image. Comparing that prediction with the desired condition supplies a guidance signal.
How it works
- Choose a pretrained diffusion generator and an auxiliary model that represents the desired task or condition.
- Construct a task loss that measures agreement with the input condition during sampling.
- Use the guidance signal to steer denoising, adapting the generator to the task without training a separate task-specific generator.
Results
The framework supports image-to-image and label-conditioned synthesis within the same plug-and-play formulation.
The paper demonstrates tasks including inpainting, colorization, super-resolution and semantic generation. The slide shows inpainting and semantic-generation samples.
Contributions
- Introduced the first plug-and-play framework applicable to both image-to-image translation and label-conditioned synthesis in the studied setting.
- Used pretrained auxiliary networks and energy-based guidance to express new conditions at inference time.
- Supported tasks including inpainting, colorization, super-resolution and semantic generation without task-specific generator training.
Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis. ICCV 2023.