MERL May – Aug 2022 · ICCV 2023
Steered Diffusion
Guide a pretrained image generator to restore images or follow new conditions, without training a separate generator for each task.
First author
Text-guided image editing
Inpainting with masked inputs
Super-resolution
How it works
-
Choose a pretrained diffusion generator and an auxiliary model that represents the desired task or condition.
-
Construct a task loss that measures agreement with the input condition during sampling.
-
Use the guidance signal to steer denoising, adapting the generator to the task without training a separate task-specific generator.
How steering works
Energy-based diffusion steering
Steered Diffusion turns a pretrained diffusion generator into a conditional generator at inference time. A pretrained inverse model defines a loss for the requested condition, and that loss steers the denoising trajectory toward an image that satisfies it.
For example, an inverse model can estimate a semantic label or a degraded observation from the current image. Comparing that prediction with the desired condition supplies a guidance signal.
Results
Each row compares one task with its baseline. The 8× super-resolution test uses 300 CelebA images. Lower FID indicates better distributional image quality, while higher PSNR indicates closer agreement with the reference.
| Task | Metric | Baseline | Baseline score | Steered Diffusion |
|---|---|---|---|---|
| Semantic faces | FID (lower) | ILVR | 107.46 | 41.38 |
| 8× super-resolution | PSNR dB (higher) | ILVR | 20.54 | 26.02 |
| Colorization | FID (lower) | ILVR | 67.66 | 49.72 |
| Thin-mask inpainting | FID (lower) | RePaint | 43.35 | 30.85 |
ICCV 2023, Steered Diffusion, Tables 2–5
The pretrained generator is guided at inference time.
Each task has its own metric and protocol. The displayed super-resolution improvement is specific to the paper’s CelebA evaluation.
Presentation
