Control2Control
Training-free diffusion editing with sketches, poses or layouts.
Revitalized attention transfer
Revitalized attention transfer is a mask-guided self-attention operation for diffusion image editing. At selected layers, it uses the target branch's keys and values within the edit region and the source branch's keys and values elsewhere.
K̂t = M ⊙ Kt + (1 − M) ⊙ Ks
V̂t = M ⊙ Vt + (1 − M) ⊙ Vs
M is the edit mask: 1 inside the changed region and 0 outside it. The subscripts t and s denote the target and source branches, respectively.
The target branch retains its own queries. This lets the edited object attend to the original scene while giving the changed region freedom to follow the new spatial condition.
How it works
- Run source and target diffusion branches with the original and user-edited spatial conditions.
- Accumulate self-attention features over intermediate denoising steps. Filter their source/target difference with a low-rank decomposition and threshold it to estimate the edit mask.
- Blend target keys and values inside the mask with source keys and values outside it at selected layers, then complete denoising.
Results
The working draft illustrates interactive edits with sketches, pose skeletons and object layouts using pretrained spatial controllers.
Large changes in pose or scene semantics can also require an updated text prompt.
Contributions
- Built a training-free diffusion pipeline for interactive edits using sketches, poses or layouts.
- Estimated edit masks from source/target self-attention differences using low-rank filtering.
- Used masked key/value transfer to support changes while retaining source content in unchanged regions.
Control2Control: Revitalizing Attention Transfer for Consistent Visual Editing. Adobe research draft 2023.