Adobe May – Aug 2023 · Research draft

Control2Control

I worked on Control2Control during my Adobe internship. The project uses sketches, poses and layouts to edit an image while helping preserve the surrounding scene.

Research intern

Edited generated corgi wearing sunglasses, with the surrounding road scene largely preserved. — Drag to compare the original and edited image.

Original generated corgi on a road, before the accessory edit. — Drag to compare the original and edited image.

User-drawn sketch specifying glasses for the corgi.

The sketch requests glasses, and the text prompt adds “wearing sunglasses.” Control2Control estimates the edit region internally.

Edited cup moved to the right of the same table. — Movement layout

Original cup on the left of a wooden table. — Movement layout

Edited person in the requested pose, in the same café. — Target pose

Original illustrated person standing in a sunlit café. — Target pose

How it works

Source and edited branches

The source branch uses the original spatial condition, while the target branch uses the edited sketch, pose or layout. Both use pretrained diffusion and spatial-control models; a new editor is not trained for each requested change.

Estimate the changed region

The method accumulates source and target self-attention features over intermediate denoising steps. Low-rank filtering of their difference and thresholding produce an edit mask, which selects where the target is allowed to change.

Complete attention-transfer diagram. The target keeps its own queries; the mask chooses target keys and values inside the edit region and source keys and values elsewhere. — The target keeps its own queries. The mask selects target keys and values inside the edit region and source keys and values elsewhere.

Revitalized attention transfer

Revitalized attention transfer is a mask-guided self-attention operation for diffusion image editing. At selected layers, it uses the target branch's keys and values within the edit region and the source branch's keys and values elsewhere.

K̂_t = M ⊙ K_t + (1 − M) ⊙ K_s

V̂_t = M ⊙ V_t + (1 − M) ⊙ V_s

M is the edit mask: 1 inside the changed region and 0 outside it. The subscripts t and s denote the target and source branches, respectively.

The target branch retains its own queries. This lets the edited object attend to the original scene while giving the changed region freedom to follow the new spatial condition.

Results

The draft demonstrates adding an accessory with a sketch, moving objects and changing poses using pretrained models.

Large edits can shift object identity, and errors in the estimated mask can change background details. These results are from the research draft.

Presentation

Control2Control: original corgi, a sketch adding glasses, and the edited corgi wearing sunglasses.

Control2Control

Original / Edited

Edit sketch

Object movement