CVPR 2023
Unite and Conquer
Generate images from combinations of text, class labels, semantic maps and sketches. Models learn each condition separately and combine them during sampling.
First author
Semantic maps, sketches and text
This person is chubby and has wavy black hair
An old person with brown hair
This person has blonde hair and black eyebrows
This person has brown hair and dark skin tone
Forest covered in snow
A starry night
How it works
Conditions with different representations
The paper combines class labels with language for generic scenes, and face/hair semantic maps, sketches and textual attributes for faces. Each conditional model translates its own input representation into a prediction in a common image-generation space.
Two ways to train
Use separately trained conditional models, including compatible off-the-shelf models, or train one model by alternating individual conditions across batches. Both routes avoid requiring every training image to have all modalities labeled together. A new condition still needs a suitable conditional model.
Learning across datasets
The face experiments use hair labels from CelebA-HQ and skin labels from FFHQ, with 27,000 training images from each. At evaluation, desired face and hair conditions are taken from the same held-out image. The generic-scene experiments compose an ImageNet class model with a text-conditioned model.
Reliability and guidance
Reliability weights combine the models’ null-condition predictions into the shared unconditional estimate. Guidance strengths separately control how strongly the requested class, text or other condition affects the sample. They are different controls, as Figures 6 and 7 show.
Compatible models
The participating models must predict compatible dimensions and channels. The main derivation also assumes a shared variance schedule. The paper could not use Stable Diffusion because its latent dimensions differed. Contradictory conditions and large differences in model capabilities can make composition difficult.
Sampling rule
p0(zt) ∝ ∏j pδⱼ(zt | φ)aⱼ, ∑j aj = 1
εcomp = ∑i wi εi(zt, xi, t) − (∑i wi − 1) ∑j aj εj(zt, φ, t), wi ≥ 1
The null condition is φ. Reliability weights aj combine the experts’ unconditional estimates. Guidance strengths wi control the individual conditions. They serve different roles. With all guidance strengths equal to one, the rule subtracts N − 1 copies of the shared unconditional estimate. For one expert, it reduces to classifier-free guidance. See Equations 5–9 in the paper.
Condition combinations
Results
The paper evaluates class-and-text scenes and semantic face generation. The scene comparison uses 500 generated images and reports the best result over reliability settings of 0, 0.5 and 1.
| Method | Conditioning | Class accuracy | NIQE | CLIP |
|---|---|---|---|---|
| GLIDE | Text only | 0.3% | 6.64 | 0.317 |
| CompGen | Text composition | 22.4% | 5.71 | 0.282 |
| Unite and Conquer | Class and text | 87.0% | 5.34 | 0.286 |
CVPR 2023, Unite and Conquer, Table 3
Class accuracy and CLIP: higher is better. NIQE: lower is better.
Semantic face generation
| Method | FID ↓ | LPIPS ↓ | SSIM ↑ | mIoU ↑ | F1 ↑ |
|---|---|---|---|---|---|
| INADE (combined) | 54.27 | 0.552 | 0.332 | 0.887 | 0.933 |
| TediGAN | 69.51 | 0.4823 | 0.417 | 0.834 | 0.905 |
| Unite and Conquer | 26.09 | 0.519 | 0.416 | 0.911 | 0.948 |
CVPR 2023, Unite and Conquer, Table 1. Selected CelebA evaluation rows.
Lower FID/LPIPS and higher SSIM/mIoU/F1 are better. The proposed method improves FID and semantic adherence in these rows; TediGAN retains lower LPIPS and slightly higher SSIM. TediGAN uses paired data across modalities.
The scene sampler receives class and text conditions, while its displayed baselines use text conditioning. GLIDE has the highest CLIP score. In the face comparison, FID and semantic adherence improve, while LPIPS and SSIM do not improve over every baseline.
Presentation
