← All projects

ECCV 2024

MaxFusion

Combine controls such as pose, depth and scene layout so they can guide the same image, even when parts of the controls disagree.

First author

Depth and edges

Depth and edge controls at left guide two generated robot portraits.

Conflicting controls

Edge and pose controls at left guide two travel scenes, even when the controls disagree.

How it works

  1. Run off-the-shelf conditional models, including ControlNet or T2I-Adapter, for the requested spatial conditions.

  2. Measure variance in their aligned intermediate features to identify the strength of each condition across spatial locations.

  3. Fuse the conditional features and use them to guide text-to-image generation with multiple simultaneous controls.

Feature fusion

Aligned features are merged and unmerged between pretrained task models during denoising.

Variance-guided feature fusion

MaxFusion combines intermediate features from pretrained conditional diffusion models. It uses feature variance as a signal of conditioning intensity, allowing different spatial controls to contribute where they are informative.

The models already know how to respond to their individual conditions. Feature fusion lets a generation use several controls without training a new controller for every combination.

Results

This comparison uses 2,000 constructed COCO images and 50 sampling steps, with conflicting pose and scene controls. Lower FID indicates better image quality. Adherence to each control is evaluated separately, with mixed results.

Method Pose + segmentation FID Pose + depth FID
T2I-Adapter 55.61 51.47
ControlNet 63.46 48.18
MaxFusion 46.72 47.37

ECCV 2024, MaxFusion, Table 1

Lower FID indicates improved image quality in these comparisons.

Lower FID does not imply better adherence on every conditioning metric. Pose-plus-depth MSE-D is worse than ControlNet in the reported comparison.

Presentation

MaxFusion combines multiple spatial conditions in text-to-image generation