add PAG support sd15 controlnet #8820

tuanh123789 · 2024-07-09T17:58:11Z

What does this PR do?

Adds PAG (Perturbed-Attention Guidance) support for SD 1.5 with controlnet models (StableDiffusionControlNetPAGPipeline)
Continuation of #8710

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Sample using SD1.5 controlnet	Sample SD1.5 controlnet PAG

from diffusers import AutoPipelineForText2Image, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch

import cv2
from PIL import Image

# download an image
image = load_image(
   "https://hf.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png"
)
image = np.array(image)

# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = AutoPipelineForText2Image.from_pretrained(
   "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16, enable_pag=True
)

# speed up diffusion process with faster scheduler and memory optimization
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

# generate image
generator = torch.manual_seed(0)
image = pipe(
   "aerial view, a futuristic research complex in a bright foggy jungle, hard lighting",
   guidance_scale=7.5,
   generator=generator,
   image=canny_image,
   pag_scale=10,
).images[0]

tuanh123789 · 2024-07-09T18:00:44Z

Hi @a-r-r-o-w I open new PR because I cannot undo deleted pipelines. I have edited it according to your comment, please review again, thank you

a-r-r-o-w

Looking great. Just small changes remaining and this should be good for merge.

src/diffusers/pipelines/__init__.py

src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd.py

HuggingFaceDocBuilderDev · 2024-07-10T09:20:20Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

a-r-r-o-w · 2024-07-10T19:42:19Z

Thank you for addressing the comments! Could we fix the failing tests?

Error logs

...
FAILED tests/pipelines/pag/test_pag_controlnet_sd.py::StableDiffusionControlNetPAGPipelineFastTests::test_pag_cfg - AssertionError: output is different from expected, [0.45505235 0.2785938  0.16334778 0.79689944 0.53095645 0.40135607
   0.7052706  0.69065094 0.41548574]
assert 0.3865616192955017 < 0.001
FAILED tests/pipelines/pag/test_pag_controlnet_sd.py::StableDiffusionControlNetPAGPipelineFastTests::test_pag_uncond - AssertionError: output is different from expected, [0.45127502 0.2797252  0.15970308 0.7993157  0.5414344  0.40160775
   0.7114598  0.69803864 0.4217583 ]
assert 0.39445382411422725 < 0.001
FAILED tests/pipelines/pag/test_pag_controlnet_sd.py::StableDiffusionControlNetPAGPipelineFastTests::test_save_load_optional_components - KeyError: 'tokenizer_2'

Let me know if you need any help. We can merge as soon as these are fixed since everything else looks good to me!

tuanh123789 · 2024-07-10T20:11:33Z

Sure can you help me on tests 🤗

a-r-r-o-w · 2024-07-10T20:23:08Z

Sure can you help me on tests 🤗

For the first two test failures, you will have to update the expected values to what is the actual value of image_slice is. You can print the values of image_slice and replace expected_slice with that 😜

For the third test, you need to remove SDXLOptionalComponentsTesterMixin in the fast tests class.

yiyixuxu

thanks!

yiyixuxu · 2024-07-11T20:48:04Z

src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd.py

+    # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents
+    def decode_latents(self, latents):
+        deprecation_message = "The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) instead"
+        deprecate("decode_latents", "1.0.0", deprecation_message, standard_warn=False)
+
+        latents = 1 / self.vae.config.scaling_factor * latents
+        image = self.vae.decode(latents, return_dict=False)[0]
+        image = (image / 2 + 0.5).clamp(0, 1)
+        # we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16
+        image = image.cpu().permute(0, 2, 3, 1).float().numpy()
+        return image


Suggested change

# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents

def decode_latents(self, latents):

deprecation_message = "The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) instead"

deprecate("decode_latents", "1.0.0", deprecation_message, standard_warn=False)

latents = 1 / self.vae.config.scaling_factor * latents

image = self.vae.decode(latents, return_dict=False)[0]

image = (image / 2 + 0.5).clamp(0, 1)

# we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16

image = image.cpu().permute(0, 2, 3, 1).float().numpy()

return image

I don't think it is used

yiyixuxu · 2024-07-11T20:49:24Z

src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd.py

+            extra_step_kwargs["generator"] = generator
+        return extra_step_kwargs
+
+    def check_inputs(


should we add a #Copied from here

I don't think we have a version of check_inputs with callback_steps removed, and this is the first of its kind

yiyixuxu · 2024-07-11T20:52:27Z

feel free to merge once you're happy with it! @a-r-r-o-w

tuanh123789 · 2024-07-12T02:42:26Z

@a-r-r-o-w Can you help me with last fail of test 🤗

a-r-r-o-w · 2024-07-12T08:31:54Z

@tuanh123789 The failing test is unrelated and nothing needs to be done on your end. I'm happy to merge this once you address YiYi's comment: #8820 (comment)

tuanh123789 · 2024-07-12T08:34:54Z

@tuanh123789 The failing test is unrelated and nothing needs to be done on your end. I'm happy to merge this once you address YiYi's comment: #8820 (comment)

:v Sure, will be done right away

a-r-r-o-w

Thanks, looking good! One other thing that I just realised - could you update the pipelines doc page with this SD15 Controlnet? https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/pag.md. Please see #7944 for an example

tuanh123789 · 2024-07-12T09:19:14Z

Thanks, looking good! One other thing that I just realised - could you update the pipelines doc page with this SD15 Controlnet? https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/pag.md. Please see #7944 for an example

Done

crapthings · 2024-07-14T03:10:16Z

pip install git+https://github.com/huggingface/diffusers

ImportError: cannot import name 'StableDiffusionControlNetPAGPipeline' from 'diffusers'

pip list

diffusers 0.30.0.dev0

how can i test?

crapthings · 2024-07-14T03:20:10Z

how do i know if its working

fix seed and test?

pip uninstall diffusers
pip install git+https://github.com/huggingface/diffusers

Bhavay-2001 · 2024-07-14T16:20:42Z

Hi @tuanh123789, could you please share the colab notebook you used to test the images which you have shown in the above output. Thanks

asomoza · 2024-07-14T19:46:17Z

fix seed and test?

yes, preferably with a complex generation that you can see pag helps to fix.

could you please share the colab notebook you used to test the images which you have shown in the above output

It's literally copy & paste the example from this PR, there's nothing else you have to do.

without pag	with pag

You could just paste everything in one cell block, but here's notebook with the steps to test it.

* add pag support sd15 controlnet * fix quality import * remove unecessary import * remove if state * fix tests * remove useless function * add sd1.5 controlnet pag docs --------- Co-authored-by: anhnct8 <anhnct8@fpt.com>

add pag support sd15 controlnet

059749b

fix quality import

e5de3c9

a-r-r-o-w reviewed Jul 10, 2024

View reviewed changes

src/diffusers/pipelines/__init__.py Outdated Show resolved Hide resolved

src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd.py Outdated Show resolved Hide resolved

tuanh123789 added 2 commits July 10, 2024 23:39

remove unecessary import

de755da

remove if state

e1cdf2c

tuanh123789 requested a review from a-r-r-o-w July 10, 2024 16:41

fix tests

af21256

yiyixuxu approved these changes Jul 11, 2024

View reviewed changes

remove useless function

a010b4b

a-r-r-o-w approved these changes Jul 12, 2024

View reviewed changes

add sd1.5 controlnet pag docs

f8d3a66

a-r-r-o-w merged commit d704b3b into huggingface:main Jul 12, 2024
12 of 15 checks passed

yiyixuxu added the PAG label Sep 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add PAG support sd15 controlnet #8820

add PAG support sd15 controlnet #8820

tuanh123789 commented Jul 9, 2024

tuanh123789 commented Jul 9, 2024

a-r-r-o-w left a comment

HuggingFaceDocBuilderDev commented Jul 10, 2024

a-r-r-o-w commented Jul 10, 2024

tuanh123789 commented Jul 10, 2024

a-r-r-o-w commented Jul 10, 2024

yiyixuxu left a comment

yiyixuxu Jul 11, 2024

yiyixuxu Jul 11, 2024

a-r-r-o-w Jul 11, 2024

yiyixuxu commented Jul 11, 2024

tuanh123789 commented Jul 12, 2024

a-r-r-o-w commented Jul 12, 2024 •

edited

Loading

tuanh123789 commented Jul 12, 2024

a-r-r-o-w left a comment •

edited

Loading

tuanh123789 commented Jul 12, 2024

crapthings commented Jul 14, 2024

crapthings commented Jul 14, 2024

Bhavay-2001 commented Jul 14, 2024

asomoza commented Jul 14, 2024

add PAG support sd15 controlnet #8820

add PAG support sd15 controlnet #8820

Conversation

tuanh123789 commented Jul 9, 2024

What does this PR do?

Before submitting

Who can review?

tuanh123789 commented Jul 9, 2024

a-r-r-o-w left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jul 10, 2024

a-r-r-o-w commented Jul 10, 2024

tuanh123789 commented Jul 10, 2024

a-r-r-o-w commented Jul 10, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

yiyixuxu Jul 11, 2024

Choose a reason for hiding this comment

yiyixuxu Jul 11, 2024

Choose a reason for hiding this comment

a-r-r-o-w Jul 11, 2024

Choose a reason for hiding this comment

yiyixuxu commented Jul 11, 2024

tuanh123789 commented Jul 12, 2024

a-r-r-o-w commented Jul 12, 2024 • edited Loading

tuanh123789 commented Jul 12, 2024

a-r-r-o-w left a comment • edited Loading

Choose a reason for hiding this comment

tuanh123789 commented Jul 12, 2024

crapthings commented Jul 14, 2024

crapthings commented Jul 14, 2024

Bhavay-2001 commented Jul 14, 2024

asomoza commented Jul 14, 2024

a-r-r-o-w commented Jul 12, 2024 •

edited

Loading

a-r-r-o-w left a comment •

edited

Loading