[IP-Adapter] adding IP adapter support to all ControlNet and T2I pipelines

[yiyixuxu]

we recently added IP-adapter support to many of our pipelines including all the text2img, img2img and inpaint pipelines, as well as the text2img ControlNet pipeline.

anyone interested in adding it to rest of the ControlNet pipelines and T2I pipelines? You can check out this PR to see how it should be added #5713

[juancopi81]

Hi @yiyixuxu,

I'd like to give it a try and add support for the StableDiffusionControlNetInpaintPipeline

[a-r-r-o-w]

Awesome! @juancopi81 Will you be working on controlnets or other pipelines as well? If not, I'd be happy to take it up for txt2img :)

[juancopi81]

Sure! 😃 go ahead, I'll be working for now only in this pipeline: StableDiffusionControlNetInpaintPipeline

[charchit7]

Hey @yiyixuxu I'll also look into it :)

[charchit7]

Hey @juancopi81 @a-r-r-o-w is it okay if I take up for remaining one(img2img)? #self-assign

[juancopi81]

Hi @charchit7, for me it is ok, I took the StableDiffusionControlNetInpaintPipeline and I already created the PR, so in my case I am not working for the moment in any other pipeline.

[charchit7]

@a-r-r-o-w I am almost done with img2img :)

[ghost]

@yiyixuxu Is there a way to control the influence of the IPAdapter to the final output, as adapter_conditioning_scale during the call? I see there is a function called set_ip_adapter_scale, but you need to set it before the call function

[jenniferzxm]

@yiyixuxu I am interested in working on this issue and I am wondering if there is any more work I can contribute to?

[yiyixuxu]

@dverdu-freepik

do you mean you need to set different adapter_conditioning_scale at different steps? can you explain why do you need to do this?

[ghost]

@yiyixuxu I think it would be nice to have control over the influence of the IPAdapter during the call function as you do with lora_scale or adapter_conditioning_scale parameters.

If I am not wrong, you cannot have a way to do it during the call. Only, you can call to set_ip_adapter_scale to set a new different scale before the call function. I would propose to use adapter_conditioning_scale for controlling the IPAdapter attention scale during the call, as it is done with T2IAdapters. What do you think?

[yiyixuxu]

@dverdu-freepik I think it makes sense too!
do you want to open a PR for it? we can treat it similarly to how we treat lora_scale

[atesgoral]

I've started a draft PR to add support to StableDiffusionXLControlNetInpaintPipeline: #6605
But I don't really know what I'm doing. I basically followed the changes in #5887 to apply by best guesses. Any help in pushing this forward will be appreciated!

[wilfrediscoming]

How to unload ip adapter?

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")

I used to .load_ip_adapter, but what if laterwards I want to unload the ipadapter?

[yiyixuxu]

@wilfrediscoming
we have a unload_ip_adapter method

diffusers/src/diffusers/loaders/ip_adapter.py

Line 229 in d5f444d

def unload_ip_adapter(self):

[wilfrediscoming]

But I got an error after using the method..

inpaint.unload_ip_adapter()

prompt = "on the rock"
neg_prompt = "3d render, cartoon, doll, anime, artwork, fake, painting, illustration"

images = inpaint(
    prompt=prompt+ ", product photography, UHD, 8k, F2.8, RAW Photo, ultra detailed",
    negative_prompt=neg_prompt + ",<BadDream>, (<UnrealisticDream:1.2>)",
    num_inference_steps=25,
    image=image_blank,
    mask_image=image_mask,
    control_image=depth_image,
    # ip_adapter_image=image_ref,
    # guidance_scale=15,
    strength=1.0,
    controlnet_conditioning_scale=1.0,
    num_images_per_prompt=4,
    # clip_skip=1,
)


idx = 0

for each in images.images:
    each.save('/content/semi-final.jpg')

    image_product = Image.open("/content/product-512.png")
    each.paste(image_product, (x,y), image_product)

    each.save(f'/content/final{idx}.jpg')
    idx += 1

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-8-54f9dce33df7>](https://localhost:8080/#) in <cell line: 8>()
      6 neg_prompt = "3d render, cartoon, doll, anime, artwork, fake, painting, illustration"
      7 
----> 8 images = inpaint(
      9     prompt=prompt+ ", product photography, UHD, 8k, F2.8, RAW Photo, ultra detailed",
     10     negative_prompt=neg_prompt + ",<BadDream>, (<UnrealisticDream:1.2>)",

2 frames
[/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py](https://localhost:8080/#) in run_safety_checker(self, image, device, dtype)
    663             else:
    664                 feature_extractor_input = self.image_processor.numpy_to_pil(image)
--> 665             safety_checker_input = self.feature_extractor(feature_extractor_input, return_tensors="pt").to(device)
    666             image, has_nsfw_concept = self.safety_checker(
    667                 images=image, clip_input=safety_checker_input.pixel_values.to(dtype)

TypeError: 'NoneType' object is not callable

[a-r-r-o-w]

I believe this can be marked as complete since it has been sufficiently addressed. Thanks for the great work everyone!