Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] [IP-Adapter] Add IP Adapter support for SDXL ControlNet Inpaint pipeline #6605

Closed

Conversation

atesgoral
Copy link

@atesgoral atesgoral commented Jan 16, 2024

What does this PR do?

Answering the call to action on #5884

Follows #5887 to add support for IP Adapters for the StableDiffusionXLControlNetInpaintPipeline

cc @yiyixuxu @patrickvonplaten @sayakpaul

Note: I don't fully know what I'm doing, but I created this draft PR to get the ball rolling.

  • Tested with this branch (public Hugging Face space coming soon)

By adding to requirements.txt:

diffusers[torch] @ git+https://github.com/atesgoral/diffusers@sdxl-cnet-inpaint-ip-adapter

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@atesgoral atesgoral changed the title Add IP Adapter support for SDXL ControlNet Inpaint pipeline [IP-Adapter] Add IP Adapter support for SDXL ControlNet Inpaint pipeline Jan 16, 2024
@atesgoral atesgoral force-pushed the sdxl-cnet-inpaint-ip-adapter branch from 45ac6c4 to 9283a08 Compare January 16, 2024 23:26
@atesgoral atesgoral changed the title [IP-Adapter] Add IP Adapter support for SDXL ControlNet Inpaint pipeline [WIP] [IP-Adapter] Add IP Adapter support for SDXL ControlNet Inpaint pipeline Jan 16, 2024
@atesgoral atesgoral force-pushed the sdxl-cnet-inpaint-ip-adapter branch from 9283a08 to d8bebb0 Compare January 16, 2024 23:42
@atesgoral atesgoral force-pushed the sdxl-cnet-inpaint-ip-adapter branch 2 times, most recently from f81cbd5 to 5a699ef Compare January 17, 2024 01:39
@atesgoral atesgoral force-pushed the sdxl-cnet-inpaint-ip-adapter branch from 5a699ef to 04d58f8 Compare January 17, 2024 01:40
@sayakpaul sayakpaul requested a review from yiyixuxu January 17, 2024 02:13
@atesgoral
Copy link
Author

Watching the work on multi IP Adapters (and the potential future with IP Adapter masks) closely: #6573

So, I'll abandon this attempt for now since IP Adapter masking could get me where I need.

@atesgoral atesgoral closed this Jan 19, 2024
@anilsathyan7
Copy link

Is this supported now? Other pipelines seems to support this -https://huggingface.co/docs/diffusers/en/api/pipelines/controlnet_sdxl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants