Question: Inpainting model for SDXL

[Markus-Pobitzer]

Dear Stability AI Team

Thanks for an other great release!

Do you plan on releasing an inpainting model for SDXL too?

[aleksmirosh]

I will much appreciate any news about inpainting too!

[mlconnor]

+1

[jiqizaisikao]

+1

[wangqyqq]

Waiting for the inpainting model for SDXL, too!

[bghira]

it will be v-prediction and zero terminal SNR, most likely. i think that will come with 1.1.

[yang-fei]

when to release the inpainting model？ there will be much use of the inpainting model

[jaidevshriram]

Thanks @bghira! Is there a timeline for this? Also, is the SDXL-base model v-prediction too or is that also in the 1.1 release?

[Markus-Pobitzer]

SDXL can already be used for inpainting, see: https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#inpainting

To me it seems like they are using the base model for inpainting.

Will there be a fine tuned version like stable-diffusion-2-inpainting?

[landmann]

Anyone notice inpainting with a mask alters the image the mask says it should not be altering? Is this expected?

[Markus-Pobitzer]

@landmann If you are referring to small changes, than it is most likely due to the encoding/decoding step of the pipeline. We bring the image in a latent space (containing less information than the original image) and after the inpainting we decode it back to an actual image but in this process we are losing some information (the encoder is lossy as mentioned by the authors). This is expected behavior.

Workaround can be found here: huggingface/diffusers#3514 (comment)

[landmann]

Haha good ol ctrl+c ctrl+v - love it.

[shreyassks]

Is it possible to finetune SDXL 1.0 base model for inpainting?

I tried to do that but I get a error from UNET that it expects the input to have 4 channels but it got 9 channels (4 - input image, 4 - masked image, 1 - mask (B/W))

Am I doing anything wrong or should I wait for inpainting model release?

[bghira]

if you're feeling adventurous you can modify the pix2pix finetuner from Diffusers project.

it already uses 8 channel input.

[nasy]

+1

[JohanHuynh0130]

+1 im waiting for inpainting model of SDXL

[JohanHuynh0130]

SDXL INPAINTING MODEL has been out!

[yjhong89]

Is there any in-paint configs for SDXL 1.0 (base)?

[gel-crabs]

A yaml file for SDXL-inpainting is desperately needed.

[kalle07]

yes pls !

[kalle07]

new inpaint model
https://huggingface.co/wangqyqq/sd_xl_base_1.0_inpainting_0.1.safetensors/blob/main/sd_xl_base_1.0_inpainting_0.1.safetensors

its also HALF
so 16GB RTX nvidia doesnt work !

[wangqyqq]

as i mentioned in webui post, this sdxl-inpaint model can work in 16-bit float and you don't need to appy --no-half-vae or --no-half.
you can download the model, merge the code in pull request AUTOMATIC1111/stable-diffusion-webui#14390 and try it!