Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: img2img inpaint pastes the original unmasked image over the generated image #14725

Open
4 of 6 tasks
Beyondo opened this issue Jan 22, 2024 · 2 comments
Open
4 of 6 tasks
Labels
bug-report Report of a bug, yet to be confirmed

Comments

@Beyondo
Copy link

Beyondo commented Jan 22, 2024

Checklist

  • The issue exists after disabling all extensions
  • The issue exists on a clean installation of webui
  • The issue is caused by an extension, but I believe it is caused by a bug in the webui
  • The issue exists in the current version of the webui
  • The issue has not been reported before recently
  • The issue has been reported before but has not been fixed yet

What happened?

For some reason, the inpainting in WebUI doesn't work "natively" like it is supposed to.

The inpainting model is supposed to take the initial image as a very hard reference,
But in WebUI, the initial image is literally copy-pasted into the output.

This is truly dumb. Because sometimes the model generates different textures, slightly different skin tones, and the output just looks so photoshopped in the end with the edges of the mask being very visible.

Is there an option to turn this off and make WebUI show the real inpaint output? I currently could only do it manually through code which is not very convenient.

Steps to reproduce the problem

  1. Outpaint.
  2. Zoom and notice all edges OR notice how perfect the original unmasked part was unaltered; this means it was copy-pasted but diffusion models are supposed to alter everything ever so slightly.

This hurts a lot of tasks like outpainting where you need the images to blend well.

What should have happened?

Real Inpaint output without the original being involved.

What browsers do you use to access the UI ?

No response

Sysinfo

.

Console logs

.

Additional information

Let WebUI give us the real unaltered raw inpaint output from the model (even if it is ugly; cause at least it blends for outpaint tasks and can be easily post-processed), and make the current behavior an option because it doesn't fit all of people's needs.

@Beyondo Beyondo added the bug-report Report of a bug, yet to be confirmed label Jan 22, 2024
@Beyondo Beyondo changed the title [Bug]: img2img pastes the original unmasked image over the generated image [Bug]: img2img inpaint pastes the original unmasked image over the generated image Jan 22, 2024
@Beyondo
Copy link
Author

Beyondo commented Jan 22, 2024

I was able to fix it by modifying the source code that relates to overlay.

Here are the results with me trying to outpaint hair for a character:

Initial:
image
Mask:
image

(Output) Current WebUI (Overlays the raw output without telling you):
image
Look closely at the edges of the mask (this becomes much more noticeable when generating limbs instead of hair)
.
.
(Output) After removing the overlay code:
image
RAW image out of the model. Seemless blend. Much smoother. That's how the AI generated it, so why mess it up with some weird cropped overylay, whoever decided that was a great idea?

Notice how the AI modified the unmasked part of the initial, like the eyes and face, but it can easily be fixed with 2 seconds of a photoshop eraser:

Final:
image
Erase the parts of the inpaint you don't want with a smooth photoshop brush with the original behind it.

Or you know, maybe you liked the face/eyes the AI generated. That's why it is important to not force users to have their AI-generated images overlayed in WebUI; because someone could always like some features that the AI generated, and erase some other features instead of all.

So yeah, the automatic overlay cropped from the initial image is overrated and should 100% be removed or be an explicit option that's unchecked by default.

@Beyondo
Copy link
Author

Beyondo commented Jan 22, 2024

Made a pull request: #14727

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug-report Report of a bug, yet to be confirmed
Projects
None yet
Development

No branches or pull requests

1 participant