Implement basic/feature parity img2img postprocess upscaling

[drhead]

Main thing in this is implementation of the same upscaling present in the old API. Images received from the img2img API will be sent to the extra-batch-images endpoint for upscaling. While this is an extra API call that ideally should not have to happen, it is plenty fast enough. The results from upscaling are then injected as layers and are masked off properly.

EDIT: Just for reference, I have not tested how this performs in anything but my main workflow of working on obscenely high res images. Consequently I completely forgot that it might not be desirable to always upscale even when the selection is lower res than the generation. Should still work because the upscaler will happily downscale images when asked to. I have a couple of other people testing it to hammer out any edge cases.

Additionally, I fixed two bugs:

override_settings_restore_afterwards is now True. This was causing persistent and unwanted setting changes, like making the webui not return grids in the results. Everything seems to work just fine with it as True, and it won't attempt to load back the previous model or anything if that's why it was off.

The process for converting the mask has been simplified to three QImage format conversions. The mask is converted to Alpha8, dropping the RGB channels, and it is reinterpreted as grayscale so that when we convert it back to RGBA8888 it goes to the RGB channels. Simple, clean, seems to be a bit faster, and doesn't require any specific mask colors, so that limitation can be crossed off the list.

[JasonS09]

@drhead hello. Any update on this?

[drhead]

@drhead hello. Any update on this?

Got a bit sidetracked on... several other things. The multithreading bugs are mostly resolved, and are at the very least much more rare than they were now that I made the current QTimer-based implementation. I have had occasional issues with it not creating a transparency mask, usually on my machine it will be the first attempt at generating an image if anything then it doesn't happen again.

It is more functional than it used to be, so I am going to set this as ready for review. The old API can be retired as soon as support is added for upscaling through the official API, the postprocess upscale code can be reused for this purpose and it may be a good opportunity to expand it to allow use of blended upscalers. Also, the options for transparency masks and layer groups need to be actually reimplemented or removed.

[JasonS09]

@drhead hello. Any update on this?

Got a bit sidetracked on... several other things. The multithreading bugs are mostly resolved, and are at the very least much more rare than they were now that I made the current QTimer-based implementation. I have had occasional issues with it not creating a transparency mask, usually on my machine it will be the first attempt at generating an image if anything then it doesn't happen again.

It is more functional than it used to be, so I am going to set this as ready for review. The old API can be retired as soon as support is added for upscaling through the official API, the postprocess upscale code can be reused for this purpose and it may be a good opportunity to expand it to allow use of blended upscalers. Also, the options for transparency masks and layer groups need to be actually reimplemented or removed.

Actually I've found an issue, probably in the way data for the mask is being set. In most cases I see the created mask is "smaller" than the actual inpaint mask, the inpainted content is being clipped. If I hide the transparency mask this becomes apparent.

[drhead]

Actually I've found an issue, probably in the way data for the mask is being set. In most cases I see the created mask is "smaller" than the actual inpaint mask, the inpainted content is being clipped. If I hide the transparency mask this becomes apparent.

I do crop it to match the original selection, mainly to make sure nothing ends up in the image that was not intended to be inpainted, can you show an example of what is being cut off and if/how it is misaligned?

[JasonS09]

I do crop it to match the original selection, mainly to make sure nothing ends up in the image that was not intended to be inpainted, can you show an example of what is being cut off and if/how it is misaligned?

Sure, this is what I found (I have been using this version while doing art and noticed this before as well "in the wild"):

I'm using the following image:

I paint the mask:

Send to inpaint, this is the result (notice how the left edge in the flower is still showing base image):

I manually create a transparency mask from inpaint mask (expected result):

This is what I see if I show both inpaint mask and plugin transparency mask, you can still see a big chunk of the inpaint mask:

Here's the file used so you can try to reproduce:
PluginTest.zip

[drhead]

Just managed to fix it, but I can't seem to get rid of this bug with garbage data on the edge of the mask:

I might look into it later but it would help to have someone else investigate it because this implementation detail of QImage makes absolutely no sense at all.

[JasonS09]

Just managed to fix it, but I can't seem to get rid of this bug with garbage data on the edge of the mask:

I might look into it later but it would help to have someone else investigate it because this implementation detail of QImage makes absolutely no sense at all.

I'm able to reproduce sometimes. My guess is this has to do with the fact bytesPerLine() returns a different value from width(). A selection of 503px of width returns 504 bytes per line in my case. Check the comments of this question. Maybe that garbage data is the result of the overflowing bytes. Setting the w variable to the correct width value for setPixelData() will distort the image, so I guess it expects (in my case) a multiple of four. It's curious because I can't make an exact selection with a multiple of four for width in Krita, it will jump from 503px to 505px when I debug, even though the UI says it's a 504px width selection. Maybe it's a bug in the way setPixelData() is implemented for selections, that is ignoring the mismatch between the actual data content and the required bytes for an image?

[JasonS09]

@drhead Any update? Should we merge it this way?

[drhead]

@drhead Any update? Should we merge it this way?

I'm currently occupied with a finetuning project as of right now and will be for the next month. These changes can be merged as-is for now.