Out of CPU Memory

[TheOnlyHolyMoly]

Hello,

writing this in reference to a question on SD.Next discord (https://discord.gg/wGtN3x4e):

When I let my 800 checkpoints run through on XYZ and let it generate 10x1 image (batch sizes greater 1 give me OOMs on my 4090 with this excercise) I frequently get "CPU out of memory" issues and then webui needs a restart. Havent tracked it well enough for an issue. @vladmandic seems to have investigated on this matter already.

[vladmandic]

i've spend counless hours on this.

is it a memory leak?

yes, its more than possible there are memory leaks, especially in secondary workflows.
for example, if you generate an image using 3 loras, but then change 1 lora and generate again? each lora has to restore model layers it modified, but when you have multiple objects modifying same thing, errors can happen.
i just used lora as an example, but same can happen when you have any extension actively modifying underlying data.

but in a simple workflows, i cannot detect any memory leaks anymore - up to the point i'm running my setup with gc disabled.

but my cpu memory keeps growing anyhow?

well, two things:

• python's built-in asyncio package has leaks
  and asyncio is used to communicate with browser (so its not blocking main thread)
  which means simply running a server and leaving it idle will have leaks. typically quite small, but still a leak
  but this means you can't run a server for a week and expect it not to run out of memory
  btw, guess how some most popular python webservers deal with this?
  they perform an internal scheduled restart - yeah, that's python
• python's default memory allocator pymalloc is just horrible
  and this is most visible when switching models
  looking at memray doesn't help, but python just refuses to free up memory even if its no longer occupied
  so eventually os will decide to terminate the process as its running into maximum allowed cpu memory size
  on linux, it shows as infamous killed... message

in theory, python supports other memory allocators, but haven't seen one that actually works
but there is a way to make it work better (still not good, but better):
by patching underlying os malloc function to use tcmalloc instead (there are others, tcmalloc is one i'm currently happy with)
but this only applies to linux

what about vram growing?

well, it depends on version of torch, it changed several times in the past
typical torch nowadays is pretty memory hungry on its own, it will use up nearly 1GB just to be alive
but above that, its pretty clean - but you need to look at not only active memory, also allocated and reserved as torch will always keep more allocated than actually active so it can perform faster allocations in the near-future. there are ways to disable this and force it to use active-only, but it does come at too high of a cost

so yeah, i don't know the actual solution. and if someone finds an actual memory leak, please let me know and i'll jump on it.

[TheOnlyHolyMoly]

Thanks for clarification Vlad, I just realized I forgot to ask if you think more RAM should compensate for the leaking then? I was just thrown off since python reports "CPU memory" issues... so we are actually talking RAM here, right?

[vladmandic]

Yeah, this is mostly about system RAM. I did add short note on VRAM at the end.

[gohan2091]

I keep hearing about GC being disabled. What is GC? I've looked in the settings and can't seem to find a GC setting. I have a 4090 so I should have GC (whatever that is) disabled?

[vladmandic]

GC is "garbage collect".
And it's not disabled, it's used when needed, but it can't do magic.