feat: update ldm_patched

[mashb1t]

currently issues with calculate_sigmas call

[parth9361]

can you please add this new ControlNet++ for sdxl released by xinsir which is controlnet for image generation and editing cause pyracanny and Cpds must need an improvement can review it here: https://huggingface.co/xinsir/controlnet-union-sdxl-1.0

[IPv6]

@mashb1t some background needed (willing to try to contribute) - is this ldm_patched from Forge? or what is the source of this patched version. Is anything expected to work just as now after this upgrade or it will break specific features

[mashb1t]

@IPv6 i actually don't know why it's called ldm_patched, but i assume ldm is a backend both comfy and Fooocus are built on and it basically IS the codebase of ConfyUI with minor adjustments (by me, specifically VAE looped through for loading / switching), but there are various overrides in Fooocus itself to optimize calculations. I've updated the whole code but could not adjust these overrides, so the procedure would be to generate => fix errors => repeat, maybe updating once more before as the Comfy version is already a bit older.

[IPv6]

@mashb1t is the purpose to upgrade to latest ldm from Comfy (as much as possible) but keeping patches on top? If yes - are this patches marked somehow? probably the best way is to go back to commits used as a base for current ldm_patched, identify the difference and apply it on top of current comfy code. Or i`m missing something here?

[mashb1t]

is the purpose to upgrade to latest ldm from Comfy (as much as possible) but keeping patches on top?

Exactly, you got it!

If yes - are this patches marked somehow?

They're in modules/patch*.py files

• https://github.com/lllyasviel/Fooocus/blob/main/modules/patch.py
• https://github.com/lllyasviel/Fooocus/blob/main/modules/patch_clip.py
• https://github.com/lllyasviel/Fooocus/blob/main/modules/patch_precision.py

probably the best way is to go back to commits used as a base for current ldm_patched, identify the difference and apply it on top of current comfy code. Or i`m missing something here?

I've already resolved any merge conflicts, so they should be pretty straight forward. The patches and some changes due to incompatibility are the most obvious thing which are currently not functional. Sadly i don't have the exact SHA of ComfyUI used for this MR saved anymore to provide it to you.

Hope this helps nevertheless!

[IPv6]

@mashb1t I see! So, to narrow down a thing:

• something broken in updated patch_clip.py and patch_precision.py
• or patch.py should use patched version differently - according to altered ldm

everything else presumingly should be ok due it`s the same ldm and update was straight forward in general.
Is that so?

BTW, what will happen without patch_all in patch.py? Will fooocus work as usual but without extra-optimizations (i should try for myself, but may be you have some hints). If yes it is possible to uncomment changed patchings one-by-one until broken one will be found (or several). Or there is something else going on

[mashb1t]

Some of the patches do set global vars/parameters, which are needed to be accessed in the generation process to allow certain features, so removing the patches will not only result in worse results but also break the code even further.

everything else presumingly should be ok due it`s the same ldm and update was straight forward in general.
Is that so?

Yes, also new features should them work such as DoRA support or other things in referenced issues.

[IPv6]

@mashb1t managed to fix exceptions and it working for me (quite solid - tested different cases except inpaint and refiners)
created a pull request - https://github.com/lllyasviel/Fooocus/pull/3454/commits

never created pull requests before, so let me know if anything wrong with it, hope github did it right.