Multi-GPU support?

[planatscher]

Hi there, I have multiple GPUs in my machine and would like to saturate them all with WebU, e.g. to run the inference in parallel for the same prompt etc.

Has anyone done that? What would be a good entry-point for parallelization? process_images in processing.py ?

[ClashSAN]

#644
#156
@NickLucche appears to be working on something for this

[NickLucche]

Hi, I could use some pointers to get started and port what I've done in my repo here :)
In particular, where is the main stable diffusion model (in code) and in which point(s) is the model fed the user input?
I could over the source myself asap but I'll need a bit more time as I've got my hands a bit full right now.

[Sionnach86]

Greetings! I was actually about to post a discussion requesting multi-gpu support for Stable Diffusion. I own a K80 and have been trying to find a means to use both 12gbs vram cores.

I would be happy to help!

[Extraltodeus]

@NickLucche Have you pursued the idea of implementing such support for this repository?

[NickLucche]

I've taken a look at the repo but there are a lot of things going on and it's not easy to add such a feature if things are nicely packed.
My idea was to make multiple parallel copies of this model by changing the way it is fed the input prompt (which I believe happens here

stable-diffusion-webui/modules/processing.py

Line 328 in 2fd7935

def process_images(p: StableDiffusionProcessing) -> Processed:

).
But am I afraid this would require digging through a lot of code without some help/docs.

[Extraltodeus]

I added the possibility to select the device and run multiple instances at the same time but this is not as good as dispatching the generation in between different GPUs.

#3377

[PylotLight]

@NickLucche Did you have any success with SD multi-GPU support?
PyTorch or Horovod supposedly have built-in support for multi-GPU.

[TikiTDO]

I poked at this enough to get it working for my use case, though it's not PR quality code. It's very ugly and hacky, but it does wonders for inference speed. If someone wants to use it as a starting point or to pull ideas you can see the changes here: TikiTDO@2619a99

I might come back to it someday to clean it up, though that may be a challenge with the way the repo is structured.

[webcoderz]

This looks alright, wonder if still works.. good job! This is the closest thing I've seen yet

[blahole]

a error:

1

spawn.py 116 spawn_main
exitcode = _main(fd, parent_sentinel)

spawn.py 126 _main
self = reduction.pickle.load(from_parent)

processing.py 21
import modules.images as images

images.py 22
from modules import sd_samplers, shared, script_callbacks, errors

sd_samplers.py 125
set_samplers()

sd_samplers.py 50 set_samplers
samplers_hidden = set(shared.opts.hide_samplers)

AttributeError:
'NoneType' object has no attribute 'hide_samplers'

[mailani19]

There's a version of SD that does do this. But when I checked it out a couple weeks ago it didn't seem to have a very user friendly interface.
Am eager to see it be a thing here.

[target-drone]

Easy Diffusion? https://stable-diffusion-ui.github.io/
That works well using all GPUs to generate images in parallel, but it is missing the more advanced knobs and levers.

[git4ai]

I tested this repo. it sends each image to a different gpu, and what I need is maybe to distribute one image to different gpus.

[schwarztim]

I don't think we will see one image distributed between multiple graphics cards anytime soon, however it would be amazing for larger models though.

[TikiTDO]

It honestly isn't that hard to do in general. Torch already supports everything you need to do it. Somewhere up above I have some code that splits batches between two GPUs. However, the codebase is kinda a mess between all the LORA / TI / Embedding / model loading code, and distributing a single image between multiple GPUs would require untangling all that, fixing it up, and then somehow getting the author's OK to merge in a humongous change.

We're talking about weeks of a senior dev's time just to make the PR, and then who knows how much time to get the maintainer to approve it. It's just not worth the effort for anyone with the skills to do so. It would be both easier and faster to write something new... Except anything new wouldn't have all the extra webui features that people use.

[Gee1111]

want to use all my GPUs for ONE picture to speed this up

[wywywywy]

Someone (don't know who) just posted a bounty-type paid-for job to get this feature implemented into Automatic1111.

Might be a good way to earn some pocket money if anyone is up to the task.

https://www.upwork.com/freelance-jobs/apply/Automatic1111-MultiGPU-Multi-Requests-simultaneously_~01d112d181f729681b/

[TomasJankauskas5]

Please make it work Ill chip in too

[gjc9620]

very cool， looking forward to this feature

[Pl8tinium]

would make it a lot easier for ppl to produce high resolution images as its way cheaper to build a rig out of multiple old cards then to buy a 3090 or similar

[TomasJankauskas5]

Big time !

[ChinaLinZhen]

Mark

[GCoffland]

Not to nag, but any update on this? Is it currently still being worked on? If not I may take a crack at it if I have time.

[virtuallyaverage]

This is the only one I found that was updated recently, though I probably missed something.
https://github.com/papuSpartan/stable-diffusion-webui-distributed

[lethyb]

@GCoffland Did you get the chance to try something so far ?

[Straafe]

Also wondering if there has been any more movement in this area?

[Lembacher]

If not for Auto1111 – maybe for SD Next? Any progress would be appreciated!

[JokerGT]

i also would love to see this soon and donate 1..2..3 bugs!!

[pyralix]

I'm interested in this feature too, hope it's still being considered/worked on

[alienware377]

pleeeaaassse dewit!

[khromatic]

interested!

[arafatx]

StableSwarmUI support this out of box. The ex employee of Stability.ai made it.

[webcoderz]

ex

Nice thanks for this. Looks promising on this front 🙏

[pvbang]

Does SD support this feature yet, everyone?

[namemechan]

I hope this will be supported one day.

[sangoi-exe]

I believe the most likely to be able to do this would be the multidiffusion dev, based on the fact that his extension splits the image into tiles for processing, similar to what the ultimate sd upscaler does, and that they could also implement this.

In multidiffusion, you can set a number of batches, and I think would be possible to throw each one onto a GPU.

[blahole]

Have you considered using GPUs only for the inference step, for example with accelerate?

[St33lMouse]

With the arrival of Flux, this becomes more important. Flux maxes out vram, swapping models in and out during inference to avoid oom, which slows things down a lot.

[ShadowPlague21]

yes very important

[crewshin]

Bump. This IS important.