| cover | coverY |
|---|---|
-201 |
Stable Diffusion is an advanced AI text-to-image synthesis algorithm that can generate very coherent images based on a text prompt.
It is commonly used for generating artistic images but can also generate images that look more like photos or sketches. Stable Diffusion is good at generating faces and realistic 3D scenes.
It is also good at mashing up concepts to create entirely novel images.
Stable Diffusion is entirely open source, and users can even train their own models based on their own dataset to get it to generate exactly the kind of images they want.
ControlNet is a neural network architecture designed to manage diffusion models by incorporating additional conditions. It duplicates the weights of neural network blocks into a "locked" copy and a "trainable" copy. The "trainable" copy learns the desired condition, while the "locked" copy** preserves the original model**. This approach ensures that training with small datasets of image pairs does not compromise the integrity of production-ready diffusion models. The "zero convolution" is a 1×1 convolution with both weight and bias initialized to zero. Before training, all zero convolutions produce zero output, preventing any distortion caused by ControlNet. No layer is trained from scratch; the process is still fine-tuning, keeping the original model secure. This method enables training on small-scale or even personal devices.