SDXL is now supported. The sdxl branch has been merged into the main branch. If you update the repository, please follow the upgrade instructions. Also, the version of accelerate has been updated, so please run accelerate config again. The documentation for SDXL training is here.

This repository contains training, generation and utility scripts for Stable Diffusion.

Change History is moved to the bottom of the page. 更新履歴はページ末尾に移しました。

日本語版READMEはこちら

For easier use (GUI and PowerShell scripts etc...), please visit the repository maintained by bmaltais. Thanks to @bmaltais!

This repository contains the scripts for:

• DreamBooth training, including U-Net and Text Encoder
• Fine-tuning (native training), including U-Net and Text Encoder
• LoRA training
• Textual Inversion training
• Image generation
• Model conversion (supports 1.x and 2.x, Stable Diffision ckpt/safetensors and Diffusers)

About requirements.txt

These files do not contain requirements for PyTorch. Because the versions of them depend on your environment. Please install PyTorch at first (see installation guide below.)

The scripts are tested with Pytorch 2.0.1. 1.12.1 is not tested but should work.

Links to usage documentation

Most of the documents are written in Japanese.

English translation by darkstorm2150 is here. Thanks to darkstorm2150!

• Training guide - common : data preparation, options etc...
  • Chinese version
• Dataset config
• DreamBooth training guide
• Step by Step fine-tuning guide:
• training LoRA
• training Textual Inversion
• Image generation
• note.com Model conversion

Windows Required Dependencies

Python 3.10.6 and Git:

• Python 3.10.6: https://www.python.org/ftp/python/3.10.6/python-3.10.6-amd64.exe
• git: https://git-scm.com/download/win

Give unrestricted script access to powershell so venv can work:

• Open an administrator powershell window
• Type Set-ExecutionPolicy Unrestricted and answer A
• Close admin powershell window

Windows Installation

Open a regular Powershell terminal and type the following inside:

git clone https://github.com/kohya-ss/sd-scripts.git
cd sd-scripts

python -m venv venv
.\venv\Scripts\activate

pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install --upgrade -r requirements.txt
pip install xformers==0.0.20

accelerate config

Note: Now bitsandbytes is optional. Please install any version of bitsandbytes as needed. Installation instructions are in the following section.

Answers to accelerate config:

- This machine
- No distributed training
- NO
- NO
- NO
- all
- fp16

note: Some user reports ValueError: fp16 mixed precision requires a GPU is occurred in training. In this case, answer 0 for the 6th question: What GPU(s) (by id) should be used for training on this machine as a comma-separated list? [all]:

(Single GPU with id 0 will be used.)

Optional: Use bitsandbytes (8bit optimizer)

For 8bit optimizer, you need to install bitsandbytes. For Linux, please install bitsandbytes as usual (0.41.1 or later is recommended.)

For Windows, there are several versions of bitsandbytes:

• bitsandbytes 0.35.0: Stable version. AdamW8bit is available. full_bf16 is not available.
• bitsandbytes 0.41.1: Lion8bit, PagedAdamW8bit and PagedLion8bit are available. full_bf16 is available.

Note: bitsandbytesabove 0.35.0 till 0.41.0 seems to have an issue: bitsandbytes-foundation/bitsandbytes#659

Follow the instructions below to install bitsandbytes for Windows.

bitsandbytes 0.35.0 for Windows

Open a regular Powershell terminal and type the following inside:

cd sd-scripts
.\venv\Scripts\activate
pip install bitsandbytes==0.35.0

cp .\bitsandbytes_windows\*.dll .\venv\Lib\site-packages\bitsandbytes\
cp .\bitsandbytes_windows\cextension.py .\venv\Lib\site-packages\bitsandbytes\cextension.py
cp .\bitsandbytes_windows\main.py .\venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py

This will install bitsandbytes 0.35.0 and copy the necessary files to the bitsandbytes directory.

bitsandbytes 0.41.1 for Windows

Install the Windows version whl file from here or other sources, like:

python -m pip install bitsandbytes==0.41.1 --prefer-binary --extra-index-url=https://jllllll.github.io/bitsandbytes-windows-webui

Upgrade

When a new release comes out you can upgrade your repo with the following command:

cd sd-scripts
git pull
.\venv\Scripts\activate
pip install --use-pep517 --upgrade -r requirements.txt

Once the commands have completed successfully you should be ready to use the new version.

Credits

The implementation for LoRA is based on cloneofsimo's repo. Thank you for great work!

The LoRA expansion to Conv2d 3x3 was initially released by cloneofsimo and its effectiveness was demonstrated at LoCon by KohakuBlueleaf. Thank you so much KohakuBlueleaf!

License

The majority of scripts is licensed under ASL 2.0 (including codes from Diffusers, cloneofsimo's and LoCon), however portions of the project are available under separate license terms:

Memory Efficient Attention Pytorch: MIT

bitsandbytes: MIT

BLIP: BSD-3-Clause

SDXL training

The documentation in this section will be moved to a separate document later.

Training scripts for SDXL

• sdxl_train.py is a script for SDXL fine-tuning. The usage is almost the same as fine_tune.py, but it also supports DreamBooth dataset.

  • --full_bf16 option is added. Thanks to KohakuBlueleaf!
    • This option enables the full bfloat16 training (includes gradients). This option is useful to reduce the GPU memory usage.
    • The full bfloat16 training might be unstable. Please use it at your own risk.
  • The different learning rates for each U-Net block are now supported in sdxl_train.py. Specify with --block_lr option. Specify 23 values separated by commas like --block_lr 1e-3,1e-3 ... 1e-3.
    • 23 values correspond to 0: time/label embed, 1-9: input blocks 0-8, 10-12: mid blocks 0-2, 13-21: output blocks 0-8, 22: out.

• prepare_buckets_latents.py now supports SDXL fine-tuning.

• sdxl_train_network.py is a script for LoRA training for SDXL. The usage is almost the same as train_network.py.

• Both scripts has following additional options:

  • --cache_text_encoder_outputs and --cache_text_encoder_outputs_to_disk: Cache the outputs of the text encoders. This option is useful to reduce the GPU memory usage. This option cannot be used with options for shuffling or dropping the captions.
  • --no_half_vae: Disable the half-precision (mixed-precision) VAE. VAE for SDXL seems to produce NaNs in some cases. This option is useful to avoid the NaNs.

• --weighted_captions option is not supported yet for both scripts.

• sdxl_train_textual_inversion.py is a script for Textual Inversion training for SDXL. The usage is almost the same as train_textual_inversion.py.

  • --cache_text_encoder_outputs is not supported.
  • There are two options for captions:
    1. Training with captions. All captions must include the token string. The token string is replaced with multiple tokens.
    2. Use --use_object_template or --use_style_template option. The captions are generated from the template. The existing captions are ignored.
  • See below for the format of the embeddings.

• --min_timestep and --max_timestep options are added to each training script. These options can be used to train U-Net with different timesteps. The default values are 0 and 1000.

Utility scripts for SDXL

• tools/cache_latents.py is added. This script can be used to cache the latents to disk in advance.

  • The options are almost the same as `sdxl_train.py'. See the help message for the usage.
  • Please launch the script as follows: accelerate launch --num_cpu_threads_per_process 1 tools/cache_latents.py ...
  • This script should work with multi-GPU, but it is not tested in my environment.

• tools/cache_text_encoder_outputs.py is added. This script can be used to cache the text encoder outputs to disk in advance.

  • The options are almost the same as cache_latents.py and sdxl_train.py. See the help message for the usage.

• sdxl_gen_img.py is added. This script can be used to generate images with SDXL, including LoRA, Textual Inversion and ControlNet-LLLite. See the help message for the usage.

Tips for SDXL training

• The default resolution of SDXL is 1024x1024.
• The fine-tuning can be done with 24GB GPU memory with the batch size of 1. For 24GB GPU, the following options are recommended for the fine-tuning with 24GB GPU memory:
  • Train U-Net only.
  • Use gradient checkpointing.
  • Use --cache_text_encoder_outputs option and caching latents.
  • Use Adafactor optimizer. RMSprop 8bit or Adagrad 8bit may work. AdamW 8bit doesn't seem to work.
• The LoRA training can be done with 8GB GPU memory (10GB recommended). For reducing the GPU memory usage, the following options are recommended:
  • Train U-Net only.
  • Use gradient checkpointing.
  • Use --cache_text_encoder_outputs option and caching latents.
  • Use one of 8bit optimizers or Adafactor optimizer.
  • Use lower dim (4 to 8 for 8GB GPU).
• --network_train_unet_only option is highly recommended for SDXL LoRA. Because SDXL has two text encoders, the result of the training will be unexpected.
• PyTorch 2 seems to use slightly less GPU memory than PyTorch 1.
• --bucket_reso_steps can be set to 32 instead of the default value 64. Smaller values than 32 will not work for SDXL training.

Example of the optimizer settings for Adafactor with the fixed learning rate:

optimizer_type = "adafactor"
optimizer_args = [ "scale_parameter=False", "relative_step=False", "warmup_init=False" ]
lr_scheduler = "constant_with_warmup"
lr_warmup_steps = 100
learning_rate = 4e-7 # SDXL original learning rate

Format of Textual Inversion embeddings for SDXL

from safetensors.torch import save_file

state_dict = {"clip_g": embs_for_text_encoder_1280, "clip_l": embs_for_text_encoder_768}
save_file(state_dict, file)

ControlNet-LLLite

ControlNet-LLLite, a novel method for ControlNet with SDXL, is added. See documentation for details.

Change History

Oct 11, 2023 / 2023/10/11

• Fix to work make_captions_by_git.py with the latest version of transformers.

• Improve gen_img_diffusers.py and sdxl_gen_img.py. Both scripts now support the following options:

  • --network_merge_n_models option can be used to merge some of the models. The remaining models aren't merged, so the multiplier can be changed, and the regional LoRA also works.
  • --network_regional_mask_max_color_codes is added. Now you can use up to 7 regions.
    • When this option is specified, the mask of the regional LoRA is the color code based instead of the channel based. The value is the maximum number of the color codes (up to 7).
    • You can specify the mask for each LoRA by colors: 0x0000ff, 0x00ff00, 0x00ffff, 0xff0000, 0xff00ff, 0xffff00, 0xffffff.

• make_captions_by_git.py が最新の transformers で動作するように修正しました。

• gen_img_diffusers.py と sdxl_gen_img.py を更新し、以下のオプションを追加しました。

  • --network_merge_n_models オプションで一部のモデルのみマージできます。残りのモデルはマージされないため、重みを変更したり、領域別LoRAを使用したりできます。
  • --network_regional_mask_max_color_codes を追加しました。最大7つの領域を使用できます。
    • このオプションを指定すると、領域別LoRAのマスクはチャンネルベースではなくカラーコードベースになります。値はカラーコードの最大数（最大7）です。
    • 各LoRAに対してマスクをカラーで指定できます：0x0000ff、0x00ff00、0x00ffff、0xff0000、0xff00ff、0xffff00、0xffffff。

Oct 9. 2023 / 2023/10/9

• tag_images_by_wd_14_tagger.py now supports Onnx. If you use Onnx, TensorFlow is not required anymore. #864 Thanks to Isotr0py!

  • --onnx option is added. If you use Onnx, specify --onnx option.
  • Please install Onnx and other required packages.
    1. Uninstall TensorFlow.
    2. pip install tensorboard==2.14.1 This is required for the specified version of protobuf.
    3. pip install protobuf==3.20.3 This is required for Onnx.
    4. pip install onnx==1.14.1
    5. pip install onnxruntime-gpu==1.16.0 or pip install onnxruntime==1.16.0

• --append_tags option is added to tag_images_by_wd_14_tagger.py. This option appends the tags to the existing tags, instead of replacing them. #858 Thanks to a-l-e-x-d-s-9!

• OFT is now supported.

  • You can use networks.oft for the network module in sdxl_train_network.py. The usage is the same as networks.lora. Some options are not supported.
  • sdxl_gen_img.py also supports OFT as --network_module.
  • OFT only supports SDXL currently. Because current OFT tweaks Q/K/V and O in the transformer, and SD1/2 have extremely fewer transformers than SDXL.
  • The implementation is heavily based on laksjdjf's OFT implementation. Thanks to laksjdjf!

• Other bug fixes and improvements.

• tag_images_by_wd_14_tagger.py が Onnx をサポートしました。Onnx を使用する場合は TensorFlow は不要です。#864 Isotr0py氏に感謝します。

  • Onnxを使用する場合は、--onnx オプションを指定してください。
  • Onnx とその他の必要なパッケージをインストールしてください。
    1. TensorFlow をアンインストールしてください。
    2. pip install tensorboard==2.14.1 protobufの指定バージョンにこれが必要。
    3. pip install protobuf==3.20.3 Onnxのために必要。
    4. pip install onnx==1.14.1
    5. pip install onnxruntime-gpu==1.16.0 または pip install onnxruntime==1.16.0

• tag_images_by_wd_14_tagger.py に --append_tags オプションが追加されました。このオプションを指定すると、既存のタグに上書きするのではなく、新しいタグのみが既存のタグに追加されます。 #858 a-l-e-x-d-s-9氏に感謝します。

• OFT をサポートしました。

  • sdxl_train_network.py の--network_moduleに networks.oft を指定してください。使用方法は networks.lora と同様ですが一部のオプションは未サポートです。
  • sdxl_gen_img.py でも同様に OFT を指定できます。
  • OFT は現在 SDXL のみサポートしています。OFT は現在 transformer の Q/K/V と O を変更しますが、SD1/2 は transformer の数が SDXL よりも極端に少ないためです。
  • 実装は laksjdjf 氏の OFT実装 を多くの部分で参考にしています。laksjdjf 氏に感謝します。

• その他のバグ修正と改善。

Oct 1. 2023 / 2023/10/1

• SDXL training is now available in the main branch. The sdxl branch is merged into the main branch.

• SAI Model Spec metadata is now supported partially. hash_sha256 is not supported yet.

  • The main items are set automatically.
  • You can set title, author, description, license and tags with --metadata_xxx options in each training script.
  • Merging scripts also support minimum SAI Model Spec metadata. See the help message for the usage.
  • Metadata editor will be available soon.

• bitsandbytes is now optional. Please install it if you want to use it. The insructions are in the later section.

• albumentations is not required anymore.

• --v_pred_like_loss ratio option is added. This option adds the loss like v-prediction loss in SDXL training. 0.1 means that the loss is added 10% of the v-prediction loss. The default value is None (disabled).

  • In v-prediction, the loss is higher in the early timesteps (near the noise). This option can be used to increase the loss in the early timesteps.

• Arbitrary options can be used for Diffusers' schedulers. For example --lr_scheduler_args "lr_end=1e-8".

• LoRA-FA is added experimentally. Specify --network_module networks.lora_fa option instead of --network_module networks.lora. The trained model can be used as a normal LoRA model.

• JPEG XL is supported. #786

• Input perturbation noise is added. See #798 for details.

• Dataset subset now has caption_prefix and caption_suffix options. The strings are added to the beginning and the end of the captions before shuffling. You can specify the options in .toml.

• Intel ARC support with IPEX is added. #825

• Other bug fixes and improvements.

Please read Releases for recent updates. 最近の更新情報は Release をご覧ください。

Naming of LoRA

The LoRA supported by train_network.py has been named to avoid confusion. The documentation has been updated. The following are the names of LoRA types in this repository.

1. LoRA-LierLa : (LoRA for Li n e a r La yers)

   LoRA for Linear layers and Conv2d layers with 1x1 kernel

2. LoRA-C3Lier : (LoRA for C olutional layers with 3 x3 Kernel and Li n e a r layers)

   In addition to 1., LoRA for Conv2d layers with 3x3 kernel

LoRA-LierLa is the default LoRA type for train_network.py (without conv_dim network arg). LoRA-LierLa can be used with our extension for AUTOMATIC1111's Web UI, or with the built-in LoRA feature of the Web UI.

To use LoRA-C3Lier with Web UI, please use our extension.

LoRAの名称について

train_network.py がサポートするLoRAについて、混乱を避けるため名前を付けました。ドキュメントは更新済みです。以下は当リポジトリ内の独自の名称です。

1. LoRA-LierLa : (LoRA for Li n e a r La yers、リエラと読みます)

   Linear 層およびカーネルサイズ 1x1 の Conv2d 層に適用されるLoRA

2. LoRA-C3Lier : (LoRA for C olutional layers with 3 x3 Kernel and Li n e a r layers、セリアと読みます)

   1.に加え、カーネルサイズ 3x3 の Conv2d 層に適用されるLoRA

LoRA-LierLa はWeb UI向け拡張、またはAUTOMATIC1111氏のWeb UIのLoRA機能で使用することができます。

LoRA-C3Lierを使いWeb UIで生成するには拡張を使用してください。

Sample image generation during training

A prompt file might look like this, for example

# prompt 1
masterpiece, best quality, (1girl), in white shirts, upper body, looking at viewer, simple background --n low quality, worst quality, bad anatomy,bad composition, poor, low effort --w 768 --h 768 --d 1 --l 7.5 --s 28

# prompt 2
masterpiece, best quality, 1boy, in business suit, standing at street, looking back --n (low quality, worst quality), bad anatomy,bad composition, poor, low effort --w 576 --h 832 --d 2 --l 5.5 --s 40

Lines beginning with # are comments. You can specify options for the generated image with options like --n after the prompt. The following can be used.

• --n Negative prompt up to the next option.
• --w Specifies the width of the generated image.
• --h Specifies the height of the generated image.
• --d Specifies the seed of the generated image.
• --l Specifies the CFG scale of the generated image.
• --s Specifies the number of steps in the generation.

The prompt weighting such as ( ) and [ ] are working.

サンプル画像生成

プロンプトファイルは例えば以下のようになります。

# prompt 1
masterpiece, best quality, (1girl), in white shirts, upper body, looking at viewer, simple background --n low quality, worst quality, bad anatomy,bad composition, poor, low effort --w 768 --h 768 --d 1 --l 7.5 --s 28

# prompt 2
masterpiece, best quality, 1boy, in business suit, standing at street, looking back --n (low quality, worst quality), bad anatomy,bad composition, poor, low effort --w 576 --h 832 --d 2 --l 5.5 --s 40

# で始まる行はコメントになります。--n のように「ハイフン二個＋英小文字」の形でオプションを指定できます。以下が使用可能できます。

• --n Negative prompt up to the next option.
• --w Specifies the width of the generated image.
• --h Specifies the height of the generated image.
• --d Specifies the seed of the generated image.
• --l Specifies the CFG scale of the generated image.
• --s Specifies the number of steps in the generation.

( ) や [ ] などの重みづけも動作します。