[core] LTX Video 0.9.1

[a-r-r-o-w]

To run conversion:

 python3 scripts/convert_ltx_to_diffusers.py --transformer_ckpt_path /raid/aryan/ltx-new/ltx-video-2b-v0.9.1.safetensors --vae_ckpt_path /raid/aryan/ltx-new/ltx-video-2b-v0.9.1.safetensors --output_path /raid/aryan/ltx-diffusers --dtype bf16 --version 0.9.1 --text_encoder_cache_dir /raid/.cache/huggingface/ --save_pipeline

(I've verified that the conversion for v0.9.0 still works after the current modifications to the script)

Inference after conversion:

import torch
from diffusers import LTXPipeline
from diffusers.utils import export_to_video

pipe = LTXPipeline.from_pretrained("/raid/aryan/ltx-diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = "A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
    decode_timestep=0.05,
    generator=torch.Generator(device="cuda").manual_seed(0),
).frames[0]
export_to_video(video, "output.mp4", fps=24)

Output on prompts from the model page:

ltxv-091-output-downscaled.mp4

Will open weights PR to the official repository soon.

cc @yoavhacohen @SapirW

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

that's fast! thanks!

[tin2tin]

Are the 0.9.1 diffusers weights up on HuggingFace?

[a-r-r-o-w]

@tin2tin We're still working with their team on how to host the weights. It might take some time :(

Until then, the model is available here as well: https://huggingface.co/a-r-r-o-w/LTX-Video-0.9.1-diffusers, mostly because I need this weight format for finetuning. Once we have it hosted officially, those can be used instead

[nitinmukesh]

Until then, the model is available here as well: https://huggingface.co/a-r-r-o-w/LTX-Video-0.9.1-diffusers, mostly because I need this weight format for finetuning. Once we have it hosted officially, those can be used instead

Thank you for sharing.

[a-r-r-o-w]

cc @DN6 for single-file related support

[tin2tin]

Wow, this 0.9.1 (with prompt input) delivers very good quality video, and very fast!
First try:

-932937092_A_woman_with_long_brown_hair_and_light_.mp4

I guess img2vid is not implemented yet - I'm getting this error - sorry, if this is just premature to test this:

Python311\site-packages\diffusers\models\autoencoders\autoencoder_kl_ltx.py", line 431, in forward
    timestep=temb.flatten(),
             ^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'flatten'

[a-r-r-o-w]

Oh, taking a look... This PR should worked for I2V as well

[a-r-r-o-w]

@tin2tin Could you try again? Thanks for catching!

[tin2tin]

@a-r-r-o-w That was quick! Yes, it is fixed now! Thank you!

1017385988_Photo_of_Photo_of_A_woman_with_long_bro.mp4

[scarbain]

@tin2tin We're still working with their team on how to host the weights. It might take some time :(

Until then, the model is available here as well: https://huggingface.co/a-r-r-o-w/LTX-Video-0.9.1-diffusers, mostly because I need this weight format for finetuning. Once we have it hosted officially, those can be used instead

Hi @a-r-r-o-w, thanks for this PR merged ! Any script for finetuning or LoRA i2v available somewhere ? :)

[a-r-r-o-w]

@scarbain There's one for T2V here: https://github.com/a-r-r-o-w/finetrainers. I2V will be supported soon!