Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can nunchaku support more Video-generator model? #23

Open
philipwan opened this issue Nov 14, 2024 · 6 comments
Open

Can nunchaku support more Video-generator model? #23

philipwan opened this issue Nov 14, 2024 · 6 comments

Comments

@philipwan
Copy link

In addition to flux, does nunchaku support prompt-to-video models, such as OpenSoraPlan, Cogvideo,etc.
Flux is also a transformer-based model

@lmxyy
Copy link
Collaborator

lmxyy commented Nov 14, 2024

We are looking into this and will definitely support video generation models like Mochi and CogVideoX. Stay tuned.

@nitinmukesh
Copy link

That's good news.

@philipwan
Copy link
Author

We are looking into this and will definitely support video generation models like Mochi and CogVideoX. Stay tuned.

Wow~Good News!!!
I notice there are too many new operators in nunchaku/src, Aren't these universal operators for other model?If it is to adapt to different models, what else does nunchaku need to do? I also want to try it

@lmxyy
Copy link
Collaborator

lmxyy commented Nov 16, 2024

I think we just need to quantize the model with deepcompressor and then convert it to nunchaku format.
Yeap, we have all the operators in the C++ files. We are still cleaning the code to support modularization. With that, I think it is easy to extend nunchaku to new models.

@philipwan
Copy link
Author

I think we just need to quantize the model with deepcompressor and then convert it to nunchaku format. Yeap, we have all the operators in the C++ files. We are still cleaning the code to support modularization. With that, I think it is easy to extend nunchaku to new models.

Thanks!

@philipwan philipwan changed the title Can numchaku support more Video-generator model? Can nunchaku support more Video-generator model? Nov 18, 2024
@marvin-0042
Copy link

marvin-0042 commented Nov 19, 2024

We are looking into this and will definitely support video generation models like Mochi and CogVideoX. Stay tuned.

@lmxyy awesome! thank you so much! Do you have rough, high level timeline when this will support Mochi 1? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants