New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[New Feature] Will MLA Be Supported? #28

Open

RanchiZhao opened this issue Aug 8, 2024 · 0 comments

RanchiZhao commented Aug 8, 2024

We've successfully applied W4A8KV4 quantization using QoQ to an MLA-based model, kind of like DeepSeek v2, but we built it from scratch. We then tested it out on both perplexity metrics and a bunch of benchmarks, and it performed really well.
Now, we're looking to actually speed things up with this W4A8KV4 model.
Any Helps or Suggestions? We are looking forward to your replies!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment