This repository has been archived by the owner on Dec 1, 2024. It is now read-only.

How to split execution of prefill and decode for Flexgen? #138

Open

sunchaesk opened this issue Jul 9, 2024 · 0 comments

Contributor

sunchaesk commented Jul 9, 2024

I was wondering what would be the best way to split the execution of flexgen to prefill and decode only.

How should I save the values from prefill and how should I load them when I am running flexgen again for decode only.

Thanks

sunchaesk changed the title ~~How to split prefill and decode for Flexgen?~~ How to split execution of prefill and decode for Flexgen?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.