Is torch.compile compatible or already integrated with TGI? #1359
Unanswered
dmorgan-clairvoyant-ai
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hope this isn't a dumb question...
This article (https://pytorch.org/blog/accelerating-generative-ai-2/#step-1-reducing-cpu-overhead-through-torchcompile-and-a-static-kv-cache-1070-toks) describes optimization tactics utilizing pytorch, specifically torch.compile. I'm curious if these optimizations are already built into tgi or somehow integrated with other equivalent techniques. Thank you in advance for any insight
Beta Was this translation helpful? Give feedback.
All reactions