[Feature] Question about kvcache offloading to disk or CPU memory

[Yang-x-Zhao]

Motivation

In sglang's paper, it mentions that sglang plans to implement a kvcache offloading mechanism. I notice that there is a disable_disk_cache=False option in sglang's Server args, and wonder if this option controls the kvcache offloading mechanism.

Related resources

No response

[merrymercy]

disable_disk_cache is for caching other things like the regex conversion. The kvcache offloading mechanism has not been implemented and we plan to support it in the next few months.