Memory usage: new dynamic cache for models supporting sliding window attention

[Cyrilvallez]

What does this PR do?

This PR introduces DynamicSlidingWindowCache, a new kind of DynamicCache that will stop growing once its size is equal to the sliding window. This allows models using it to have a (dynamic) fix-sized cache, which is a big win for large inputs.
The idea is that it becomes the default for models with sliding window attention when no cache arguments are given in generate. Let me know what you think @gante @ArthurZucker

Here is a simple visual representation of the new cache for mistralai/Mistral-7B-v0.1 (we stop growing after hitting the 4096 sliding window):

BTW: SlidingWindowCache (the static one) is completely broken atm, cannot even be instantiated. Will take a look.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gante]

BTW: SlidingWindowCache (the static one) is completely broken atm, cannot even be instantiated. Will take a look.

@Cyrilvallez in a recent PR, extra super().__init__() lines slipped in. I started working on it on #33297, where it is fixed and a test that checks all caches is added. That PR is actually about an often-requested feature, cache reuse.

I don't think I will be able to continue it next week -- would you like to finish that PR for me? :p (the test needs to be finished, a few cache classes are not yet working according to the test)

[gante]

@Cyrilvallez suggestion: you can paste images directly to a PR header/comment, which will render the image directly here. It's more convenient for the reader than downloading a file 🤗

like this: screenshot -> drag image file into this text box
(old)

(new)

[gante]

LGTM, thank you for adding this cache 💛

Have you confirmed that slow tests for relevant models (like mistral) are passing? Or, at least, they introduce no new failure, in case some tests are failing on main

[gante]

It seems Qwen2 is not happy with these changes :)

[Cyrilvallez]

BTW: SlidingWindowCache (the static one) is completely broken atm, cannot even be instantiated. Will take a look.

@Cyrilvallez in a recent PR, extra super().__init__() lines slipped in. I started working on it on #33297, where it is fixed and a test that checks all caches is added. That PR is actually about an often-requested feature, cache reuse.

I don't think I will be able to continue it next week -- would you like to finish that PR for me? :p (the test needs to be finished, a few cache classes are not yet working according to the test)

Sure, I'll have a look into it next week 🤗

Slow tests with Mistral were passing, but indeed Qwen2 started to complain, I'm investigating

[ArthurZucker]

ping me once ready for review! 🤗