Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory usage: new dynamic cache for models supporting sliding window attention #33619

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Memory usage: new dynamic cache for models supporting sliding window attention #33619

wants to merge 15 commits into from
+170 −13

Commits on Sep 20, 2024

  1. Add new dynamic cache

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6a33e28 View commit details
    Browse the repository at this point in the history

  2. Add cache by default in generate for models supporting it

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    257f10b View commit details
    Browse the repository at this point in the history

  3. Add to __init__ and correct typo

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    80c2434 View commit details
    Browse the repository at this point in the history

  4. Correct output if prefill larger than sliding window + compatibility

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f8a5553 View commit details
    Browse the repository at this point in the history

  5. Add legacy format handling

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    693808f View commit details
    Browse the repository at this point in the history

  6. Update utils.py

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    220a543 View commit details
    Browse the repository at this point in the history

  7. Update utils.py

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1f0687d View commit details
    Browse the repository at this point in the history

  8. style

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    0f05dc4 View commit details
    Browse the repository at this point in the history

  9. add docs

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    fc92887 View commit details
    Browse the repository at this point in the history

  10. fix import

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ce2b70d View commit details
    Browse the repository at this point in the history

  11. Update dummy_pt_objects.py

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    2ba750e View commit details
    Browse the repository at this point in the history

  12. Update test

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f4c3df9 View commit details
    Browse the repository at this point in the history

  13. style

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    89b56a4 View commit details
    Browse the repository at this point in the history

  14. fix typo

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    617726c View commit details
    Browse the repository at this point in the history

  15. update cache conversion in test

    @Cyrilvallez
    Cyrilvallez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    4f2e6b2 View commit details
    Browse the repository at this point in the history