Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New dynamic cache for sliding window attention #34352

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open

New dynamic cache for sliding window attention #34352

wants to merge 60 commits into from

Commits on Nov 1, 2024

  1. Add new dynamic cache

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    96f0100 View commit details
    Browse the repository at this point in the history

  2. Add cache by default in generate for models supporting it

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    568f807 View commit details
    Browse the repository at this point in the history

  3. Add to __init__ and correct typo

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    be62f53 View commit details
    Browse the repository at this point in the history

  4. Correct output if prefill larger than sliding window + compatibility

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    52b920d View commit details
    Browse the repository at this point in the history

  5. Add legacy format handling

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    0c0836e View commit details
    Browse the repository at this point in the history

  6. style

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    b087839 View commit details
    Browse the repository at this point in the history

  7. add docs

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    91a1fee View commit details
    Browse the repository at this point in the history

  8. fix import

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    a662de2 View commit details
    Browse the repository at this point in the history

  9. Update dummy_pt_objects.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    ce203ea View commit details
    Browse the repository at this point in the history

  10. Update test

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    1719984 View commit details
    Browse the repository at this point in the history

  11. style

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    05d1053 View commit details
    Browse the repository at this point in the history

  12. update cache conversion in test

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    cb2de20 View commit details
    Browse the repository at this point in the history

  13. style

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    7dfc86d View commit details
    Browse the repository at this point in the history

  14. Allow the cache to support new states of more than 1 token, even afte… 

    …r prefill stage
    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    cc13c4c View commit details
    Browse the repository at this point in the history

  15. Update cache_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    1887bda View commit details
    Browse the repository at this point in the history

  16. maybe change test

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    ad86620 View commit details
    Browse the repository at this point in the history

  17. revert tests diffs

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    a6f0d8d View commit details
    Browse the repository at this point in the history

  18. define get_seen_tokens

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    5394c77 View commit details
    Browse the repository at this point in the history

  19. Modify all current .get_seq_length names

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    25b8f80 View commit details
    Browse the repository at this point in the history

  20. style

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    daae19a View commit details
    Browse the repository at this point in the history

  21. trigger CIs

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    6b7cb5a View commit details
    Browse the repository at this point in the history

  22. Add tests

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    9cc7077 View commit details
    Browse the repository at this point in the history

  23. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    e217859 View commit details
    Browse the repository at this point in the history

  24. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    cd7002e View commit details
    Browse the repository at this point in the history

  25. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    86bbd86 View commit details
    Browse the repository at this point in the history

  26. Update causal mask generation in case of DynamicSlidingCache (only Mi… 

    …stral)
    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    851c8c8 View commit details
    Browse the repository at this point in the history

  27. Improve tests

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    a95ca15 View commit details
    Browse the repository at this point in the history

  28. improve cache

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    94861ea View commit details
    Browse the repository at this point in the history

  29. add exceptions

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    82aecd8 View commit details
    Browse the repository at this point in the history

  30. Update utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    8f48003 View commit details
    Browse the repository at this point in the history

  31. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    c0c07f9 View commit details
    Browse the repository at this point in the history

  32. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    c3fe380 View commit details
    Browse the repository at this point in the history

  33. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    12a7576 View commit details
    Browse the repository at this point in the history

  34. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    22abbb6 View commit details
    Browse the repository at this point in the history

  35. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    d82001e View commit details
    Browse the repository at this point in the history

  36. Update 4d mask creation in Mistral

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    005a187 View commit details
    Browse the repository at this point in the history

  37. fix missed conflict

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    d240402 View commit details
    Browse the repository at this point in the history

  38. Apply to other models

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    a7ae24d View commit details
    Browse the repository at this point in the history

  39. Add required arg in prepare_inoput

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    e0de263 View commit details
    Browse the repository at this point in the history

  40. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    8bc872a View commit details
    Browse the repository at this point in the history

  41. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    5ae270b View commit details
    Browse the repository at this point in the history

  42. Fix kv_seq_length and rotary_seq_length

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    5e248fa View commit details
    Browse the repository at this point in the history

  43. up

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    4c04f89 View commit details
    Browse the repository at this point in the history

  44. up

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    cba5ae4 View commit details
    Browse the repository at this point in the history

  45. up

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    d200e77 View commit details
    Browse the repository at this point in the history

  46. up

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    2e40fd3 View commit details
    Browse the repository at this point in the history

  47. CIs

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    50731b5 View commit details
    Browse the repository at this point in the history

  48. improve sdpa is_causal escape

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    bbd6069 View commit details
    Browse the repository at this point in the history

  49. make fix-copies

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    6ae7ec0 View commit details
    Browse the repository at this point in the history

  50. add check for models with sliding window

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    efc1131 View commit details
    Browse the repository at this point in the history

  51. Update modeling_git.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    6478126 View commit details
    Browse the repository at this point in the history

  52. style

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    2055eda View commit details
    Browse the repository at this point in the history

  53. Update modeling_mimi.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    35dd895 View commit details
    Browse the repository at this point in the history

  54. Update utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    fb231cf View commit details
    Browse the repository at this point in the history

  55. replace get_seq_length

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    8affc79 View commit details
    Browse the repository at this point in the history

  56. Update test_utils.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    bc5b036 View commit details
    Browse the repository at this point in the history

  57. CIs

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    09bab35 View commit details
    Browse the repository at this point in the history

  58. CIs

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    6c80731 View commit details
    Browse the repository at this point in the history

  59. Update modeling_longt5.py

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    5efa057 View commit details
    Browse the repository at this point in the history

  60. Update skip test for moshi

    @Cyrilvallez
    Cyrilvallez committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    0f83f21 View commit details
    Browse the repository at this point in the history