Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More improvements to FSDP, benchmark against DDP #13

Merged
merged 30 commits into from
May 2, 2024
Merged

More improvements to FSDP, benchmark against DDP #13

merged 30 commits into from
May 2, 2024

Commits on Apr 19, 2024

  1. leave root params in mem

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    a0e8e84 View commit details
    Browse the repository at this point in the history

  2. fix logic

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    431a140 View commit details
    Browse the repository at this point in the history

  3. don't show mem usage all the time

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    98025a3 View commit details
    Browse the repository at this point in the history

  4. fix

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    60aec7f View commit details
    Browse the repository at this point in the history

  5. make configurable

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    69d74c4 View commit details
    Browse the repository at this point in the history

  6. add alloc/free for unsharded data

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    78146a3 View commit details
    Browse the repository at this point in the history

  7. add alloc/free for unsharded grad

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    e6f5fc2 View commit details
    Browse the repository at this point in the history

  8. fix

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    c343413 View commit details
    Browse the repository at this point in the history

  9. record for

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    d1cf74d View commit details
    Browse the repository at this point in the history

  10. revert 69d74c4 - alloc/free for unsharded grad

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    34765e7 View commit details
    Browse the repository at this point in the history

  11. revert alloc/free trick for unsharded params data

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    7ea0651 View commit details
    Browse the repository at this point in the history

  12. add support for DDP in benchmark

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    ce69f24 View commit details
    Browse the repository at this point in the history

  13. set device ids explicitly

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    dbbf0ae View commit details
    Browse the repository at this point in the history

  14. fix

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    28401f1 View commit details
    Browse the repository at this point in the history

  15. change up how weights are initialized

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    811b6c3 View commit details
    Browse the repository at this point in the history

  16. fix test

    @epwalsh
    epwalsh committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    82378a9 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Add back alloc/free hack for unsharded data

    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    0386841 View commit details
    Browse the repository at this point in the history

  2. Revert "Add back alloc/free hack for unsharded data" 

    This reverts commit 0386841.
    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    5f140f2 View commit details
    Browse the repository at this point in the history

  3. Handle frozen layers with reshard-only post-backward hook

    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    0f408d2 View commit details
    Browse the repository at this point in the history

  4. Revert "Handle frozen layers with reshard-only post-backward hook" 

    This reverts commit 0f408d2.
    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    3d63078 View commit details
    Browse the repository at this point in the history

  5. add to test

    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    450d8e8 View commit details
    Browse the repository at this point in the history

  6. add to test

    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    9c1e52f View commit details
    Browse the repository at this point in the history

  7. Fixes for frozen modules

    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    48abc4d View commit details
    Browse the repository at this point in the history

  8. Divide grad before and after reducing for stability

    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    c41d2d4 View commit details
    Browse the repository at this point in the history

  9. Add support for hybrid sharding

    @epwalsh
    epwalsh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    494be77 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. make grad clipping optional

    @epwalsh
    epwalsh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    a9f8194 View commit details
    Browse the repository at this point in the history

  2. clean up

    @epwalsh
    epwalsh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    6f136b6 View commit details
    Browse the repository at this point in the history

  3. calculate grad norm more efficiently

    @epwalsh
    epwalsh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    d66a683 View commit details
    Browse the repository at this point in the history

  4. Revert "calculate grad norm more efficiently" 

    This reverts commit d66a683.
    @epwalsh
    epwalsh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    d9579b1 View commit details
    Browse the repository at this point in the history

  5. fix

    @epwalsh
    epwalsh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    a48bfb7 View commit details
    Browse the repository at this point in the history