Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Batch Optimization Scripts for NVIDIA Instances #498

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

Add Batch Optimization Scripts for NVIDIA Instances #498

wants to merge 23 commits into from

Commits on Jul 11, 2024

  1. Add python training script, requirements.txt (dependencies), and dock… 

    …erfile for the e2e BERT training task
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    af9fda0 View commit details
    Browse the repository at this point in the history

  2. Add github action to build bert-testing image on PR

    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    104fa93 View commit details
    Browse the repository at this point in the history

  3. Specify directory the BERT training image should be built in for the … 

    …github action
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    477f672 View commit details
    Browse the repository at this point in the history

  4. Add default values and include in docker env for MASTER_ADDR and MAST… 

    …ER_HOST
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    fb7d18f View commit details
    Browse the repository at this point in the history

  5. Slightly change env var value retrieval. Also ran a formatter to pret… 

    …ty it up.
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    b5aedc7 View commit details
    Browse the repository at this point in the history

  6. Update bert training dockerfile to include amazon specific packages f… 

    …or MPI, NCCL, and EFA.
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    7f9480b View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Change Dockerfile.bert-training file name to just Dockerfile

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    19613e1 View commit details
    Browse the repository at this point in the history

  2. Update git workflow to use new Dockerfile path since the name was upd… 

    …ated
    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    974da50 View commit details
    Browse the repository at this point in the history

  3. Update Docker image to use Python version 3.10.12 and build from sour… 

    …ce to be consistent with the other test images
    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    5b4ae1a View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6bc3ef4 View commit details
    Browse the repository at this point in the history

  5. Remove extra line

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    fa8d244 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. Had been setting MASTER_ADDR and MASTER_PORT env vars twice. Removed … 

    …duplicate
    @mattcjo
    mattcjo committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    f87ba65 View commit details
    Browse the repository at this point in the history

  2. Set each process to a GPU via local rank instead of overall rank

    @mattcjo
    mattcjo committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    7af6b13 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    1a3ad52 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. Change comment describing section in dockerfile

    @mattcjo
    mattcjo committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    1f5b1c9 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Merge branch 'aws:main' into main

    @mattcjo
    mattcjo authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    b67026c View commit details
    Browse the repository at this point in the history

  2. parameterize number of gpus per node in Dockerfile and train.py

    @mattcjo
    mattcjo committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    4a8e0ec View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    60ddc02 View commit details
    Browse the repository at this point in the history

  2. formatting in train.py

    @mattcjo
    mattcjo committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    01d8270 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    21fd336 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Merge branch 'aws:main' into main

    @mattcjo
    mattcjo authored Aug 30, 2024
    Configuration menu
    Copy the full SHA
    f250ede View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Add nvidia batch optimization scripts for both training and inference

    @mattcjo
    mattcjo committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    f000ec6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'aws:main' into batch-optimization-nvidia

    @mattcjo
    mattcjo authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    1907a82 View commit details
    Browse the repository at this point in the history