Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add BERT e2e training test #467

Open
wants to merge 50 commits into
base: main
Choose a base branch
from
Open

Add BERT e2e training test #467

wants to merge 50 commits into from

Commits on Jun 26, 2024

  1. Add python training script, requirements.txt (dependencies), and dock… 

    …erfile for the e2e BERT training task
    @mattcjo
    mattcjo committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    94d99b1 View commit details
    Browse the repository at this point in the history

  2. Add github action to build bert-testing image on PR

    @mattcjo
    mattcjo committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    7f50245 View commit details
    Browse the repository at this point in the history

  3. Specify directory the BERT training image should be built in for the … 

    …github action
    @mattcjo
    mattcjo committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    58ccf5d View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Add default values and include in docker env for MASTER_ADDR and MAST… 

    …ER_HOST
    @mattcjo
    mattcjo committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    9c30c45 View commit details
    Browse the repository at this point in the history

  2. Slightly change env var value retrieval. Also ran a formatter to pret… 

    …ty it up.
    @mattcjo
    mattcjo committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    4c82b9a View commit details
    Browse the repository at this point in the history

  3. Create new base directory structure for BERT training test

    @mattcjo
    mattcjo committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    1b615c4 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'aws:main' into bert-training-test

    @mattcjo
    mattcjo authored Jun 27, 2024
    Configuration menu
    Copy the full SHA
    7dfb3d6 View commit details
    Browse the repository at this point in the history

  5. Add main_test.go from nvidia test suite

    @mattcjo
    mattcjo committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    7fe40e2 View commit details
    Browse the repository at this point in the history

  6. Create file to run bert training test

    @mattcjo
    mattcjo committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    9ee61f8 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. base manifest for mpi training

    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    3cbd31a View commit details
    Browse the repository at this point in the history

  2. Remove unused dependency

    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    eccc502 View commit details
    Browse the repository at this point in the history

  3. Add python training script, requirements.txt (dependencies), and dock… 

    …erfile for the e2e BERT training task
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    af9fda0 View commit details
    Browse the repository at this point in the history

  4. Add github action to build bert-testing image on PR

    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    104fa93 View commit details
    Browse the repository at this point in the history

  5. Specify directory the BERT training image should be built in for the … 

    …github action
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    477f672 View commit details
    Browse the repository at this point in the history

  6. Add default values and include in docker env for MASTER_ADDR and MAST… 

    …ER_HOST
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    fb7d18f View commit details
    Browse the repository at this point in the history

  7. Slightly change env var value retrieval. Also ran a formatter to pret… 

    …ty it up.
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    b5aedc7 View commit details
    Browse the repository at this point in the history

  8. Update bert training dockerfile to include amazon specific packages f… 

    …or MPI, NCCL, and EFA.
    @mattcjo
    mattcjo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    7f9480b View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Change Dockerfile.bert-training file name to just Dockerfile

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    19613e1 View commit details
    Browse the repository at this point in the history

  2. Update git workflow to use new Dockerfile path since the name was upd… 

    …ated
    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    974da50 View commit details
    Browse the repository at this point in the history

  3. Update Docker image to use Python version 3.10.12 and build from sour… 

    …ce to be consistent with the other test images
    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    5b4ae1a View commit details
    Browse the repository at this point in the history

  4. Update image that is used

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    0df187d View commit details
    Browse the repository at this point in the history

  5. Updated dockerfile using Python 3.10.12 and installing from source

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    831d228 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into bert-training-test

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    8144b57 View commit details
    Browse the repository at this point in the history

  7. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6bc3ef4 View commit details
    Browse the repository at this point in the history

  8. Remove extra line

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    fa8d244 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'main' into bert-training-test

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    72cba38 View commit details
    Browse the repository at this point in the history

  10. Remove comments from training manifest

    @mattcjo
    mattcjo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    b05db50 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Remove memory limits from manifest

    @mattcjo
    mattcjo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    bfc18dc View commit details
    Browse the repository at this point in the history

  2. Add print statements to training script for debugging

    @mattcjo
    mattcjo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    4933d64 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. Had been setting MASTER_ADDR and MASTER_PORT env vars twice. Removed … 

    …duplicate
    @mattcjo
    mattcjo committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    f87ba65 View commit details
    Browse the repository at this point in the history

  2. Set each process to a GPU via local rank instead of overall rank

    @mattcjo
    mattcjo committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    7af6b13 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    1a3ad52 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. Change comment describing section in dockerfile

    @mattcjo
    mattcjo committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    1f5b1c9 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Merge branch 'aws:main' into main

    @mattcjo
    mattcjo authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    b67026c View commit details
    Browse the repository at this point in the history

  2. parameterize number of gpus per node in Dockerfile and train.py

    @mattcjo
    mattcjo committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    4a8e0ec View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Merge remote-tracking branch 'origin/main' into bert-training-test

    @mattcjo
    mattcjo committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    920aff0 View commit details
    Browse the repository at this point in the history

  2. Rename directory from bert-training to just training

    @mattcjo
    mattcjo committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    c0b78b5 View commit details
    Browse the repository at this point in the history

  3. rename manifest bert-mpi-training.yaml to bert-training.yaml

    @mattcjo
    mattcjo committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    4af829b View commit details
    Browse the repository at this point in the history

  4. Change MPIJob name from bert-mpi-training to bert-training

    @mattcjo
    mattcjo committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    67a1266 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. Remove efa manifest since it currently isn't being utilized

    @mattcjo
    mattcjo committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    a61e833 View commit details
    Browse the repository at this point in the history

  2. Update finalized bert-training manifest that's parameterized

    @mattcjo
    mattcjo committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    d08bec1 View commit details
    Browse the repository at this point in the history

  3. Add steps to tear down manifests that were applied in both tests

    @mattcjo
    mattcjo committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    31ba8ee View commit details
    Browse the repository at this point in the history

  4. Delete old, unused dockerfile

    @mattcjo
    mattcjo committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    6fa38c5 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    60ddc02 View commit details
    Browse the repository at this point in the history

  2. formatting in train.py

    @mattcjo
    mattcjo committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    01d8270 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Merge remote-tracking branch 'upstream/main'

    @mattcjo
    mattcjo committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    21fd336 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into bert-training-test

    @mattcjo
    mattcjo committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    ed40f33 View commit details
    Browse the repository at this point in the history

  3. Add training test e2e Go binary to Dockerfile.kubetest2

    @mattcjo
    mattcjo committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    2df0a42 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Merge branch 'aws:main' into bert-training-test

    @mattcjo
    mattcjo authored Aug 30, 2024
    Configuration menu
    Copy the full SHA
    da86684 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Merge branch 'aws:main' into bert-training-test

    @mattcjo
    mattcjo authored Sep 12, 2024
    Configuration menu
    Copy the full SHA
    4f4229f View commit details
    Browse the repository at this point in the history