Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Diff with nvidia main #84

Draft
wants to merge 177 commits into
base: nvidia_main
Choose a base branch
from
Draft

Diff with nvidia main #84

wants to merge 177 commits into from

Commits on Mar 1, 2022

  1. add iteration argument to load_checkpoint

    @RaymondLi0
    RaymondLi0 committed Mar 1, 2022
    Configuration menu
    Copy the full SHA
    04a72ed View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2022

  1. fix load_checkpoint when starting from scratch

    @RaymondLi0
    RaymondLi0 committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    22005a1 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2022

  1. reset consumed_train_samples when finetune=True

    @RaymondLi0
    RaymondLi0 committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    e51c3c5 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2022

  1. add wandb reporting

    @RaymondLi0
    RaymondLi0 committed Apr 22, 2022
    Configuration menu
    Copy the full SHA
    884e8b8 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2022

  1. add validation loss wandb reporting

    @RaymondLi0
    RaymondLi0 committed Apr 25, 2022
    Configuration menu
    Copy the full SHA
    ad1f261 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2022

  1. skip wandb if not provided

    @RaymondLi0
    RaymondLi0 committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    cc54e57 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2022

  1. when finetuning, load optimizer state

    @RaymondLi0
    RaymondLi0 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    97954ac View commit details
    Browse the repository at this point in the history

Commits on May 13, 2022

  1. do not load lr scheduler in finetuning

    @RaymondLi0
    RaymondLi0 committed May 13, 2022
    Configuration menu
    Copy the full SHA
    81c72ab View commit details
    Browse the repository at this point in the history

Commits on May 17, 2022

  1. add torchrun support

    @RaymondLi0
    RaymondLi0 committed May 17, 2022
    Configuration menu
    Copy the full SHA
    03c1aa0 View commit details
    Browse the repository at this point in the history

  2. reload weights into optimizer after loading model weights

    @RaymondLi0
    RaymondLi0 committed May 17, 2022
    Configuration menu
    Copy the full SHA
    00cfaad View commit details
    Browse the repository at this point in the history

Commits on May 18, 2022

  1. add --finetune-from argument, so there is no need to modify arguments… 

    … to resume a finetuning job
    @RaymondLi0
    RaymondLi0 committed May 18, 2022
    Configuration menu
    Copy the full SHA
    ea9ec20 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2022

  1. make wandb not a requirement

    @RaymondLi0
    RaymondLi0 committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    fdc697f View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2022

  1. Merge branch 'main' of github.com:NVIDIA/Megatron-LM into NVIDIA-main

    @RaymondLi0
    RaymondLi0 committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    e52d91c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'NVIDIA-main' into load-iter

    @RaymondLi0
    RaymondLi0 committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    f45d856 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'load-iter' of https://github.com/ElementAI/Megatron-LM … 

    …into load-iter
    @RaymondLi0
    RaymondLi0 committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    2f751a5 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2022

  1. add tokens_per_epoch print

    @RaymondLi0
    RaymondLi0 committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    8036392 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2022

  1. add: initial port of alibi from bigscience

    @bigximik
    bigximik committed Jul 22, 2022
    Configuration menu
    Copy the full SHA
    652625b View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2022

  1. chg: change back import for this version of Megatron

    @bigximik
    bigximik committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    9734afd View commit details
    Browse the repository at this point in the history

  2. cnh: direct import of MixedFusedLayerNorm

    @bigximik
    bigximik committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    f49506b View commit details
    Browse the repository at this point in the history

  3. chg: enums moved to megatron root

    @bigximik
    bigximik committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    d3ce018 View commit details
    Browse the repository at this point in the history

  4. fix: commented logging funcionality which is not implemented yet

    @bigximik
    bigximik committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    73c7130 View commit details
    Browse the repository at this point in the history

  5. chg: refactor for moved enums

    @bigximik
    bigximik committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    b09a8d1 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2022

  1. add: port support for positional embedding param from bigscience

    @bigximik
    bigximik committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    4e07321 View commit details
    Browse the repository at this point in the history

  2. add: port from bigscience for pos embeding and glu activations args

    @bigximik
    bigximik committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    4ccf237 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2022

  1. add multi-query attention logic in attention module

    @RaymondLi0
    RaymondLi0 committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    4064969 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2022

  1. add kv weight gradient reduction in tensor-parallel group

    @RaymondLi0
    RaymondLi0 committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    190e328 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2022

  1. more efficient multiquery attention

    @RaymondLi0
    RaymondLi0 committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    6fd0c29 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2022

  1. raise if trying to uyse multi-query cross-atteention

    @RaymondLi0
    RaymondLi0 committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    254ff4b View commit details
    Browse the repository at this point in the history

  2. remove expand_key_value parameter since CoreAttention for multi-query… 

    … is now in a separate class
    @RaymondLi0
    RaymondLi0 committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    eaf6174 View commit details
    Browse the repository at this point in the history

  3. remove most timers

    @RaymondLi0
    RaymondLi0 committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    1513137 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2022

  1. chg: move enums back to model

    @bigximik
    bigximik committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    b4d6017 View commit details
    Browse the repository at this point in the history

  2. fix: breaking circular import

    @bigximik
    bigximik committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    691226a View commit details
    Browse the repository at this point in the history

  3. allow to load old checkpoints

    @RaymondLi0
    RaymondLi0 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    d4ba492 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #5 from bigcode-project/alibi 

    Alibi
    @RaymondLi0
    RaymondLi0 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    5101e5a View commit details
    Browse the repository at this point in the history

  5. Merge branch 'load-iter' into multi-query-attention

    @RaymondLi0
    RaymondLi0 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    d63c4b6 View commit details
    Browse the repository at this point in the history

  6. resolve conflict

    @RaymondLi0
    RaymondLi0 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    5045d6f View commit details
    Browse the repository at this point in the history

  7. implement alibi in multiquery core-attention

    @RaymondLi0
    RaymondLi0 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    2117058 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2022

  1. Merge branch 'multi-query-attention' into main

    @RaymondLi0
    RaymondLi0 committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    16b2b1a View commit details
    Browse the repository at this point in the history

  2. Merge pull request #1 from bigcode-project/main 

    Main
    @RaymondLi0
    RaymondLi0 committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    f8547ff View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2022

  1. add necessary fixes for toolkit-infiniband-example

    @RaymondLi0
    RaymondLi0 committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    e8d47a9 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2022

  1. add FIM code from EleutherAI, some comments and todo

    @RaymondLi0
    RaymondLi0 committed Nov 2, 2022
    Configuration menu
    Copy the full SHA
    a360666 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2022

  1. add a tokenizer-type for FIM

    @RaymondLi0
    RaymondLi0 committed Nov 3, 2022
    Configuration menu
    Copy the full SHA
    4390812 View commit details
    Browse the repository at this point in the history

  2. add spm+psm variants

    @RaymondLi0
    RaymondLi0 committed Nov 3, 2022
    Configuration menu
    Copy the full SHA
    5ab8702 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2022

  1. also permute the segment after last eod token, fix permute boundaries

    @RaymondLi0
    RaymondLi0 committed Nov 4, 2022
    Configuration menu
    Copy the full SHA
    1f85184 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2022

  1. fix data type in permutation

    @RaymondLi0
    RaymondLi0 committed Nov 5, 2022
    Configuration menu
    Copy the full SHA
    1290e49 View commit details
    Browse the repository at this point in the history

  2. truncate or pad after all segments are joined back

    @RaymondLi0
    RaymondLi0 committed Nov 5, 2022
    Configuration menu
    Copy the full SHA
    a5161d7 View commit details
    Browse the repository at this point in the history

  3. some cleanup

    @RaymondLi0
    RaymondLi0 committed Nov 5, 2022
    Configuration menu
    Copy the full SHA
    641af1d View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2022

  1. add preprocessing of HF datasets directly

    @RaymondLi0
    RaymondLi0 committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    66e61e7 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2022

  1. modify max seq-length from 2048 to 8192

    @RaymondLi0
    RaymondLi0 committed Nov 8, 2022
    Configuration menu
    Copy the full SHA
    a79988a View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2022

  1. add missing cases in fused kernels

    @RaymondLi0
    RaymondLi0 committed Nov 14, 2022
    Configuration menu
    Copy the full SHA
    db3809b View commit details
    Browse the repository at this point in the history

  2. add longer sequence lengths in fused kernels test

    @RaymondLi0
    RaymondLi0 committed Nov 14, 2022
    Configuration menu
    Copy the full SHA
    acda627 View commit details
    Browse the repository at this point in the history

  3. larger MAX_TOKENS_TO_OOM

    @RaymondLi0
    RaymondLi0 committed Nov 14, 2022
    Configuration menu
    Copy the full SHA
    d59c85b View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2022

  1. use custom barrier with device_ids

    Raymond Li committed Nov 18, 2022
    Configuration menu
    Copy the full SHA
    7b0cee2 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2022

  1. add HF tokenizer

    Raymond Li committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    93cb6a0 View commit details
    Browse the repository at this point in the history

  2. add special tokens in HF tokenizer

    @RaymondLi0
    RaymondLi0 committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    9f2c442 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #9 from bigcode-project/fim 

    Fim
    @RaymondLi0
    RaymondLi0 committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    8169dec View commit details
    Browse the repository at this point in the history

  4. fix vocab_size in _HFTokenizer

    @RaymondLi0
    RaymondLi0 committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    9fe3bcb View commit details
    Browse the repository at this point in the history

  5. fix: initialize tokenizer with TokenizerFromFile

    Raymond Li committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    6982c4e View commit details
    Browse the repository at this point in the history

  6. Merge branch 'preprocess-hf' of github.com:bigcode-project/Megatron-L… 

    …M into preprocess-hf
    Raymond Li committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    0348b3a View commit details
    Browse the repository at this point in the history

  7. fix: add special_tokens dict for FIM

    Raymond Li committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    4f060a2 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2022

  1. load attention-head-type from checkpoint

    Raymond Li committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    332e8db View commit details
    Browse the repository at this point in the history

  2. attention-head-type defaults to None instead

    Raymond Li committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    0717dab View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2022

  1. use detokenize method un text_generation

    Raymond Li committed Nov 24, 2022
    Configuration menu
    Copy the full SHA
    96daa55 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2022

  1. add mqa conversion to huggingface

    @RaymondLi0
    RaymondLi0 committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    2d36c14 View commit details
    Browse the repository at this point in the history

  2. remove config and tokenizer save

    @RaymondLi0
    RaymondLi0 committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    760eed9 View commit details
    Browse the repository at this point in the history

  3. add Readme

    @RaymondLi0
    RaymondLi0 committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    baa7b3b View commit details
    Browse the repository at this point in the history

  4. add some documentation

    @RaymondLi0
    RaymondLi0 committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    2ceaf70 View commit details
    Browse the repository at this point in the history

  5. add push to hub logic

    leandro committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    66beabe View commit details
    Browse the repository at this point in the history

  6. add docs

    leandro committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    de83476 View commit details
    Browse the repository at this point in the history

  7. convert_checkpoint as function, push starting from last pushed iteration

    @RaymondLi0
    RaymondLi0 committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    1b7c96f View commit details
    Browse the repository at this point in the history

  8. add iter_interval argument

    @RaymondLi0
    RaymondLi0 committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    5cb878f View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2022

  1. use relative imports in modeling file

    @RaymondLi0
    RaymondLi0 committed Dec 8, 2022
    Configuration menu
    Copy the full SHA
    ab1c4cc View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2022

  1. Fixes for MQA (#12)

    @jlamypoirier
    jlamypoirier committed Dec 9, 2022
    Configuration menu
    Copy the full SHA
    93461dc View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2022

  1. Run with toolkit

    @jlamypoirier
    jlamypoirier committed Dec 10, 2022
    Configuration menu
    Copy the full SHA
    63c6fbc View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2022

  1. Redirect output to logs

    @jlamypoirier
    jlamypoirier committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    92e2ca2 View commit details
    Browse the repository at this point in the history

  2. Remove store

    @jlamypoirier
    jlamypoirier committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    e2c2c2b View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2022

  1. update readme

    @RaymondLi0
    RaymondLi0 committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    60fbd1d View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2023

  1. Merge pull request #13 from bigcode-project/run_with_toolkit 

    Structured logs and distributed timeout.
    @RaymondLi0
    RaymondLi0 committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    6c4bf90 View commit details
    Browse the repository at this point in the history

  2. remove debug prints

    Raymond Li committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    732396a View commit details
    Browse the repository at this point in the history

  3. more precise error for attention_type/head_type values

    Raymond Li committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    9d80f8a View commit details
    Browse the repository at this point in the history

  4. attention-head-type defaults to multihead again to avoid breaking pre… 

    …vious configs
    @RaymondLi0
    RaymondLi0 committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    7457e32 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2023

  1. documentation on the --tokenizer-file argument

    @RaymondLi0
    RaymondLi0 committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    cdbcfc9 View commit details
    Browse the repository at this point in the history

  2. add missing newlines

    @RaymondLi0
    RaymondLi0 committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    94306d1 View commit details
    Browse the repository at this point in the history

  3. revert barrier() to torch.distributed.barrier()

    @RaymondLi0
    RaymondLi0 committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    506fbd4 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2023

  1. Merge pull request #10 from bigcode-project/preprocess-hf 

    Preprocess hf
    @RaymondLi0
    RaymondLi0 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    58884e0 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2023

  1. Remove hf transformers tools

    @jlamypoirier
    jlamypoirier committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    d47f623 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2023

  1. add santacoder example script

    @RaymondLi0
    RaymondLi0 committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    c446836 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2023

  1. update arguments in example srcipt

    @RaymondLi0
    RaymondLi0 committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    afd079a View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. add multi-validation for gpt training

    @RaymondLi0
    RaymondLi0 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    a16826e View commit details
    Browse the repository at this point in the history

  2. add subset argument to preprocessing

    Raymond Li committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    c73ff5c View commit details
    Browse the repository at this point in the history

  3. add valid-num-workers argument

    @RaymondLi0
    RaymondLi0 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    1d7768a View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2023

  1. change fim special tokens to use underscore

    @RaymondLi0
    RaymondLi0 committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    3032821 View commit details
    Browse the repository at this point in the history

  2. log tflops

    @RaymondLi0
    RaymondLi0 committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    4978321 View commit details
    Browse the repository at this point in the history

  3. make assert less strict for very small datasets (typically when one e… 

    …poch is less than 5 samples)
    @RaymondLi0
    RaymondLi0 committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    3a6286b View commit details
    Browse the repository at this point in the history

  4. fix fim for new tokenizer

    @RaymondLi0
    RaymondLi0 committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    a950409 View commit details
    Browse the repository at this point in the history

  5. fix fim

    @RaymondLi0
    RaymondLi0 committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    294ef35 View commit details
    Browse the repository at this point in the history

  6. more explicit error when trying to create empty splits

    @RaymondLi0
    RaymondLi0 committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    9ce9611 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. add multi-validation for gpt training (#32) 

    * add multi-validation for gpt training

* add valid-num-workers argument
    @RaymondLi0
    RaymondLi0 committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    654d0d8 View commit details
    Browse the repository at this point in the history

  2. Take MQA into account in flops formula, fix glu-activation factor 

    Co-authored-by: Nouamane Tazi <[email protected]>
    @RaymondLi0 @NouamaneTazi
    RaymondLi0 and NouamaneTazi committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    042a091 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'multi-query-attention' into log-tflops

    @RaymondLi0
    RaymondLi0 committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    e280d3a View commit details
    Browse the repository at this point in the history

  4. adjust formula in comments

    @RaymondLi0
    RaymondLi0 committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    b18ecf6 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'multi-query-attention' into remove_hf_transformers

    @jlamypoirier
    jlamypoirier committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    b0f3cfb View commit details
    Browse the repository at this point in the history

  6. Merge pull request #26 from bigcode-project/remove_hf_transformers 

    Remove hf transformers tools
    @RaymondLi0
    RaymondLi0 committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    e969456 View commit details
    Browse the repository at this point in the history

  7. Kv grad allreduce v2 (#39) 

    Co-authored-by: thomasw21 <[email protected]>
    @jlamypoirier @thomasw21
    jlamypoirier and thomasw21 committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    659295a View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2023

  1. support mqa in checkpoint-merging tools

    @RaymondLi0
    RaymondLi0 committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    bd12802 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #33 from bigcode-project/log-tflops 

    Log tflops and other fixes
    @RaymondLi0
    RaymondLi0 committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    8b38744 View commit details
    Browse the repository at this point in the history

  3. add flash-attn

    @RaymondLi0
    RaymondLi0 committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    7d5154f View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2023

  1. flash-attn: assert that alibi is not used

    @RaymondLi0
    RaymondLi0 committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    118f0a8 View commit details
    Browse the repository at this point in the history

  2. fix import

    @RaymondLi0
    RaymondLi0 committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    d50a89b View commit details
    Browse the repository at this point in the history

  3. update readme

    @RaymondLi0
    RaymondLi0 committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    61fe86d View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2023

  1. raise if using flash-attn with selective recomputation, swap if/else

    @RaymondLi0
    RaymondLi0 committed Mar 24, 2023
    Configuration menu
    Copy the full SHA
    f5019c8 View commit details
    Browse the repository at this point in the history

  2. change back to warning

    @RaymondLi0
    RaymondLi0 committed Mar 24, 2023
    Configuration menu
    Copy the full SHA
    0ff5746 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #41 from bigcode-project/flash-attention 

    Add flash-attn
    @RaymondLi0
    RaymondLi0 committed Mar 24, 2023
    Configuration menu
    Copy the full SHA
    e0b644b View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2023

  1. add token/s/gpu to wandb

    leandro von werra committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    36d0435 View commit details
    Browse the repository at this point in the history

  2. fix distributed optimizer

    leandro von werra committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    b691302 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #43 from bigcode-project/tokens-per-second-gpu 

    add token/s/gpu to wandb
    @lvwerra
    lvwerra committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    c41f2b1 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #44 from bigcode-project/fix-dist-opt 

    fix distributed optimizer
    @lvwerra
    lvwerra committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    86ba4c0 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2023

  1. support checkpoints with distrib optimizer in checkpoint-util

    @RaymondLi0
    RaymondLi0 committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    a8e64f6 View commit details
    Browse the repository at this point in the history

  2. don't load optimizer instead of arbitrarily loading dp-rank 0

    @RaymondLi0
    RaymondLi0 committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    57f21b7 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. add bigcode model slurm script

    loubnabnl committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    22b8611 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2023

  1. Merge pull request #40 from bigcode-project/mqa-checkpoint-utils 

    support mqa in checkpoint-merging tools
    @RaymondLi0
    RaymondLi0 committed May 8, 2023
    Configuration menu
    Copy the full SHA
    1a7d54b View commit details
    Browse the repository at this point in the history

Commits on May 12, 2023

  1. Update slurm script

    @loubnabnl
    loubnabnl committed May 12, 2023
    Configuration menu
    Copy the full SHA
    c988cf2 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2023

  1. Finetune StarCoder Megatron

    @lvwerra
    lvwerra committed May 16, 2023
    Configuration menu
    Copy the full SHA
    0048491 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2023

  1. assert Flash Attention doesn't get arbitrary mask

    @mayank31398
    mayank31398 committed May 22, 2023
    Configuration menu
    Copy the full SHA
    893aaa5 View commit details
    Browse the repository at this point in the history

  2. fix dtypes for new numpy versions

    @mayank31398
    mayank31398 committed May 22, 2023
    Configuration menu
    Copy the full SHA
    d06e737 View commit details
    Browse the repository at this point in the history

  3. fused layer norm

    @mayank31398
    mayank31398 committed May 22, 2023
    Configuration menu
    Copy the full SHA
    0e2415a View commit details
    Browse the repository at this point in the history

  4. move cuda kernels

    @mayank31398
    mayank31398 committed May 22, 2023
    Configuration menu
    Copy the full SHA
    041b733 View commit details
    Browse the repository at this point in the history

  5. add rocm

    @mayank31398
    mayank31398 committed May 22, 2023
    Configuration menu
    Copy the full SHA
    28780a7 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2023

  1. Merge pull request #53 from mayank31398/error-reset 

    assert Flash Attention doesn't get arbitrary mask
    @RaymondLi0
    RaymondLi0 committed May 23, 2023
    Configuration menu
    Copy the full SHA
    beaf2f2 View commit details
    Browse the repository at this point in the history

Commits on May 26, 2023

  1. Merge branch 'multi-query-attention' into ontocord

    @mayank31398
    mayank31398 committed May 26, 2023
    Configuration menu
    Copy the full SHA
    5432115 View commit details
    Browse the repository at this point in the history

  2. Add tokens-per-second-per-gpu to the printed logs instead of just wan… 

    …db logs
    @loubnabnl
    loubnabnl committed May 26, 2023
    Configuration menu
    Copy the full SHA
    22de429 View commit details
    Browse the repository at this point in the history

  3. fix

    @mayank31398
    mayank31398 committed May 26, 2023
    Configuration menu
    Copy the full SHA
    6a77fd0 View commit details
    Browse the repository at this point in the history

  4. fused

    @mayank31398
    mayank31398 committed May 26, 2023
    Configuration menu
    Copy the full SHA
    9008fbe View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. add missing get_batch_per_block

    @mayank31398
    mayank31398 committed May 31, 2023
    Configuration menu
    Copy the full SHA
    23cf759 View commit details
    Browse the repository at this point in the history

  2. increase sequence length to 8k

    @mayank31398
    mayank31398 committed May 31, 2023
    Configuration menu
    Copy the full SHA
    f20d10a View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2023

  1. don't use Apex kernels

    @mayank31398
    mayank31398 committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    cc965d9 View commit details
    Browse the repository at this point in the history

  2. 8192 upper

    @mayank31398
    mayank31398 committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    e9a7e7e View commit details
    Browse the repository at this point in the history

  3. 8192 upper

    @mayank31398
    mayank31398 committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    250ab29 View commit details
    Browse the repository at this point in the history

  4. drop useless script

    @mayank31398
    mayank31398 committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    e8c74d5 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2023

  1. Merge branch 'main' of github.com:NVIDIA/Megatron-LM into NVIDIA-main

    @RaymondLi0
    RaymondLi0 committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    648f32f View commit details
    Browse the repository at this point in the history

  2. fused kernel import

    @mayank31398
    mayank31398 committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    4a33f29 View commit details
    Browse the repository at this point in the history

  3. drop use_kernels_from_apex

    @mayank31398
    mayank31398 committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    21045b5 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. Merge pull request #55 from mayank31398/ontocord 

    add ROCm devices support
    @RaymondLi0
    RaymondLi0 committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    b4efd14 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'multi-query-attention' into NVIDIA-main

    @RaymondLi0
    RaymondLi0 committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    8354f89 View commit details
    Browse the repository at this point in the history

  3. remove unused kernels

    @RaymondLi0
    RaymondLi0 committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    972f301 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. Create finetune_starcoderplus.slurm

    @loubnabnl
    loubnabnl committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    b291323 View commit details
    Browse the repository at this point in the history

  2. try with LayerNorm import from megatron.model

    Raymond Li committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    203b071 View commit details
    Browse the repository at this point in the history

  3. fix the merge

    Raymond Li committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    48c8046 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2023

  1. move setting of TORCH_CUDA_ARCH_LIST

    Raymond Li committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    ac497ce View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2023

  1. fix call to blendable dataset

    Raymond Li committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    04031a8 View commit details
    Browse the repository at this point in the history

  2. fix blended dataset size in dataset groups

    @RaymondLi0
    RaymondLi0 committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    17217f8 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. Merge pull request #52 from bigcode-project/finetune-starcoder 

    Finetune StarCoder Megatron
    @RaymondLi0
    RaymondLi0 committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    39a75ee View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2023

  1. find_checkpoint_rank_0 returns a single value

    @RaymondLi0
    RaymondLi0 committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    0229a69 View commit details
    Browse the repository at this point in the history

  2. fix checkpoint merge tools

    @RaymondLi0
    RaymondLi0 committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    37353b1 View commit details
    Browse the repository at this point in the history

  3. remove --finetune-from argument to make checkpoint loading logic simpler

    @RaymondLi0
    RaymondLi0 committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    3dbd929 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #58 from bigcode-project/NVIDIA-main 

    merge from Nvidia main
    @RaymondLi0
    RaymondLi0 committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    3e22c9f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'multi-query-attention' into loubnabnl-patch-1

    @RaymondLi0
    RaymondLi0 committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    1397ac0 View commit details
    Browse the repository at this point in the history

  6. Merge pull request #54 from bigcode-project/loubnabnl-patch-1 

    Add tokens-per-second-gpu to the printed logs instead of just wandb
    @RaymondLi0
    RaymondLi0 committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    8c1889e View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2023

  1. Skip unnecessary compilation

    @jlamypoirier
    jlamypoirier committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    8196de1 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #65 from bigcode-project/skip_compile 

    Skip unnecessary compilation
    @RaymondLi0
    RaymondLi0 committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    2223891 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. Create pretrain_bigcode_7b.slurm

    @loubnabnl
    loubnabnl committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    5b06c12 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2023

  1. Create pretrain_bigcode_1b.slurm

    @loubnabnl
    loubnabnl committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    513d00d View commit details
    Browse the repository at this point in the history

  2. Create pretrain_bigcode_3b.slurm

    @loubnabnl
    loubnabnl committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    5a9c239 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. outputs not matching non-flash case in MQA

    @mayank31398
    mayank31398 committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    a993f05 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'multi-query-attention' into mqa

    @mayank31398
    mayank31398 committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    1809fc1 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2023

  1. Merge pull request #71 from mayank31398/mqa 

    Fixed MQA outputs not matching with HF model with non-flash case
    @RaymondLi0
    RaymondLi0 committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    c82a5a1 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2023

  1. convert reshape to view (#73)

    @mayank31398
    mayank31398 committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    ebea9f2 View commit details
    Browse the repository at this point in the history

  2. Support flash attn 2 (#72)

    @jlamypoirier
    jlamypoirier committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    462980b View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. Fix train-iters typo & format script (#74)

    @huybery
    huybery committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    ebd38e9 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2023

  1. Merge pull request #70 from bigcode-project/script_7b-starcoder 

    [WIP] Add training scripts
    @loubnabnl
    loubnabnl committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    bd0aaba View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. add file level FIM and sanity check

    @loubnabnl
    loubnabnl committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    f598110 View commit details
    Browse the repository at this point in the history

  2. use default None for sanity check interval

    @loubnabnl
    loubnabnl committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    fd6d705 View commit details
    Browse the repository at this point in the history

  3. remove extra prints

    @loubnabnl
    loubnabnl committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    4f8a0e4 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #81 from bigcode-project/file-level-fim 

    add file level FIM and sanity check
    @loubnabnl
    loubnabnl committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    25f3c89 View commit details
    Browse the repository at this point in the history

  5. add thera rope as arg

    @loubnabnl
    loubnabnl committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    01e9ce6 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. add humaneval generations using a server

    @loubnabnl
    loubnabnl committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    c8372cb View commit details
    Browse the repository at this point in the history

  2. Revert "add humaneval generations using a server" 

    This reverts commit c8372cb.
    @loubnabnl
    loubnabnl committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    7c325cd View commit details
    Browse the repository at this point in the history