Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Diff with nvidia main #84

Draft
wants to merge 177 commits into
base: nvidia_main
Choose a base branch
from
Draft

Diff with nvidia main #84

wants to merge 177 commits into from

Conversation

jlamypoirier
Copy link
Collaborator

No description provided.

RaymondLi0 and others added 30 commits March 1, 2022 14:50
@RaymondLi0
add iteration argument to load_checkpoint
04a72ed
@RaymondLi0
fix load_checkpoint when starting from scratch
22005a1
@RaymondLi0
reset consumed_train_samples when finetune=True
e51c3c5
@RaymondLi0
add wandb reporting
884e8b8
@RaymondLi0
add validation loss wandb reporting
ad1f261
@RaymondLi0
skip wandb if not provided
cc54e57
@RaymondLi0
when finetuning, load optimizer state
97954ac
@RaymondLi0
do not load lr scheduler in finetuning
81c72ab
@RaymondLi0
add torchrun support
03c1aa0
@RaymondLi0
reload weights into optimizer after loading model weights
00cfaad
@RaymondLi0
add --finetune-from argument, so there is no need to modify arguments…
ea9ec20 
… to resume a finetuning job
@RaymondLi0
make wandb not a requirement
fdc697f
@RaymondLi0
Merge branch 'main' of github.com:NVIDIA/Megatron-LM into NVIDIA-main
e52d91c
@RaymondLi0
Merge branch 'NVIDIA-main' into load-iter
f45d856
@RaymondLi0
Merge branch 'load-iter' of https://github.com/ElementAI/Megatron-LM …
2f751a5 
…into load-iter
@RaymondLi0
add tokens_per_epoch print
8036392
@bigximik
add: initial port of alibi from bigscience
652625b
@bigximik
chg: change back import for this version of Megatron
9734afd
@bigximik
cnh: direct import of MixedFusedLayerNorm
f49506b
@bigximik
chg: enums moved to megatron root
d3ce018
@bigximik
fix: commented logging funcionality which is not implemented yet
73c7130
@bigximik
chg: refactor for moved enums
b09a8d1
@bigximik
add: port support for positional embedding param from bigscience
4e07321
@bigximik
add: port from bigscience for pos embeding and glu activations args
4ccf237
@RaymondLi0
add multi-query attention logic in attention module
4064969
@RaymondLi0
add kv weight gradient reduction in tensor-parallel group
190e328
@RaymondLi0
more efficient multiquery attention
6fd0c29
@RaymondLi0
raise if trying to uyse multi-query cross-atteention
254ff4b
@RaymondLi0
remove expand_key_value parameter since CoreAttention for multi-query…
eaf6174 
… is now in a separate class
@RaymondLi0
remove most timers
1513137
Raymond Li and others added 29 commits June 13, 2023 21:46
move setting of TORCH_CUDA_ARCH_LIST
ac497ce
fix call to blendable dataset
04031a8
@RaymondLi0
fix blended dataset size in dataset groups
17217f8
@RaymondLi0
Merge pull request #52 from bigcode-project/finetune-starcoder
39a75ee 
Finetune StarCoder Megatron
@RaymondLi0
find_checkpoint_rank_0 returns a single value
0229a69
@RaymondLi0
fix checkpoint merge tools
37353b1
@RaymondLi0
remove --finetune-from argument to make checkpoint loading logic simpler
3dbd929
@RaymondLi0
Merge pull request #58 from bigcode-project/NVIDIA-main
3e22c9f 
merge from Nvidia main
@RaymondLi0
Merge branch 'multi-query-attention' into loubnabnl-patch-1
1397ac0
@RaymondLi0
Merge pull request #54 from bigcode-project/loubnabnl-patch-1
8c1889e 
Add tokens-per-second-gpu to the printed logs instead of just wandb
@jlamypoirier
Skip unnecessary compilation
8196de1
@RaymondLi0
Merge pull request #65 from bigcode-project/skip_compile
2223891 
Skip unnecessary compilation
@loubnabnl
Create pretrain_bigcode_7b.slurm
5b06c12
@loubnabnl
Create pretrain_bigcode_1b.slurm
513d00d
@loubnabnl
Create pretrain_bigcode_3b.slurm
5a9c239
@mayank31398
outputs not matching non-flash case in MQA
a993f05
@mayank31398
Merge branch 'multi-query-attention' into mqa
1809fc1
@RaymondLi0
Merge pull request #71 from mayank31398/mqa
c82a5a1 
Fixed MQA outputs not matching with HF model with non-flash case
@mayank31398
convert reshape to view (#73)
ebea9f2
@jlamypoirier
Support flash attn 2 (#72)
462980b
@huybery
Fix train-iters typo & format script (#74)
ebd38e9
@loubnabnl
Merge pull request #70 from bigcode-project/script_7b-starcoder
bd0aaba 
[WIP] Add training scripts
@loubnabnl
add file level FIM and sanity check
f598110
@loubnabnl
use default None for sanity check interval
fd6d705
@loubnabnl
remove extra prints
4f8a0e4
@loubnabnl
Merge pull request #81 from bigcode-project/file-level-fim
25f3c89 
add file level FIM and sanity check
@loubnabnl
add thera rope as arg
01e9ce6
@loubnabnl
add humaneval generations using a server
c8372cb
@loubnabnl
Revert "add humaneval generations using a server"
7c325cd 
This reverts commit c8372cb.
@jlamypoirier jlamypoirier changed the title Diff with nvidia Diff with nvidia main Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@jlamypoirier @RaymondLi0 @bigximik @lvwerra @loubnabnl @mayank31398 @huybery