Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Added training scripts Independent of Hugging Face's Trainer #27

Open
wants to merge 50 commits into
base: main
Choose a base branch
from
Open

feat: Added training scripts Independent of Hugging Face's Trainer #27

wants to merge 50 commits into from

Conversation

chizuchizu
Copy link

@chizuchizu chizuchizu commented Nov 27, 2023 •
edited
Loading

Added training scripts Independent of Hugging Face's Trainer

Description

Purpose of the PR

  • Added training scripts that don't depend on Hugging Face's Trainer class.

Related Issue(s)

  • Nothing.

Changes

  • Added train_ds.py
  • Added utils scripts imported from Microsoft VisualChat.
  • Added the training config and the training shell script.
  • Update documentation on training and inference methods.

Model/Algorithm Performance

  • Trained the GIT model following projects/opt/exp002_ds.yml with both ZeRO-2 and ZeRO-3, confirming valid inference.
  • Trained Video BLIP with ZeRO-3.
  • Observed an increase in GPU memory consumption compared to when using Trainer.

Dependencies

  • Nothing.

Reviewer Notes

Confirmed

  • I have updated the documentation accordingly.
  • I have adhered to the coding standards and guidelines of this project.
  • I have added comments, especially in hard-to-understand areas.

@chizuchizu
Add [train_ds]: initial commit for train_ds.py
c5b6618
@chizuchizu
WIP update [train_ds]: cp from VisualChat
9974578
@chizuchizu
rm comment and argparser, set fire
9f32e64
@chizuchizu
update [train_ds.py, config]
5c82169
@chizuchizu
fix [train_ds.py]: set to half the input batch and eval model input
62c4ed5
@chizuchizu
add [utils.py]: for train_dspy
62cb28d
@chizuchizu
update [train_ds]: fix input, rm config element
2ac2834
@chizuchizu
fix [train_ds.py]: saving model
ffdf56a
@chizuchizu
add wandb [train_ds.py]
dae2e3b
@chizuchizu
fix avg loss metric
8043bbf
@chizuchizu
update config for train_ds.py[exp_002.yml]
0050f85
@chizuchizu
update [train_ds.py]
555a1c7
@chizuchizu
update train_ds.py
103737c
@chizuchizu
fix print -> print_rank_0[train_ds.py]
97edccb
@chizuchizu
add simple coco (captioning) dataset
ceda701
@chizuchizu
add progressbar, fix model save point(rm merging LoRA while training)
102fbca
@chizuchizu
add beta to config
3ac13c5
@chizuchizu
restore utils.py
1c0a7ea
@chizuchizu
rm comment
51695a2
@chizuchizu
Jp -> En [comment]
0cb821e
@chizuchizu
Log lr to wandb
2aade14
@chizuchizu
support full parameter tuning
02289e1
@chizuchizu
add license
9f02ae7
@chizuchizu
change license
281d1e0
@chizuchizu
add DeepSpeedExamples to acknowledge
debe6be
@chizuchizu
change: coco.yaml -> m3it_coco.yaml
4464c10
@chizuchizu
add notice, copyright
b675e82
@chizuchizu
add todo (merge LoRA)
45ef591
@chizuchizu
add uses [README]
d9d4635
@chizuchizu
fix path
f1acdba
@chizuchizu chizuchizu self-assigned this Nov 27, 2023
@chizuchizu chizuchizu marked this pull request as ready for review November 27, 2023 10:14
@chizuchizu chizuchizu changed the title feat: Add training scripts Independent of Hugging Face's Trainer feat: Added training scripts Independent of Hugging Face's Trainer Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@chizuchizu chizuchizu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@chizuchizu