Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically determine/override config settings when loading from checkpoint #275

Open
jstjohn opened this issue Oct 8, 2024 · 1 comment
Assignees

Comments

@jstjohn
Copy link
Collaborator

jstjohn commented Oct 8, 2024

Currently checkpoint loading flow requires options to be specified in a list in sub-packages/bionemo-llm/src/bionemo/llm/model/biobert/model.py, specifically OVERRIDE_BIONEMO_CONFIG_DEFAULTS. We should come up with a CLI that "knows" what a user wants to change and applies those changes over the config in some easy to explain precedence procedure.

@jstjohn
Copy link
Collaborator Author

jstjohn commented Oct 8, 2024

@skothenhill-nv let's include this as something we demonstrate we can improve on in our future CLI. Something where the runner is "smart" about which settings a user wants to override. Some kind of sane precedence rules. Maybe a case for something like a config with user-defined CLI overrides.

jstjohn added a commit that referenced this issue Oct 8, 2024
…pretrain2 checkpoints. (#276)

* Fix issue where we were only partially setting TP changes during checkpoint resumption

* Reference issue #275
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants