Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add 'resume training' feature #6

Open
CBroz1 opened this issue Mar 3, 2022 · 2 comments
Open

Add 'resume training' feature #6

CBroz1 opened this issue Mar 3, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@CBroz1
Copy link
Contributor

CBroz1 commented Mar 3, 2022

Our current draft manages config.yaml within the dataframe, but doesn't make adjustments to the model training file located under models: <project_folder>/dlc-models/iteration-<#>/<scorer>/train/pose_cfg.yaml

To resume training from the previous training snapshot, we would need to modify the init_weights value to a full path without extension dlc-models/iteration-<#>/<scorer>/train/snapshot-<#>
@CBroz1 CBroz1 self-assigned this May 4, 2022
@CBroz1 CBroz1 added this to the NIH U24 Year 3 milestone May 4, 2022
@CBroz1 CBroz1 changed the title Training start point is encoded in pose_cfg.yaml Add 'resume training' feature May 5, 2022
@CBroz1 CBroz1 added the enhancement New feature or request label May 5, 2022
@CBroz1 CBroz1 removed this from the 2022Q2 milestone May 6, 2022
@kabilar kabilar added this to the 2023Q2+ milestone May 9, 2022
@CBroz1 CBroz1 added the help wanted Extra attention is needed label May 10, 2022
@CBroz1
Copy link
Contributor Author

CBroz1 commented Jul 7, 2022

partially addressed in upcoming workflow pr

@CBroz1
Copy link
Contributor Author

CBroz1 commented Aug 15, 2022

This function provides the ability to update the training config file to point to the relevant checkpoint. The team will consider migrating this code to the element itself.

@CBroz1 CBroz1 removed the help wanted Extra attention is needed label Aug 15, 2022
@kabilar kabilar removed this from the 2023Q2+ milestone Feb 18, 2023
@CBroz1 CBroz1 removed their assignment Feb 25, 2023
MilagrosMarin pushed a commit to MilagrosMarin/element-deeplabcut that referenced this issue Aug 3, 2023
@CBroz1
Merge pull request datajoint#6 from kabilar/main
bcc4eb9 
Add issue templates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kabilar @CBroz1