Recipe linting, cleaning up dataloader checkpointing (#1245)

Update some comments and remove some unnecessary lines in training
recipes.

Simplifies dataloader checkpointing at the expense of making it less
verbose, we'll see what we need to support here in the future and can
always revert some of this if needed.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Training preserves model heads so contact outputs remain available
during distributed runs.

* **Improved Checkpointing / Resume**
* Dataloader state handling moved into centralized checkpoint helpers;
resume restores dataloader state and advances past the last completed
step.

* **Observability**
* Per-step performance logging added immediately after optimizer
updates.

* **Breaking Changes**
* Checkpoint return types and some checkpoint helper signatures changed
— callers must adapt.

* **Tests / Style**
* Tests updated for checkpoint/dataloader changes; lint rules relaxed
for test files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Peter St. John <[email protected]>

Author: pstjohn

bionemo-recipes/recipes/.ruff.toml

@@ -52,7 +52,7 @@ exclude = [
 [lint.per-file-ignores]
 "esm_nv.py" = ["C901"]
 "__init__.py" = ["D104", "E402", "F401", "F403", "F811"]
-"test_*.py" = ["D"]
+"test_*.py" = ["C", "D"]
 "conftest.py" = ["D"]
 "scripts/*.py" = ["D"]
 "**/*.ipynb" = ["D"]