Skip to content

Fix TrackioCallback fails to log evaluation metrics after training ends#46935

Open
lewtun wants to merge 3 commits into
mainfrom
fix-trackio-sync
Open

Fix TrackioCallback fails to log evaluation metrics after training ends#46935
lewtun wants to merge 3 commits into
mainfrom
fix-trackio-sync

Conversation

@lewtun

@lewtun lewtun commented Jun 27, 2026

Copy link
Copy Markdown
Member

What does this PR do?

This PR fixes the following issue: TrackioCallback.on_train_end() calls trackio.finish(), which clears Trackio’s active run. However, the callback keeps _initialized=True. If user code calls trainer.evaluate() or trainer.predict() after trainer.train(), the callback later receives on_log()/on_predict(). Because _initialized is still True, it skips setup() and calls trackio.log() without an active Trackio run, raising:

RuntimeError: Call trackio.init() before trackio.log().

The fix is to reset TrackioCallback._initialized after trackio.finish() in on_train_end(). Then subsequent logging re-runs setup() and reinitializes/resumes the Trackio run before logging metrics.

I've also added a regression test which triggers the problem on main and verified the fix resolves it.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

  • I confirm that this is not a pure code agent PR.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline and the
    Pull Request checks?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes according to the guidelines?
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@lewtun lewtun changed the title Fix trackio sync Fix TrackioCallback fails to log evaluation metrics after training ends Jun 27, 2026
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

is_torch_available,
)
from transformers.integrations.integration_utils import KubeflowCallback, SwanLabCallback
from transformers.integrations.integration_utils import KubeflowCallback, SwanLabCallback, TrackioCallback

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a regression test since we didn't have any at all for trackio. Happy to remove it if that's preferred

@lewtun

lewtun commented Jun 27, 2026

Copy link
Copy Markdown
Member Author

The failing tests seem unrelated to my changes

@github-actions

Copy link
Copy Markdown
Contributor

CI Dashboard: View test results in Grafana

@qgallouedec qgallouedec left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't try myself but the justification and fix lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants