Telemetry (opt-in version) #2366
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
djsaunde
requested review from
winglian,
NanoCode012,
bursteratom and
SalmanMohammadi
winglian
reviewed
Feb 27, 2025
winglian
reviewed
Feb 27, 2025
winglian
reviewed
Feb 27, 2025
winglian
reviewed
Feb 27, 2025
winglian
reviewed
Feb 27, 2025
winglian
approved these changes
Mar 5, 2025
…/ end, error tracking
djsaunde
force-pushed
the
telemetry-opt-in
branch
from
8df8433 to
7f7e566
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Copying from #2359 with minor changes for opt-in telemetry:
Description
Telemetry for Axolotl. Implementation of system described in #2323.
Features the following:
axolotl.telemetry.manager:TelemetryManagerclass that interacts with the PostHog Python SDK and sends telemetry. Handles opt-in (viaDO_NOT_TRACKorAXOLOTL_DO_NOT_TRACKenv vars), organization whitelisting (allbase_models except for those that are whitelisted are redacted), and path sanitization (_pathor_dirconfig (key, value) pairs are redacted).axolotl.telemetry.errors: Function decorator implementation for tracking errors in various important methods in the codebase (preprocess,train,evaluate,inference, etc.). Includes path sanitization to remove local paths, but keeps axolotl paths / other Python library paths for debugging purposes.axolotl.telemetry.runtime_metrics: Dataclass for metrics tracked during model training, including timing metrics, memory metrics, and elapsed steps / epochs.axolotl.telemetry.callbacks: Includes callback for trainer to send telemetry related to the aforementioned runtime metrics class.Motivation and Context
See #2323.
How has this been tested?
Pytest tests for each added module + some manual testing and viewing of telemetry in PostHog console.