miner.py: implement checks and error handling for failing axons #105
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The miner template seemingly assumes that starting an axon never fails. In e2e tests the axon failed to start, due to mixing next_asyncio in bittensor and regular asyncio in uvicorn. It would be better to terminate the miner process when the axon never starts at all. This patch addresses this by: - wrapping run() in a try/except (this is a must in any Python threading application) - signalling exceptions from worker to main thread in a thread safe manner - terminating the miner if starting the axon fails - monitoring and reporting on whether the axon still runs Whether to keep the miner running if axon issues arise later is another question; the code as-is indicates this is indented behavior: "# In case of unforeseen errors, the miner will log the error and continue operations." so this is not changed. This patch depends on another patch to bittensor that adds .is_running() and .exception to class axon.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The miner template seemingly assumes that starting an axon never fails. In e2e tests (that use this template as their miner) the axon failed to start, due to mixing next_asyncio in bittensor and regular asyncio in uvicorn. It would be better to terminate the miner process when the axon never starts at all.
This patch addresses this by:
Whether to keep the miner running if axon issues arise later is another question; the code as-is indicates this is indented behavior: "# In case of unforeseen errors, the miner will log the error and continue operations." so this is not changed.
This patch depends on another patch to bittensor that adds .is_running() and .exception to class axon.