fix: do not count downtime towards crawler_runtime after migration or resurrection - #2204
Draft
Pijukatel wants to merge 3 commits into
Draft
fix: do not count downtime towards crawler_runtime after migration or resurrection#2204Pijukatel wants to merge 3 commits into
Pijukatel wants to merge 3 commits into
Conversation
… resurrection When a crawler run was resumed from a persisted state, two issues inflated the reported `crawler_runtime`: - `Statistics.__aenter__` started the periodic logger before updating `crawler_last_started_at`, so the first statistics log of the resumed run computed the runtime from the previous run's start time and included all the downtime between the runs (later stats were computed correctly, which made the value appear to "jump back"). - When the state came from a run that did not finish cleanly (migration, abort), `StatisticsState.model_post_init` computed the accumulated runtime offset using the current time, permanently baking the downtime between the runs into all subsequent statistics. The offset is now computed from the moment the state was last persisted instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2204 +/- ##
=======================================
Coverage 93.74% 93.74%
=======================================
Files 181 181
Lines 12852 12853 +1
=======================================
+ Hits 12048 12049 +1
Misses 804 804
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The first periodic statistics log must report exactly the runtime accumulated by previous runs: zero for a fresh start, the previous total for a resumed run. `Statistics.__aenter__` now resets the run timestamps before starting the periodic logger and sets the new start time after the initial log entry is printed. The accumulated runtime is now also restored from the persisted `crawlerRuntimeMillis` value instead of being reconstructed from the last run's timestamps, so runs before the most recent one are no longer lost from the total, and a state persisted mid-run keeps the correct runtime. The timestamp-based reconstruction is kept as a fallback for states persisted by older versions. `crawlerRuntimeMillis` is also serialized as milliseconds now, matching its name and the other `*Millis` fields, instead of an ISO 8601 duration string. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy
… on Python 3.10 `datetime.fromisoformat` does not accept the 'Z' suffix until Python 3.11, which failed the resumed-run test on the 3.10 CI jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When a crawler run is resumed from a persisted state (Actor migration or resurrection), the reported
crawler_runtimewas inflated by the downtime between the runs, in several ways:Statistics.__aenter__started the periodic logger whilecrawler_last_started_atstill pointed at the previous run's start, so the first statistics log of the resumed run included all the downtime between the runs. Later stats were computed from the new start time, which made the value appear to jump from a large number (e.g. 38s) back down to the real runtime (e.g. 542ms). The run timestamps are now reset before the periodic logger starts and the new start time is set after the initial log entry, so the first log reports exactly the runtime accumulated so far: zero for a fresh start, the previous total for a resumed run.StatisticsState.model_post_initcomputed the accumulated runtime offset using the current time, so the downtime was permanently baked into all subsequent statistics of the resumed run.The accumulated runtime is now restored directly from the persisted
crawlerRuntimeMillisvalue, which is correct for both clean and unclean shutdowns and accumulates across any number of resurrections. The timestamp-based reconstruction is kept as a fallback for states persisted by older versions, usingstats_persisted_at(the moment the state was last saved) to approximate when an uncleanly finished run stopped.crawlerRuntimeMillisis also serialized as milliseconds now, matching its name and the other*Millisfields, instead of an ISO 8601 duration string.Issues
crawler_runtimeof 38.01s that later dropped to 542.8ms.Testing
tests/unit/_statistics/test_persistence.py: the first periodic log of a fresh run reports exactly zero, the first log of a resumed run reports the previous runtime without a simulated 2-hour downtime, the runtime offset stays correct after an unclean shutdown, and the total runtime accumulates across multiple resurrections instead of keeping only the last run's segment. All fail without the fix and pass with it.poe lint,poe type-check, and the statistics and basic-crawler unit tests pass locally.Checklist
🤖 Generated with Claude Code
https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy