Exclude archived stacks from the scheduler loops - #1496
Conversation
timothysmith0609
left a comment
There was a problem hiding this comment.
Approving — cleanest PR in the stack, and the reasoning holds up on inspection.
I verified the two things the change depends on:
not_archivedis a real scope (where(archived_since: nil)), andarchive!does setarchived_since, so the scope and the lock check agree on what "archived" means.- The no-gap claim checks out:
sync_github_if_necessaryfires onarchived_since_previously_changed? && archived_since.nil?, so unarchiving re-enters both loops on the next tick.
The FetchDeployedRevisionJob half is the better find of the two — a full clone + checkout every 60 seconds to refresh a revision display on a locked stack is strictly worse than the CD case, which at least exits at the lock check.
Two non-blocking notes:
🟢 Adjacent win while you're in the file
not_archived.find_each.select(&:supports_fetch_deployed_revision?) still deserializes every non-archived stack's cached_deploy_spec JSON once a minute just to discover most of them have no fetch steps. Pre-existing and not introduced here, so entirely your call whether it belongs in this PR — but where.not(cached_deploy_spec: nil) would cut most of that for free, and it's the same loop you're already touching.
🟢 Test fragility
assert enqueued_args.any?, "expected FetchDeployedRevisionJobs for active stacks with fetch steps" passes because several unrelated fixtures happen to carry "fetch": ["echo '42'"]. It'll keep passing, but it's asserting on ambient fixture state rather than on something the test set up. Giving the positive case its own active stack with fetch steps — the way the CD test explicitly sets @stack.update!(continuous_deployment: true) — would make it self-contained.
Worth noting archived_6hours_ago is a Shipit::ReviewStack; STI means the class-method scopes cover it correctly, so the test is valid as written. Just flagging it since it's easy to misread the fixture as a plain Stack.
Stack.schedule_continuous_delivery and Stack.refresh_deployed_revisions run every minute from the host application's scheduler, but neither scope excluded archived stacks. Archiving locks a stack, so the jobs were guaranteed no-ops: ContinuousDeliveryJob bails on the lock and FetchDeployedRevisionJob refreshes state nobody can see. At Shopify's scale that was ~1300 archived stacks enqueuing ~2000 no-op ContinuousDeliveryJobs per minute, plus full git clones from FetchDeployedRevisionJob for archived stacks with fetch steps, all on the same worker pool as deploys. Unarchiving already triggers a GithubSyncJob via sync_github_if_necessary, so a revived stack re-enters both loops naturally.
635a50d to
fb8e257
Compare
f0e8263 to
389373b
Compare
|
Both nits taken in the rebased |
Stacked on #1495 (→ #1494 → #1493).
What
Add
.not_archivedto the two per-minute scheduler scopes:Stack.schedule_continuous_delivery— waswhere(continuous_deployment: true)over all stacksStack.refresh_deployed_revisions— wasfind_eachover all stacksWhy
Both are driven every 60 seconds by the host app's scheduler. Archived stacks are locked (
archive!setslock_reason), so the enqueued jobs are guaranteed no-ops:ContinuousDeliveryJobexits at the lock check — but in shipit-production ~1300 archived stacks were enqueuing ~2,000 no-op jobs/minute (~70% of all CD triggers) onto thedeploysqueue shared with real deploys. This was mitigated by hand during the 2026-08-11 incident; this makes the fix structural.FetchDeployedRevisionJobis worse than a no-op for archived stacks withfetchsteps: it performs a full clone + checkout viawith_temporary_working_directoryevery minute to refresh a revision display nobody can act on.Behavior notes
GithubSyncJobvia thesync_github_if_necessarycallback, so revived stacks re-enter both loops with fresh state on the next tick — no gap.Tests
.schedule_continuous_deliveryenqueues for an active CD stack, not for an archived one withcontinuous_deployment: true.refresh_deployed_revisionsstill enqueues for active stacks with fetch steps, not for an archived one