Skip to content

Exclude archived stacks from the scheduler loops - #1496

Merged
aqeelvn merged 1 commit into
mainfrom
scheduler-skip-archived-stacks
Aug 13, 2026
Merged

Exclude archived stacks from the scheduler loops#1496
aqeelvn merged 1 commit into
mainfrom
scheduler-skip-archived-stacks

Conversation

@aqeelvn

@aqeelvn aqeelvn commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1495 (→ #1494#1493).

What

Add .not_archived to the two per-minute scheduler scopes:

  • Stack.schedule_continuous_delivery — was where(continuous_deployment: true) over all stacks
  • Stack.refresh_deployed_revisions — was find_each over all stacks

Why

Both are driven every 60 seconds by the host app's scheduler. Archived stacks are locked (archive! sets lock_reason), so the enqueued jobs are guaranteed no-ops:

  • ContinuousDeliveryJob exits at the lock check — but in shipit-production ~1300 archived stacks were enqueuing ~2,000 no-op jobs/minute (~70% of all CD triggers) onto the deploys queue shared with real deploys. This was mitigated by hand during the 2026-08-11 incident; this makes the fix structural.
  • FetchDeployedRevisionJob is worse than a no-op for archived stacks with fetch steps: it performs a full clone + checkout via with_temporary_working_directory every minute to refresh a revision display nobody can act on.

Behavior notes

  • Unarchiving already re-triggers GithubSyncJob via the sync_github_if_necessary callback, so revived stacks re-enter both loops with fresh state on the next tick — no gap.
  • No behavior change for active stacks.

Tests

  • .schedule_continuous_delivery enqueues for an active CD stack, not for an archived one with continuous_deployment: true
  • .refresh_deployed_revisions still enqueues for active stacks with fetch steps, not for an archived one

@timothysmith0609 timothysmith0609 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — cleanest PR in the stack, and the reasoning holds up on inspection.

I verified the two things the change depends on:

  • not_archived is a real scope (where(archived_since: nil)), and archive! does set archived_since, so the scope and the lock check agree on what "archived" means.
  • The no-gap claim checks out: sync_github_if_necessary fires on archived_since_previously_changed? && archived_since.nil?, so unarchiving re-enters both loops on the next tick.

The FetchDeployedRevisionJob half is the better find of the two — a full clone + checkout every 60 seconds to refresh a revision display on a locked stack is strictly worse than the CD case, which at least exits at the lock check.

Two non-blocking notes:

🟢 Adjacent win while you're in the file

not_archived.find_each.select(&:supports_fetch_deployed_revision?) still deserializes every non-archived stack's cached_deploy_spec JSON once a minute just to discover most of them have no fetch steps. Pre-existing and not introduced here, so entirely your call whether it belongs in this PR — but where.not(cached_deploy_spec: nil) would cut most of that for free, and it's the same loop you're already touching.

🟢 Test fragility

assert enqueued_args.any?, "expected FetchDeployedRevisionJobs for active stacks with fetch steps" passes because several unrelated fixtures happen to carry "fetch": ["echo '42'"]. It'll keep passing, but it's asserting on ambient fixture state rather than on something the test set up. Giving the positive case its own active stack with fetch steps — the way the CD test explicitly sets @stack.update!(continuous_deployment: true) — would make it self-contained.

Worth noting archived_6hours_ago is a Shipit::ReviewStack; STI means the class-method scopes cover it correctly, so the test is valid as written. Just flagging it since it's easy to misread the fixture as a plain Stack.

Stack.schedule_continuous_delivery and Stack.refresh_deployed_revisions
run every minute from the host application's scheduler, but neither
scope excluded archived stacks. Archiving locks a stack, so the jobs
were guaranteed no-ops: ContinuousDeliveryJob bails on the lock and
FetchDeployedRevisionJob refreshes state nobody can see.

At Shopify's scale that was ~1300 archived stacks enqueuing ~2000
no-op ContinuousDeliveryJobs per minute, plus full git clones from
FetchDeployedRevisionJob for archived stacks with fetch steps, all on
the same worker pool as deploys.

Unarchiving already triggers a GithubSyncJob via
sync_github_if_necessary, so a revived stack re-enters both loops
naturally.
@aqeelvn
aqeelvn force-pushed the github-sync-conditional-spec-cache branch from 635a50d to fb8e257 Compare August 12, 2026 14:14
@aqeelvn
aqeelvn force-pushed the scheduler-skip-archived-stacks branch from f0e8263 to 389373b Compare August 12, 2026 14:15
@aqeelvn

aqeelvn commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Both nits taken in the rebased 389373b0: where.not(cached_deploy_spec: nil) added to the refresh loop (a nil-spec stack can't have fetch steps), and the positive test now sets up its own active stack with fetch steps instead of leaning on ambient fixtures.

Base automatically changed from github-sync-conditional-spec-cache to main August 13, 2026 04:52
@aqeelvn
aqeelvn merged commit 5e344cd into main Aug 13, 2026
15 checks passed
@aqeelvn
aqeelvn deleted the scheduler-skip-archived-stacks branch August 13, 2026 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants