Skip to content

feat(migrate-optimizely): add migration scope policy and bulk execution mode#46

Merged
fabriziodemaria merged 1 commit into
mainfrom
feat/migrate-optimizely-scope-policy
Jul 9, 2026
Merged

feat(migrate-optimizely): add migration scope policy and bulk execution mode#46
fabriziodemaria merged 1 commit into
mainfrom
feat/migrate-optimizely-scope-policy

Conversation

@fabriziodemaria

Copy link
Copy Markdown
Member

Why

A migration test against a large real-world summary export (360 flags) surfaced several problems with the current Phase 1 flow:

  • The export marked hundreds of long-dead experiments as status: running; treating them as live A/B tests would exclude ~90% of the account, while migrating them with the skill's silent 50/50 split assumption would corrupt metrics for any experiment actually in use.
  • Flags whose arms all serve the same variation (a common CMS pattern) were BLOCKED instead of migrated.
  • Per-flag opt-in checkboxes + per-flag confirmation prompts don't scale to 300+ flags.
  • The plan file's Progress table wasn't updated during execute, breaking resume.
  • Gap reports leaked internal vocabulary ("B2 flattened per-flag summary export") that reviewers couldn't act on.

What

SKILL.md

  • New Migration Scope Policy section: every flag is classified (stable rollout / same-variant experiment / stale vs live A/B test / partial-% rollout / bandit / paused / blocked) with safe defaults and a live-vs-stale confirmation driven by days_running and other staleness signals — never by export status alone.
  • New plan step 2 "Review migration scope": counts-first scope summary in Optimizely terminology + plain-English gap reporting rules (what's missing, how many flags, what it means, what unblocks it).
  • Duplicate variation names collapse to a single fully-rolled-out variant (no split between identical arms) instead of blocking.
  • No more silent even-split assumption: splits are only written with an explicit, recorded source (full ruleset API or user-confirmed UI numbers); unknown-split experiments follow the scope policy.
  • New "migrate all eligible" execution mode for large migrations (Phase 1 is a shadow deployment — nothing resolves flags until Phase 2), with the interactive review-each mode still available. Checkbox pre-ticking follows the confirmed scope.
  • Mandatory plan-file updates during execute (Action line + Progress table after every flag, both modes, flag and code plans) — the plan is the resume state.
  • Synthetic UUID flag keys now surface the Optimizely description as the display name everywhere and carry it into the Confidence flag.
  • Documented summary-export blind spots (whitelists/forced variations, exclusion groups, project archived status) as known unknowns to raise with the customer.
  • Smaller fixes: archived/paused questions only asked when the data actually has those fields; overview box states scope exclusions up front and scopes "one PR per flag" to Phase 2; bucketing-ID step asks authenticated vs anonymous IDs; unknown export fields trigger an Optimizely docs lookup instead of a guess; plan template gains an Optimizely→Confidence glossary and an explicit Excluded-flags table.

Test fixtures

  • Four new synthetic summary-export flags exercising the new scope categories: duplicate-name collapse, a stale "running" A/B test (live-vs-stale question), a stable 100% rollout, and a partial 40% rollout — kept in sync between summary-export-sample.json (file path) and server.py (fake-API path), with README coverage table updated.

All fixture data is synthetic; no customer data is included.

Testing

  • server.py imports and serves all 10 summary-scenario flags (_flag_public smoke over both datasets)
  • summary-export-sample.json parses and remains an append-only change
  • python3 -m py_compile clean on all fixture scripts

🤖 Generated with Claude Code

…on mode

Learnings from a real-world migration test against a large summary
export (360 flags):

- New Migration Scope Policy: classify every flag (stable rollout,
  same-variant experiment, stale vs live A/B test, partial-% rollout,
  bandit, paused, blocked) with meeting-aligned defaults; live
  experiments, partial rollouts, and paused flags are excluded by
  default. "status: running" in exports is not trusted — days_running
  and other staleness signals feed a live-vs-stale confirmation.
- New plan step 2 "Review migration scope" with a counts-first scope
  summary and plain-English gap reporting (internal labels like B1/B2
  banned from user-facing output).
- Duplicate variation names collapse to a single fully-rolled-out
  variant instead of blocking.
- The silent even-split assumption is removed: splits are only written
  with an explicit source (ruleset API or user-confirmed UI numbers).
- New execution mode "migrate all eligible" for large migrations
  (shadow deployment; per-flag review remains available), replacing
  mandatory per-flag opt-in for 300+ flag sets.
- Execute loops must update the plan file (Action line + Progress
  table) after every flag, in both modes — the plan is the resume state.
- Synthetic UUID flag keys surface the Optimizely description as the
  display name and carry it into the Confidence flag.
- Documented export blind spots (whitelists, exclusion groups, archived
  project status); archived/paused questions only asked when the data
  actually contains them; bucketing-ID step asks authenticated vs
  anonymous; unknown export fields trigger a docs lookup, not a guess.
- Test fixtures: four new summary-export flags covering the new scope
  categories (duplicate-name collapse, stale running A/B, 100% rollout,
  partial 40% rollout) on both the file and fake-server paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fabriziodemaria fabriziodemaria merged commit 7061682 into main Jul 9, 2026
1 check passed
@fabriziodemaria fabriziodemaria deleted the feat/migrate-optimizely-scope-policy branch July 9, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants