feat(migrate-optimizely): add migration scope policy and bulk execution mode#46
Merged
Merged
Conversation
…on mode Learnings from a real-world migration test against a large summary export (360 flags): - New Migration Scope Policy: classify every flag (stable rollout, same-variant experiment, stale vs live A/B test, partial-% rollout, bandit, paused, blocked) with meeting-aligned defaults; live experiments, partial rollouts, and paused flags are excluded by default. "status: running" in exports is not trusted — days_running and other staleness signals feed a live-vs-stale confirmation. - New plan step 2 "Review migration scope" with a counts-first scope summary and plain-English gap reporting (internal labels like B1/B2 banned from user-facing output). - Duplicate variation names collapse to a single fully-rolled-out variant instead of blocking. - The silent even-split assumption is removed: splits are only written with an explicit source (ruleset API or user-confirmed UI numbers). - New execution mode "migrate all eligible" for large migrations (shadow deployment; per-flag review remains available), replacing mandatory per-flag opt-in for 300+ flag sets. - Execute loops must update the plan file (Action line + Progress table) after every flag, in both modes — the plan is the resume state. - Synthetic UUID flag keys surface the Optimizely description as the display name and carry it into the Confidence flag. - Documented export blind spots (whitelists, exclusion groups, archived project status); archived/paused questions only asked when the data actually contains them; bucketing-ID step asks authenticated vs anonymous; unknown export fields trigger a docs lookup, not a guess. - Test fixtures: four new summary-export flags covering the new scope categories (duplicate-name collapse, stale running A/B, 100% rollout, partial 40% rollout) on both the file and fake-server paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nicklasl
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A migration test against a large real-world summary export (360 flags) surfaced several problems with the current Phase 1 flow:
status: running; treating them as live A/B tests would exclude ~90% of the account, while migrating them with the skill's silent 50/50 split assumption would corrupt metrics for any experiment actually in use.What
SKILL.md
days_runningand other staleness signals — never by exportstatusalone.descriptionas the display name everywhere and carry it into the Confidence flag.Test fixtures
summary-export-sample.json(file path) andserver.py(fake-API path), with README coverage table updated.All fixture data is synthetic; no customer data is included.
Testing
server.pyimports and serves all 10 summary-scenario flags (_flag_publicsmoke over both datasets)summary-export-sample.jsonparses and remains an append-only changepython3 -m py_compileclean on all fixture scripts🤖 Generated with Claude Code