Rolling Restyler releases

[pbrisbin]

Motivation

For this project, keeping maintenance low is a priority, especially "incident response". More so than feature velocity. But I would still like to be able to take on larger changes and ship them. Ideally, we'd have solid, multi-level testing in place to prevent breakage that requires response, but that's not always possible (and boosting it is really low-fun work).

Can we make working on Restyler safer without spending cycles on test coverage improvement?

Idea

Move selection of a Restyler version (the Docker image tag) out of App configuration and into a run-time choice, driven by an Admin or API-managed DB table.

0- Steady state

restyler_versions
  name, tag, percentage
  current, 20200301, 100
  next, null, 0

1- CI attempts to release 20200409

restyler_versions
  name, tag, percentage
  current, 20200301, 100
  next, 20200409, 0

2- Slowly increase next, if metrics go south, go back to (1)

restyler_versions
  name, tag, percentage
  current, 20200301, 90
  next, 20200409, 10

...

restyler_versions
  name, tag, percentage
  current, 20200301, 10
  next, 20200409, 90

3- Finalize release, to (0) with new version

restyler_versions
  name, tag, percentage
  current, 20200409, 100
  next, null, 0

Open Questions

• We don't have a programattic API yet, what does that look like? We do now!
• If a roll-out is in progress when we ship another version, we should probably maintain a queue of pending releases (a naive hard-fail there goes directly against our goals), what does that look like?
• What does "if metrics go south" look like?
• This only makes Restyler changes safe, how do we fold more into this design?
  • I.e. does Restyled.Backend need to be shifted down into restyler (and so covered by versioning)?

Extensions

repos could grow a restyler_version column. If NULL, they get the thing by percentage, but if set they (always) run that. It could be a name like current (to opt-out of the rollout group), something arbitrary like pb/secret-branch (to staff-ship new features), or an explicit tag like 20200301 (for uber-conservative users). A version of this was easy to do on its own, and has been implemented already.