[Feature Request]: Introduce new version code strategy

[BenHenning]

Is your feature request related to a problem? Please describe.
Version codes currently need to be manually updated and then cherry-picked for each release candidate going out (#5029 as an example).

Describe the solution you'd like
While there is a plan to move toward automated releases, the amount of develop & release branch churn needed just to bump version codes seems far too high.

I'm proposing that we move to a calculated approach. If we assume the following:

• Every commit to the develop branch is a prospective release.
• Every commit to a release branch is a prospective release candidate.
• We'll never exceed X release candidates per release (and if we did, we would just abandon the release).
• We'll never exceed Y build flavors at one time (removed ones can be repurposed), since each build flavor requires a unique version code.

Then we can calculate that Y version codes are needed per release candidate, and thus X * Y are needed per release branch/commit to develop. This provides us a way to absolutely calculate the release version code in a completely stateless way (or, rather, only depending on the state of Git itself).

The two main challenges here are:

• Finding the correct X & Y values to use since these cannot (easily) change. Ideally, both values should provide maximums above the maximum we'd practically ever hit for each of these individually.
• Finding a way to implement this in Bazel such that version.bzl no longer needs to store version codes.

Tuning X & Y

I think it's a good idea to plan backwards from the maximum, actually. Per https://developer.android.com/studio/publish/versioning#versioningsettings the largest version code that can be accepted is 2100000000 (though https://stackoverflow.com/a/38847981/3689782 indicates this might actually be 2000000000). Technically SDK 28 supports long integer versions (https://stackoverflow.com/a/38847981/3689782) but it's unclear how this interoperates with Play Store. I think we have to assume that's not going to be an option for a long time both because of compatibility and because it's unlikely for us have a min target SDK of 28 for many years.

Based on that, I'm aiming for using no more than 1000 version codes per prospective release as this will net us a capacity of submitting up to 2 million commits to our develop branch before we run out of release "slots." Our current development behavior is comprised entirely of manually reviewed PRs, and I suspect that won't change anytime soon (since the only prospective auto-submitted PRs would be generating version codes, and this proposal avoids that). The team's velocity is approximately 1.3 PRs submitted per day. If we 100x'd that (i.e. 130 PRs merged per day), this upper limit of version codes would require 45 years of sustained 100x velocity before we risk running out of version codes.

With 1000 version codes per release we can tune X and Y as follows:

• X (# of release candidates per release) = 40
• Y (# of unique build flavors) = 25

We need to account for a larger X because each unique cherry-pick could potentially be a new commit (where we aren't squashing/combining them, plus future automation may lead to a 1:1 generation of cherry-picked PRs to release branch commits), as well as potential manual branch changes (e.g. if a failed cherry-pick requires fixing). Since the team is planning on having a fairly fast release cycle, 40 commits to the release branch seems well above anything we'd need.

We currently have 7 build flavors, 2 of which are KitKat-specific and will be removed (per #5012), and 1 of which is unique to a user study and will be removed (per #4419). As we progress, there are essentially two factors which determine the number of build flavors needed:

1. The number of unique release tracks that we maintain (e.g. GA, beta, testing, different alpha tracks, etc.).
2. Different build configurations (e.g. KitKat, ABIs, etc.).

I think we can assume (1) will always be dev + internal testing + alpha + beta + GA + extra alpha/Firebase tracks (so a minimum of 5 build flavors required).

(2) is being simplified with the removal of KitKat, but we can't be sure future releases of Android won't lead to us needing to segment the release in a similar way. This needs to be done carefully since each configuration doubles the number of flavors needed. At 5 base build flavors + 1 extra alpha track, this means we can only support 2 build configurations (putting us at 24 total flavors).

Fortunately, Android App Bundles (AABs) help simplify (2) by providing ways of condensing build configurations into single binaries. While KitKat was a special case, other scenarios are quite possibly going to lead to need different variations of a build flavor based on the device: screen densities, ABIs (if we ever ship native code), and even languages. AABs can support all of these to avoid needing extra flavors (see https://github.com/google/bundletool/blob/f17ce94a4c10555e7ba87bbf4d4f2af35b70cc61/src/main/proto/config.proto#L258 for currently supported split configs). Given this, the likelihood of needing such a configuration is quite low over the long-term, and we're effectively budgeting for 2 simultaneous possibilities.

Note that we can always re-tune X & Y if we don't hit their respective maximums, or by using a different formula after a particular version code.

Residual benefits

This approach guarantees that any single valid version code corresponds to exactly a single unique AAB configuration with unique changes. That can actually simplify our infrastructure as version codes can become a proper check for "is this binary different?" whereas it can't today since multiple commit branches share the same version code.

Describe alternatives you've considered
I considered all of the following (& why I dismissed each):

• Continuing with the original plan to generate version codes: this has the problem of resulting in fairly significant commit churn (at least 1 extra commit to develop per day, and more on days when we need to manually instigate builds, e.g. for cherry-picks).
• Trying to compute the version codes based on the total number of commits in the repository: this has issues if we ever need to clean up history by removing commits, and just simply that the develop branch is no longer the source of truth. Plus, the entire repo has to be pulled to correctly compute the version codes since the count of commits will differ for feature branches.
• Trying to compute version codes by counting releases either using release branches or tags. Release branches don't work well because the fact that we hold release branches indefinitely is not a guaranteed invariant. We likely will remove these branches in the future since they're tagged. Conversely, tags don't quite work because we only ever tag a release after it's finalized (since we want the tag to point to the commit that was released). We could alter the latter strategy slightly, but it's still a bit messy (especially with the prospect of there being multiple types of tags in the future, such as for temporary release branches).
• Using tags as a "storage" mechanism for storing the current release count or base version code. This could work better than the other alternatives, but it still has the disadvantage of not directly tying to the content on the branches themselves.
• Counting all commits on develop and release branches rather than segmenting blocks of version codes for each release. This could work pretty well, however it can result in older versions having higher version codes than newer versions which can result in the wrong binaries going out to users. This is definitely a likely scenario to occur as the team will be managing multiple release branches at any given time (e.g. for alpha/beta/GA).
• Restricting a release to its corresponding track to limit the number of flavors needed. This might be something we look into in the future if we actually run into the risk of running out of version codes, but given that the planned release pipeline includes promoting releases from previous branches, having multiple flavors available is probably going to be helpful to simplify the pipeline's complexity.
• Using a separate branch to hold the version codes. This is messy and quite error-prone as the build system should not be dependent on the underlying repository system.

I'm sure there are many other possibilities--feel free to follow up on this issue thread if you have any that might work better than what's proposed above.

Additional context

Implementation thoughts

Version codes are essentially only needed when configuring the app's APK and AAB binaries, but they're needed as integers at compile time within Bazel. This poses an issue because computing the version codes requires access to the local Git repository and the Git tool. I think rather than trying to get this to work properly in Bazel, we can short-circuit it by removing the version code management in Bazel and just compute the corresponding version within TransformAndroidManifest directly.

As for computing the version codes, there are two parts:

1. Computing the RC01 (first release candidate) version codes for a release (these always correspond to develop commits).
2. Computing later RCs for a given release branch commit.

Both require computing the number of commits that a branch has, and need to be done carefully so that the calculation works both on develop and release branches.

The number of commits to develop can be computed using:

git rev-list --count develop

The number of commits to a particular release branch can be computed using (see https://stackoverflow.com/a/11657647/3689782):

git rev-list --count HEAD ^develop

The above correctly evaluates to 0 when on the develop branch (or any of its earlier branches).

Thus, the formula (expressed in Bash), assuming we don't exceed 1000 version codes with the current solution, would be:

BASE_VERSION_CODE=$((100+(($(git rev-list --count develop)-1835)*1000)+($(git rev-list --count HEAD ^develop)*25)))

Note this intends to offset the calculation by the number of develop commits at the time the solution is implemented (to avoid wasting ~2 million version codes).

Implementation caveats

One important caveat to note on the implementation approach is that it will result in non-release/non-develop branches effectively becoming release candidate-like in their version codes. This will be disruptive to development as switching between commits or even branches on a PR chain should not result in version downgrades during installation.

To mitigate for this, we should verify that the branch the user is on is either develop or a release branch (e.g. starts with "release-") when computing the version codes. If they are not on either, assume they are on their develop branch merge-base (the common commit, i.e. the latest commit of develop from which the branch is based). This will maintain parity with the way version codes are handled today (minus that updating to develop will result in new version codes, but that's already a possibility today).

[BenHenning]

Curious about your thoughts on the approach @seanlip and @adhiamboperes. This is semi-permanent so we need to be careful about how we proceed when it comes to "wasting" version codes, but I think that:

• The buffers being accounted for here are quite a bit higher than anything we'd realistically need across all facets.
• We can always change the system by offsetting at whichever version code we "stop" at (per the time we change the system), and then start counting with the new system. Since it scales linearly with number of develop commits, we can easily predict how much the version code space is shrinking in order to account for changes. It may also be possible to "rewind" version codes, but I am very sure that will be complicated and need to be pursued very carefully. We'll also need to be careful in tracking which version codes are actually release to users to avoid Play Console errors (as it will enforce not reusing version codes).

If you're both fine with this approach, this is something that could probably be implemented while I'm out (though I ask that it's kept from merging until I get back to review it, if that's okay).

[seanlip]

Thanks @BenHenning. I thought about this a bit and I want to make sure we're on the same page about the problem before discussing the solution. What is the actual problem statement? I'm going to have a go at enumerating it in terms that I understand, but it's almost certainly got some wrong bits so please feel free to correct it. (You're welcome to edit this comment if you like.)

Please also feel free to restate the problem in your own terms if what follows is largely off-base.

Background info

• We have the following release tracks: alpha, beta, GA.
• We may deploy release candidates to a subset of these tracks at any time. The release candidates for different tracks are different.
• Each release candidate has a unique version code. There may be other version codes that go unused.
• When an end user has access to two or more version codes, the higher version code will be the one they are able to download.
• Every time we make a deployment, its version code(s) must be higher than all previously-deployed ones.

Use cases

• A user with no special permissions (they have access to beta).
• A user who opts in to do an internal test of an unreleased beta.
• A member of the closed alpha testing group.
• (Are there others? E.g. internal tests of an unreleased alpha? What are the developer use cases and user flows, if any?)

Constraints and Assumptions

This is the part I am not sure about.

• Currently, every time we make a release, we generate release codes for alpha, beta, GA simultaneously. So even if we only wanted to make an alpha release and nothing else, we would still generate release codes for all flavours. Does this continue to be the case?
• We currently have a requirement that, every time we generate release codes, the codes satisfy the relationship: dev > alpha > beta > GA. Does this continue to be the case? I assume so because users can always opt out of alpha to get the beta (and presumably opt out of beta to get the GA version? -- I haven't checked this).
• Is it correct that beta and GA will always be built off a release branch, and that alpha may be built off of a release branch or off of develop?

Proposed Solution

• In your proposal, what is the "formula" that determines the range of version codes allotted to a particular commit? Also, do version codes only exist for the develop and release branches, and not for other branches?
• Let us say we cut a release branch off of develop. Then we make a commit to develop. Then we make a commit to the release branch. Under your system, what determines the numerical ordering of the version codes assigned to each of these two commits? Will the order of those be reversed if we made the commit to the release branch before the commit to develop? (The thing I haven't quite been able to get my head around yet is that there's a time ordering in our deployments and it seems hard to capture this relationship across different branches. I do like the idea of having each build be uniquely identifiable by a version code, but in Web world the corresponding unique codes would simply be the commithashes, rather than integers which also need to satisfy other constraints.)

[BenHenning]

Thanks for summarizing @seanlip. You have most of the details correct, but there are some adjustments needed. I also want to clarify on the points indicated by your questions.

Background info (adjusted)

• We have the following release tracks: internal testing, alpha (2 versions: standard alpha & Kenya-specific), beta, and GA.
• Each release has a single release branch and, once launched, a single release tag.
• Each release has one or more release candidates (that is, the exact commit at which point we can build versions of the app to deploy to users).
• Each release candidate comes in the following build flavors: dev, dev_kitkat*, alpha, alpha kitkat*, alpha kenya*, beta, ga (* are flavors that will soon be removed). We are planning on also adding a "testing" flavor that sits between dev & alpha.
• Build flavors are versions of a particular release candidate from which we can build a binary to upload to Play Store.
• Each build flavor binary must have a unique version code to avoid them conflicting with one another.
• Each deployed binary on Play Store must have a never-before-used version code (only from the perspective of Play Store--if we've used the version code but never uploaded the binary, that's technically still an available version code).
• Play Store will launch exactly whatever version is in its release track. However, both the device and Play Store assume that larger version code is newer, so trying to release a smaller version code on the same release track may result in failures or edge cases.
• Due to the "greater than" property, we order our version codes such that "newer changes" correspond to larger version codes. Simply put: dev > testing > alpha > beta > ga. This ensures that users who move from the ga track to beta, for instance, will actually be able to update to beta (whereas moving back to ga would require waiting for the next ga release to "downgrade").

Use cases

• A user with no special permissions (they have access to beta).
• A user who opts in to do an internal test of an unreleased beta.
• A member of the closed alpha testing group.
• A member of the QA team needing access to perform feature testing.
• A member of the QA team needing access to perform regression testing.**
• A member of the development team needing to be able to build and deploy the changes at HEAD to their local device or emulator.
• A member of the broader Oppia team needing to access the closed alpha testing group for product testing.
• A member of the Android release team needing to promote releases across different channels.

Re: Constraints and Assumptions

Currently, every time we make a release, we generate release codes for alpha, beta, GA simultaneously. So even if we only wanted to make an alpha release and nothing else, we would still generate release codes for all flavours. Does this continue to be the case?
Yes, this continues to be the case. Part of the reason for this (besides pipeline simplicity) is the planned release pipeline goes as follows:

• Every hour (or some cadence below daily): if there's changes and CI is passing, cut a new hourly release branch & push it to the internal testing track (testing flavor).
• Every day, pick the latest hourly build: if it's new and it's passing P0 tests, push it to the alpha track (alpha flavor).
• Every week, pick the latest alpha build: if it's new and it's passing P1 tests, push it to the beta track (beta flavor).
• Every month, pick the latest beta build: if it's new and passing P2 tests & has full QA sign-off, push it to the GA track (GA flavor). Note there's more complexity to this one in particular, but discussing that's outside the scope of this issue.

Notice that in the above a perfect release (i.e. one with no CPs required) would actually use the same base commit for the release all the way to GA which means every version code (minus dev) would actually end up being used. The alternative would be to somehow tie the release branch type to the flavors using its version codes, and that gets really complicated & messy. We really only "lose" version codes in this system due from:

• Regular releases not being promoted (most hourly builds won't become alphas, most alphas won't become betas, etc.).
• Later releases requiring CPs (since then the "earlier" flavors won't be used).

The system proposed is planning against the worst case, but in reality we'd likely have such a large pool of version codes that even with liberally wasting them we should still never run out (see ~40 year calculation). It's also assuming 1 hourly release per PR at 100 PRs per day (which is mathematically not possible, plus CI runs will delay the hourly rate). In reality, it's probably more like ~200 years of version codes even at 100x productivity.

** This is a special case. The regression testing does NOT occur with builds distributed through the Play Store. The reason for this is that the needs of testing for regressions vs. features are different, and we can't have more than one internal testing track. Instead, testers will manually install builds from a shared location (probably an automatically maintained Google Drive folder) that are undergoing regression testing. The assumption also here is that this is largely minimized over time by automated end-to-end testing.

We currently have a requirement that, every time we generate release codes, the codes satisfy the relationship: dev > alpha > beta > GA. Does this continue to be the case? I assume so because users can always opt out of alpha to get the beta (and presumably opt out of beta to get the GA version? -- I haven't checked this).

Yep, exactly for the reason you mentioned we want to maintain this invariant.

Is it correct that beta and GA will always be built off a release branch, and that alpha may be built off of a release branch or off of develop?

Yes to the first, and not quite to the second. See my reply two prompts up to see the planned process for how releases will be promoted from develop to GA.

Re: Proposed Solution

In your proposal, what is the "formula" that determines the range of version codes allotted to a particular commit? Also, do version codes only exist for the develop and release branches, and not for other branches?

Because of the worst case being that 1 commit to develop (i.e. one merged PR) can correspond to a single GA release, we need to make sure that each commit has enough version codes to sustain both the build flavors we manage today (plus others we may add), as well as the number of potential RCs we may create (assuming 1 RC per direct commit to a release branch, e.g. for a cherry-pick). The original comment goes into the specifics on calculating this, but I'm proposing we allocate 1000 version codes per prospective release (i.e. per single commit to develop) which, divided the proposed above, allows for up to 40 release candidates and 25 build flavors per release candidate (both of which should be more than we ever need).

At that amount, we can create 2 million release branches before running out of version codes.

Per your second question, version codes "exist" for all the branches, but we only compute them relative to the develop & release branches. This is an important thing to note because version codes greatly affect the ease with which local development can take place (switching between branches is painful if the version code downgrades as it requires uninstalling the app locally to install a new build).

Let us say we cut a release branch off of develop. Then we make a commit to develop. Then we make a commit to the release branch. Under your system, what determines the numerical ordering of the version codes assigned to each of these two commits? Will the order of those be reversed if we made the commit to the release branch before the commit to develop? (The thing I haven't quite been able to get my head around yet is that there's a time ordering in our deployments and it seems hard to capture this relationship across different branches. I do like the idea of having each build be uniquely identifiable by a version code, but in Web world the corresponding unique codes would simply be the commithashes, rather than integers which also need to satisfy other constraints.)

Per the last bit, I'd also rather use commit hashes (which is what we use for the app version names) in conjunction with build flavors, but unfortunately we have somehow turn that into a monotonically increasing integer.

The quick way to do the math for version code allotment under the proposal is to multiply the number of commits to develop by 1000 (with an initial shift at the time of introducing the system so we don't waste ~1.8 million version codes outright), and then add the number of active build flavors for each commit to the release branch. Following are examples for better illustrating the two cases you asked about. Both cases are assuming the following build flavors: dev, testing, alpha, beta, ga.

Case 1: cut release branch, push to develop, commit to release branch.

• Let's say the commit to develop is commit number 1900 from the start of the repository, and we start with an initial shift of 1844 (per the number of commits checked in as of this comment). This means develop is at commit 57 from the initial shift (it needs to start at 1).
• The base version codes at develop would be:
  • dev: 57000
  • testing: 57001
  • alpha: 57002
  • beta: 57003
  • ga: 57004
• A release branch is cut. It has the same version codes as above (since it contains the same changes as develop currently).
• A commit is made to develop. Its latest now has the version codes:
  • dev: 58000
  • testing: 58001
  • alpha: 58002
  • beta: 58003
  • ga: 58004
• A commit is made to the release branch. Its latest now has the version codes:
  • dev: 57005
  • testing: 57006
  • alpha: 57007
  • beta: 57008
  • ga: 57009

Case 2: cut release branch, commit to release branch, push to develop.

• Taking the same base assumptions as case 1.
• The base version codes at develop would be:
  • dev: 57000
  • testing: 57001
  • alpha: 57002
  • beta: 57003
  • ga: 57004
• A release branch is cut. It has the same version codes as above (since it contains the same changes as develop currently).
• A commit is made to the release branch. Its latest now has the version codes:
  • dev: 57005
  • testing: 57006
  • alpha: 57007
  • beta: 57008
  • ga: 57009
• A commit is made to develop. Its latest now has the version codes:
  • dev: 58000
  • testing: 58001
  • alpha: 58002
  • beta: 58003
  • ga: 58004

It can be seen that the proposed solution is order independent since it's absolutely computing version codes based on a change's position relative to the first commit to the develop branch (where changes on develop increment by 1000 version codes, and changes on release branches increment by the number of build flavors).

Does this better help illustrate how this system would work?

Edit: it should be notes that the proposed solution is using the current number of build flavors available at a given release commit. Since we're using up the version codes in the pool anyway, we could also change the formula to always account for the maximum number of version codes. For example, this changes scenario 1 above to:

Case 1 (revised): cut release branch, push to develop, commit to release branch.

• Same starting conditions.
• The base version codes at develop would be:
  • dev: 57000
  • testing: 57001
  • alpha: 57002
  • beta: 57003
  • ga: 57004
• A release branch is cut. It has the same version codes as above (since it contains the same changes as develop currently).
• A commit is made to develop. Its latest now has the version codes:
  • dev: 58000
  • testing: 58001
  • alpha: 58002
  • beta: 58003
  • ga: 58004
• A commit is made to the release branch. Its latest now has the version codes:
  • dev: 57025
  • testing: 57026
  • alpha: 57027
  • beta: 57028
  • ga: 57029

This has the advantage of making it possible to calculate the exact release number and candidate within that release for any given version code. I'm not sure that really matters, but it's a potential nice side effect of this other approach.