Fix #1875, #3744, #3735, #2432: Add support for app bundles via Bazel…

…, Proguard, and build flavors (#3750)

* Add support for AABs, build flavors, and proguard.

There are a lot of details to cover here--see the PR for the complete
context.

* Lint & codeowner fixes.

* Fix failures.

- Add missing codeowner
- Add support for configuring base branch reference
- Update CI for dev/alpha AAB builds to use 'develop' since there's no
  origin configured by default in the workflows

* Different attempt to fix bad develop reference in CI.

* Initial commit.

This is needed to open a PR on GitHub. This commit is being made so that
the PR can start off in a broken Actions state.

This also initially disables most non-Bazel workflows to make workflow
iteration faster and less impacting on other team members.

* Introduce infrastructure for batching.

This introduces a new mechanism for passing lists of tests to sharded
test targets in CI, and hooks it up. No actual sharding is occurring
yet. This led to some simplifications in the CI workflow since the
script can be more dynamic in computing the full list of targets (which
also works around a previous bug with instrumentation tests being run).
Java proto lite also needed to be upgraded for the scripts to be able to
use it.

More testing/documentation needed as this functionality continues to
expand.

* Add bucketing strategy.

This simply partitions bucketed groups of targets into chunks of 10 for
each run. Only 3 buckets are currently retained to test sharding in CI
before introducing full support.

* Fix caching & stabilize builds.

Fixes some caching bucket and output bugs. Also, introduces while loop &
keep_going to introduce resilience against app test build failures (or
just test failures in general).

* Increase sharding & add randomization.

Also, enable other workflows.

Note that CI shouldn't fully pass yet since some documentation and
testing needs to be added yet, but this is meant to be a more realistic
test of the CI environment before the PR is finished.

* Improving partitionin & readability.

Adds a human-readable prefix to make the shards look a bit nicer.

Also, adds more fine-tuned partitioning to bucket & reduce shard counts
to improve overall timing. Will need to be tested in CI.

* Add new tests & fix static analysis errors.

* Fix script.

A newly computed variable wasn't updated to be used in an earlier
change.

* Fix broken tests & test configuration.

Add docstrings for proto.

* Fix mistake from earlier commit.

* Try 10 max parallel actions instead.

See
#3757 (comment)
for context.

* Fix another error from an earlier commit.

* Fix mv command so it works on Linux & OSX.

Neither 'mv -t' nor piping to mv work on OSX so we needed to find an
alternative (in this case just trying to move everything). This actually
works a bit better since it's doing a per-file move rather than
accommodating for files that shouldn't be moved (which isn't an issue
since the destination directory is different than the one containing the
AAB file).

.github/CODEOWNERS

@@ -58,7 +58,7 @@ gradlew.bat @BenHenning
 /app/src/main/res/values*/strings.xml @BenHenning
 /app/src/main/res/values*/untranslated_strings.xml @BenHenning
 
-# Proguard configuration.
+# Proguard configurations.
 *.pro @BenHenning
 
 # Lesson assets.
@@ -86,6 +86,9 @@ buf.yaml @anandwana001
 # Binary files.
 *.png @BenHenning
 
+# Configurations for Bazel-built Android App Bundles.
+bundle_config.pb.json @BenHenning
+
 # Important codebase files.
 LICENSE @BenHenning
 NOTICE @BenHenning
@@ -225,3 +228,9 @@ WORKSPACE @BenHenning
 
 # License texts.
 /app/src/main/res/values/third_party_dependencies.xml @BenHenning
+
+# Exemptions to resource shrinking.
+app/src/main/res/raw/shrink_exemptions.xml @BenHenning
+
+# Version tracking.
+version.bzl @BenHenning

.github/workflows/build_tests.yml

@@ -137,3 +137,261 @@ jobs:
         with:
           name: oppia-bazel.apk
           path: /home/runner/work/oppia-android/oppia-android/oppia.apk
+
+  build_oppia_dev_aab:
+    name: Build Oppia AAB (developer flavor)
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-18.04]
+    env:
+      ENABLE_CACHING: false
+      CACHE_DIRECTORY: ~/.bazel_cache
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up JDK 9
+        uses: actions/setup-java@v1
+        with:
+          java-version: 9
+
+      - name: Set up Bazel
+        uses: abhinavsingh/setup-bazel@v3
+        with:
+          version: 4.0.0
+
+      - name: Set up build environment
+        uses: ./.github/actions/set-up-android-bazel-build-environment
+
+      # For reference on this & the later cache actions, see:
+      # https://github.com/actions/cache/issues/239#issuecomment-606950711 &
+      # https://github.com/actions/cache/issues/109#issuecomment-558771281. Note that these work
+      # with Bazel since Bazel can share the most recent cache from an unrelated build and still
+      # benefit from incremental build performance (assuming that actions/cache aggressively removes
+      # older caches due to the 5GB cache limit size & Bazel's large cache size).
+      - uses: actions/cache@v2
+        id: cache
+        with:
+          path: ${{ env.CACHE_DIRECTORY }}
+          key: ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-binary-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-binary-
+            ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-tests-
+            ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-
+
+      # This check is needed to ensure that Bazel's unbounded cache growth doesn't result in a
+      # situation where the cache never updates (e.g. due to exceeding GitHub's cache size limit)
+      # thereby only ever using the last successful cache version. This solution will result in a
+      # few slower CI actions around the time cache is detected to be too large, but it should
+      # incrementally improve thereafter.
+      - name: Ensure cache size
+        env:
+          BAZEL_CACHE_DIR: ${{ env.CACHE_DIRECTORY }}
+        run: |
+          # See https://stackoverflow.com/a/27485157 for reference.
+          EXPANDED_BAZEL_CACHE_PATH="${BAZEL_CACHE_DIR/#\~/$HOME}"
+          CACHE_SIZE_MB=$(du -smc $EXPANDED_BAZEL_CACHE_PATH | grep total | cut -f1)
+          echo "Total size of Bazel cache (rounded up to MBs): $CACHE_SIZE_MB"
+          # Use a 4.5GB threshold since actions/cache compresses the results, and Bazel caches seem
+          # to only increase by a few hundred megabytes across changes for unrelated branches. This
+          # is also a reasonable upper-bound (local tests as of 2021-03-31 suggest that a full build
+          # of the codebase (e.g. //...) from scratch only requires a ~2.1GB uncompressed/~900MB
+          # compressed cache).
+          if [[ "$CACHE_SIZE_MB" -gt 4500 ]]; then
+            echo "Cache exceeds cut-off; resetting it (will result in a slow build)"
+            rm -rf $EXPANDED_BAZEL_CACHE_PATH
+          fi
+
+      - name: Configure Bazel to use a local cache
+        env:
+          BAZEL_CACHE_DIR: ${{ env.CACHE_DIRECTORY }}
+        run: |
+          EXPANDED_BAZEL_CACHE_PATH="${BAZEL_CACHE_DIR/#\~/$HOME}"
+          echo "Using $EXPANDED_BAZEL_CACHE_PATH as Bazel's cache path"
+          echo "build --disk_cache=$EXPANDED_BAZEL_CACHE_PATH" >> $HOME/.bazelrc
+        shell: bash
+
+      - name: Check Bazel environment
+        run: bazel info
+
+      # See https://git-secret.io/installation for details on installing git-secret. Note that the
+      # apt-get method isn't used since it's much slower to update & upgrade apt before installation
+      # versus just directly cloning & installing the project. Further, the specific version
+      # shouldn't matter since git-secret relies on a future-proof storage mechanism for secrets.
+      # This also uses a different directory to install git-secret to avoid requiring root access
+      # when running the git secret command.
+      - name: Install git-secret (non-fork only)
+        if: ${{ env.ENABLE_CACHING == 'true' && github.event.pull_request.head.repo.full_name == 'oppia/oppia-android' }}
+        shell: bash
+        run: |
+          cd $HOME
+          mkdir -p $HOME/gitsecret
+          git clone https://github.com/sobolevn/git-secret.git git-secret
+          cd git-secret && make build
+          PREFIX="$HOME/gitsecret" make install
+          echo "$HOME/gitsecret" >> $GITHUB_PATH
+          echo "$HOME/gitsecret/bin" >> $GITHUB_PATH
+
+      - name: Decrypt secrets (non-fork only)
+        if: ${{ env.ENABLE_CACHING == 'true' && github.event.pull_request.head.repo.full_name == 'oppia/oppia-android' }}
+        env:
+          GIT_SECRET_GPG_PRIVATE_KEY: ${{ secrets.GIT_SECRET_GPG_PRIVATE_KEY }}
+        run: |
+          cd $HOME
+          # NOTE TO DEVELOPERS: Make sure to never print this key directly to stdout!
+          echo $GIT_SECRET_GPG_PRIVATE_KEY | base64 --decode > ./git_secret_private_key.gpg
+          gpg --import ./git_secret_private_key.gpg
+          cd $GITHUB_WORKSPACE
+          git secret reveal
+
+      # Note that caching only works on non-forks.
+      - name: Build Oppia developer AAB (with caching, non-fork only)
+        if: ${{ env.ENABLE_CACHING == 'true' && github.event.pull_request.head.repo.full_name == 'oppia/oppia-android' }}
+        env:
+          BAZEL_REMOTE_CACHE_URL: ${{ secrets.BAZEL_REMOTE_CACHE_URL }}
+        run: |
+          bazel build --remote_http_cache=$BAZEL_REMOTE_CACHE_URL --google_credentials=./config/oppia-dev-workflow-remote-cache-credentials.json -- //:oppia_dev
+
+      - name: Build Oppia developer AAB (without caching, or on a fork)
+        if: ${{ env.ENABLE_CACHING == 'false' || github.event.pull_request.head.repo.full_name != 'oppia/oppia-android' }}
+        run: |
+          bazel build -- //:oppia_dev
+
+      - name: Copy Oppia APK for uploading
+        run: cp $GITHUB_WORKSPACE/bazel-bin/oppia_dev.aab /home/runner/work/oppia-android/oppia-android/
+
+      - uses: actions/upload-artifact@v2
+        with:
+          name: oppia_dev.aab
+          path: /home/runner/work/oppia-android/oppia-android/oppia_dev.aab
+
+  build_oppia_alpha_aab:
+    name: Build Oppia AAB (alpha flavor)
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-18.04]
+    env:
+      ENABLE_CACHING: false
+      CACHE_DIRECTORY: ~/.bazel_cache
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up JDK 9
+        uses: actions/setup-java@v1
+        with:
+          java-version: 9
+
+      - name: Set up Bazel
+        uses: abhinavsingh/setup-bazel@v3
+        with:
+          version: 4.0.0
+
+      - name: Set up build environment
+        uses: ./.github/actions/set-up-android-bazel-build-environment
+
+      # For reference on this & the later cache actions, see:
+      # https://github.com/actions/cache/issues/239#issuecomment-606950711 &
+      # https://github.com/actions/cache/issues/109#issuecomment-558771281. Note that these work
+      # with Bazel since Bazel can share the most recent cache from an unrelated build and still
+      # benefit from incremental build performance (assuming that actions/cache aggressively removes
+      # older caches due to the 5GB cache limit size & Bazel's large cache size).
+      - uses: actions/cache@v2
+        id: cache
+        with:
+          path: ${{ env.CACHE_DIRECTORY }}
+          key: ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-binary-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-binary-
+            ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-tests-
+            ${{ runner.os }}-${{ env.CACHE_DIRECTORY }}-bazel-
+
+      # This check is needed to ensure that Bazel's unbounded cache growth doesn't result in a
+      # situation where the cache never updates (e.g. due to exceeding GitHub's cache size limit)
+      # thereby only ever using the last successful cache version. This solution will result in a
+      # few slower CI actions around the time cache is detected to be too large, but it should
+      # incrementally improve thereafter.
+      - name: Ensure cache size
+        env:
+          BAZEL_CACHE_DIR: ${{ env.CACHE_DIRECTORY }}
+        run: |
+          # See https://stackoverflow.com/a/27485157 for reference.
+          EXPANDED_BAZEL_CACHE_PATH="${BAZEL_CACHE_DIR/#\~/$HOME}"
+          CACHE_SIZE_MB=$(du -smc $EXPANDED_BAZEL_CACHE_PATH | grep total | cut -f1)
+          echo "Total size of Bazel cache (rounded up to MBs): $CACHE_SIZE_MB"
+          # Use a 4.5GB threshold since actions/cache compresses the results, and Bazel caches seem
+          # to only increase by a few hundred megabytes across changes for unrelated branches. This
+          # is also a reasonable upper-bound (local tests as of 2021-03-31 suggest that a full build
+          # of the codebase (e.g. //...) from scratch only requires a ~2.1GB uncompressed/~900MB
+          # compressed cache).
+          if [[ "$CACHE_SIZE_MB" -gt 4500 ]]; then
+            echo "Cache exceeds cut-off; resetting it (will result in a slow build)"
+            rm -rf $EXPANDED_BAZEL_CACHE_PATH
+          fi
+
+      - name: Configure Bazel to use a local cache
+        env:
+          BAZEL_CACHE_DIR: ${{ env.CACHE_DIRECTORY }}
+        run: |
+          EXPANDED_BAZEL_CACHE_PATH="${BAZEL_CACHE_DIR/#\~/$HOME}"
+          echo "Using $EXPANDED_BAZEL_CACHE_PATH as Bazel's cache path"
+          echo "build --disk_cache=$EXPANDED_BAZEL_CACHE_PATH" >> $HOME/.bazelrc
+        shell: bash
+
+      - name: Check Bazel environment
+        run: bazel info
+
+      # See https://git-secret.io/installation for details on installing git-secret. Note that the
+      # apt-get method isn't used since it's much slower to update & upgrade apt before installation
+      # versus just directly cloning & installing the project. Further, the specific version
+      # shouldn't matter since git-secret relies on a future-proof storage mechanism for secrets.
+      # This also uses a different directory to install git-secret to avoid requiring root access
+      # when running the git secret command.
+      - name: Install git-secret (non-fork only)
+        if: ${{ env.ENABLE_CACHING == 'true' && github.event.pull_request.head.repo.full_name == 'oppia/oppia-android' }}
+        shell: bash
+        run: |
+          cd $HOME
+          mkdir -p $HOME/gitsecret
+          git clone https://github.com/sobolevn/git-secret.git git-secret
+          cd git-secret && make build
+          PREFIX="$HOME/gitsecret" make install
+          echo "$HOME/gitsecret" >> $GITHUB_PATH
+          echo "$HOME/gitsecret/bin" >> $GITHUB_PATH
+
+      - name: Decrypt secrets (non-fork only)
+        if: ${{ env.ENABLE_CACHING == 'true' && github.event.pull_request.head.repo.full_name == 'oppia/oppia-android' }}
+        env:
+          GIT_SECRET_GPG_PRIVATE_KEY: ${{ secrets.GIT_SECRET_GPG_PRIVATE_KEY }}
+        run: |
+          cd $HOME
+          # NOTE TO DEVELOPERS: Make sure to never print this key directly to stdout!
+          echo $GIT_SECRET_GPG_PRIVATE_KEY | base64 --decode > ./git_secret_private_key.gpg
+          gpg --import ./git_secret_private_key.gpg
+          cd $GITHUB_WORKSPACE
+          git secret reveal
+
+      # Note that caching only works on non-forks.
+      - name: Build Oppia alpha AAB (with caching, non-fork only)
+        if: ${{ env.ENABLE_CACHING == 'true' && github.event.pull_request.head.repo.full_name == 'oppia/oppia-android' }}
+        env:
+          BAZEL_REMOTE_CACHE_URL: ${{ secrets.BAZEL_REMOTE_CACHE_URL }}
+        run: |
+          bazel build --compilation_mode=opt --remote_http_cache=$BAZEL_REMOTE_CACHE_URL --google_credentials=./config/oppia-dev-workflow-remote-cache-credentials.json -- //:oppia_alpha
+
+      - name: Build Oppia alpha AAB (without caching, or on a fork)
+        if: ${{ env.ENABLE_CACHING == 'false' || github.event.pull_request.head.repo.full_name != 'oppia/oppia-android' }}
+        run: |
+          bazel build --compilation_mode=opt -- //:oppia_alpha
+
+      - name: Copy Oppia APK for uploading
+        run: cp $GITHUB_WORKSPACE/bazel-bin/oppia_alpha.aab /home/runner/work/oppia-android/oppia-android/
+
+      - uses: actions/upload-artifact@v2
+        with:
+          name: oppia_alpha.aab
+          path: /home/runner/work/oppia-android/oppia-android/oppia_alpha.aab

BUILD.bazel

@@ -1,5 +1,7 @@
 # TODO(#1532): Rename file to 'BUILD' post-Gradle.
 
+load("//:build_flavors.bzl", "AVAILABLE_FLAVORS", "define_oppia_binary_flavor")
+
 # Corresponds to being accessible to all Oppia targets. This should be used for production APIs &
 # modules that may be used both in production targets and in tests.
 package_group(
@@ -78,8 +80,16 @@ android_binary(
         "versionCode": "0",
         "versionName": "0.1-alpha",
     },
-    multidex = "native",  # TODO(#1875): Re-enable legacy for optimized release builds.
+    multidex = "native",
     deps = [
         "//app",
     ],
 )
+
+# Define all binary flavors that can be built. Note that these are AABs, not APKs, and can be
+# be installed on a local device or emulator using a 'bazel run' command like so:
+#  bazel run //:install_oppia_dev
+[
+    define_oppia_binary_flavor(flavor = flavor)
+    for flavor in AVAILABLE_FLAVORS
+]

WORKSPACE

@@ -3,7 +3,7 @@ This file lists and imports all external dependencies needed to build Oppia Andr
 """
 
 load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
-load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_jar")
 load("//third_party:versions.bzl", "HTTP_DEPENDENCY_VERSIONS", "get_maven_dependencies")
 
 # Android SDK configuration. For more details, see:
@@ -157,6 +157,13 @@ load("@android_test_support//:repo.bzl", "android_test_repositories")
 
 android_test_repositories()
 
+# Android bundle tool.
+http_jar(
+    name = "android_bundletool",
+    sha256 = HTTP_DEPENDENCY_VERSIONS["android_bundletool"]["sha"],
+    url = "https://github.com/google/bundletool/releases/download/{0}/bundletool-all-{0}.jar".format(HTTP_DEPENDENCY_VERSIONS["android_bundletool"]["version"]),
+)
+
 # Note to developers: new dependencies should be added to //third_party:versions.bzl, not here.
 maven_install(
     artifacts = DAGGER_ARTIFACTS + get_maven_dependencies(),

app/src/main/res/raw/shrink_exemptions.xml

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Reference: https://developer.android.com/studio/build/shrink-code#keep-resources. -->
+<resources xmlns:tools="http://schemas.android.com/tools"
+  tools:keep="@string/default_web_client_id,@string/firebase_database_url,@string/gcm_defaultSenderId,@string/google_api_key,@string/google_app_id,@string/google_crash_reporting_api_key,@string/google_storage_bucket,@string/project_id,@string/com.crashlytics.android.build_id" />