Consolidated-dockerfiles

.github/workflows/README.md

@@ -1,7 +1,7 @@
 # Understanding the CI/CD workflows
 
 > [!NOTE]
-> At the moment, this workflow does not build for the ARM architecture. Please build these locally until Microsoft releases the feature for private repositories. [source one](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories), [source two](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for--private-repositories). For now, Jetson builds done online will simply be skipped.
+> At the moment, this workflow does not build for the ARM architecture. Please build these locally until Microsoft releases the feature for private repositories. [source one](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories), [source two](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for--private-repositories). For now, Jetson builds done online will simply be commented out in [`build.yaml`](/.github/workflows/build.yaml).
 
 ## Desired behaviour
 1.  The user creates a new branch for their changes.
@@ -22,26 +22,16 @@ This is a high-level understanding of the workflow described in [build.yaml](/.g
 
 1.  The workflow is triggered when a tag starting with `v` (or `dev`) is pushed to the repository.
 
-1.  Firstly, the workflow checks out the code from the repository and looks for changes. At a time, a change may be present in a full directory only. We do not test for individual Dockerfiles because certain images depend on others, and we want to build them in the correct order.
-
-    Places tested for changes:
-    -   `ROS2/AMD64x86/`: desktop images for AMD64x86 architecture
-    -   `ROS2/Jetson`: edge-computing images for NVIDIA Jetson architecture (once implemented)
-    -   `.github/workflows/`: GitHub Actions workflows
-
 1.  Next, the workflow looks for the Git tag. If it finds a tag that starts with `v`/`dev`, records this to a variable, so that it may be used later in the workflow.
 
-1.  If changes were detected, the workflow proceeds to build the images. If any GitHub Action changes were detected, everything is rebuilt; otherwise only the changed images are rebuilt.
-
 1.  The remainder of the build is split into five sections, which run in three steps (some sections run in parallel):
 
     | Step | Sections that run in parallel |
     | --- | --- |
-    | 1 | - ROS2 base humble images (AMD64/x86) <br> - ROS2 base humble images (Jetson/ARM) |
-    | 2 | - ROS2 humble_harmonic (AMD64/x86) |
-    | 3 | - ROS2 wheelchair2_base images (AMD64/x86) <br> - ROS2 wheelchair2_base_jetson (depends on humble_jetson) |
+    | 1 | - ROS2 base `humble` images (AMD64/x86) <br> - ROS2 base `humble` images (Jetson/ARM) |
+    | 2 | - ROS2 `wheelchair2_base` images (AMD64/x86) <br> - ROS2 wheelchair2_base_jetson (depends on `humble_jetson`) |
 
-    By default, all of these sections are configured as [matrix strategies](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow). Although some sections have only one entry, this allows for easy expansion in the future.
+    By default, all of these sections are configured as [matrix strategies](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow). Although some sections have only one entry as of now, this allows for easy expansion in the future.
 
 ## Overview of the reusable workflow
 It quickly became evident that several steps in the workflow were repetitive, so the majority of the actual work was refactored into a [reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows). This is described in [build-workflow.yaml](/.github/workflows/build-workflow.yaml), and is briefly explained below.
@@ -50,9 +40,10 @@ This workflow is triggered by the main workflow, and receives the following inpu
 
 | Input | Description | Default |
 | --- | --- | --- |
-| `build_context` | The build context for the Dockerfile | - |
-| `image_name` | The target name of the image to build | - |
+| `docker_build_context` | The build context for the Dockerfile | - |
+| `target_image_name` | The target name of the image to build | - |
 | `image_version` | The version tag of the image to build | - |
+| `base_image` | The base image to use for the build | - |
 | `runs_on` | The GitHub runner to use for the build | `ubuntu-latest` |
 
 First, code is checked out, [buildx is setup](https://github.com/docker/setup-buildx-action/tree/v3/), and [GHCR is authenticated](https://github.com/docker/setup-buildx-action/tree/v3/).
@@ -65,35 +56,47 @@ Here, we discuss only the potentially confusing parts of the main workflow, whic
 
 You are encouraged to read the [GitHub Actions documentation](https://docs.github.com/en/actions) for more information on how GitHub Actions work, how to write workflows, and reference for syntax. If you use VS Code to write workflows, the [GitHub Actions extension](https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-github-actions) is useful.
 
-### Checking for changes
-We use the [`dorny/paths-filter`](https://github.com/dorny/paths-filter/tree/v2/) action to check for changes in the repository. This action allows us to specify paths to check for changes, and it will return a boolean value indicating whether any changes were detected.
+### Check conditions and set variables
+We want our workflow to run only if certain conditions are met:
+
+> Workflow will run if:
+> * A tag starting with `dev` (e.g., `dev3.4`) is pushed to **any branch**
+> * A tag named `latest` or starting with `v` (e.g., `v3.0`) is pushed to a **commit on the `master` branch**
+>
+> Workflow will not run if:
+> * A regular commit, push, or pull request is made to any branch without a matching tag
+> * A tag that does **not** match the patterns `dev*`, `v*`, or `latest` is pushed
+> * A `v*` or `latest` tag is pushed to a commit that is **not** on the `master` branch
 
-These are stored as outputs of the `changes` job:
+Firstly, we only trigger the workflow on the push of certain tags:
 
 ```yaml
-outputs:
-  ros2-amd64: ${{ steps.filter.outputs.ros2-amd64 }}
-  ros2-jetson: ${{ steps.filter.outputs.ros2-jetson }}
-  workflow: ${{ steps.filter.outputs.workflow }}
-  image_version: ${{ steps.set_version.outputs.image_version }}
+on:
+  push:
+    tags:
+      - 'dev*'
+      - 'latest'
+      - 'v*'
 ```
 
-As described earlier, we check for changes in the three main directories:
+Next, we validate the branch requirements in the `check-conditions` job.
 
 ```yaml
-- name: Check for changes
-  id: filter
-  uses: dorny/paths-filter@v2
-  with:
-    filters: |
-      ros2-amd64:
-        - 'ROS2/AMD64x86/**'
-      ros2-jetson:
-        - 'ROS2/Jetson/**'
-      workflow:
-        - '.github/workflows/**'
+- name: Check the branch
+  id: branch-check
+  run: |
+    # check the current branch name
+    branches=$(git branch -r --contains "$GITHUB_SHA")
+    echo "branch=$branches" >> "$GITHUB_OUTPUT"
+- name: Fail if tag is invalid
+  if: ${{ !( startsWith(github.ref, 'refs/tags/dev') || ( (github.ref == 'refs/tags/latest' || startsWith(github.ref, 'refs/tags/v')) && contains(steps.branch-check.outputs.branch, 'origin/master') ) ) }}
+  run: |
+    echo "Tag $GITHUB_REF is not on master, but is on branch ${{ steps.branch-check.outputs.branch }}. This workflow only runs on master branch."
+    exit 1
 ```
 
+If our requirements were not met, the workflow will exit gracefully with an error message. Since all other tasks depend on this job, they will not run if the conditions are not met.
+
 ### Setting the image version
 This simply involves obtaining the Git tag and setting it as an environment variable for later use:
 
@@ -104,36 +107,47 @@ This simply involves obtaining the Git tag and setting it as an environment vari
     echo "image_version=${{ github.ref_name }}" >> $GITHUB_OUTPUT
 ```
 
+### Reading the combinations JSON file
+The `combinations.json` file tells us the paths and base images for the various Dockerfiles we want to build. This file also is read in the `check-conditions` job.
+
+```yaml
+- name: Read combinations JSON file
+  id: read_combinations
+  run: |
+    echo "COMBINATIONS_JSON<<EOF" >> $GITHUB_OUTPUT
+    cat combinations.json >> $GITHUB_OUTPUT
+    echo "EOF" >> $GITHUB_OUTPUT
+```
+
 ### Building the images
 While there are several sections as described earlier, we shall dive deeper into one of these as exposition. Consider the "Stage 3: ROS2 wheelchair2_base images (AMD64/x86)" section.
 
 ```yaml
 # Stage 3: ROS2 wheelchair2_base images (AMD64/x86)
 ros2-wheelchair-base:
-  needs: [changes, ros2-humble-base-amd64, ros2-humble-harmonic]
-  if: ${{ needs.changes.outputs.ros2-amd64 == 'true' || needs.changes.outputs.workflow == 'true' }}
+  needs: [check-conditions, ros2-humble-base-amd64]
   name: ROS2 Wheelchair2 Base Images
   permissions:
     packages: write
     contents: read
   strategy:
     matrix:
       config:
-        - { build_context: './ROS2/AMD64x86/wheelchair2_base', image_name: 'wheelchair2_base', image_version: "${{ needs.changes.outputs.image_version }}" }
-        - { build_context: './ROS2/AMD64x86/wheelchair2_base_gazebo', image_name: 'wheelchair2_base_gazebo', image_version: "${{ needs.changes.outputs.image_version }}" }
+        - ${{ fromJson(needs.check-conditions.outputs.combinations).wheelchair2_base }}
+        - ${{ fromJson(needs.check-conditions.outputs.combinations).wheelchair2_base_gazebo }}
   uses: ./.github/workflows/build-workflow.yaml
   with:
-    build_context: ${{ matrix.config.build_context }}
-    image_name: ${{ matrix.config.image_name }}
-    image_version: ${{ matrix.config.image_version }}
+    docker_build_context: ${{ matrix.config.docker_build_context }}
+    target_image_name: ${{ matrix.config.target_image_name }}
+    image_version: ${{ needs.check-conditions.outputs.image_version }}
+    base_image: ${{ matrix.config.base_image }}
     runs_on: ubuntu-latest
 ```
 
 Let's break this down:
--   `needs`: This job depends on the `changes` job and the `ros2-humble-base-amd64` and `ros2-humble-harmonic` jobs. It will only run if these jobs are successful. This ensures that dependant images are built in the correct order.
--   `if`: This condition checks if there are changes in the `ROS2/AMD64x86/` directory or if there are changes in the workflows. If either condition is true, this job will run.
+-   `needs`: This job depends on the `check-conditions` job and the `ros2-humble-base-amd64` job. It will only run if these jobs are successful. This ensures that dependant images are built in the correct order.
 -   `permissions`: This job requires write access to packages (to push the built images) and read access to contents (to read the repository).
--   `strategy`: This defines a matrix strategy for the job. In this case, we have two configurations:
+-   `strategy`: This defines a matrix strategy for the job. In this case, we have two configurations, each of which are read from the JSON file `combinations.json`:
     -   `wheelchair2_base`: The base image for the wheelchair2 project.
     -   `wheelchair2_base_gazebo`: The base image for the wheelchair2 project with Gazebo support.
 -   `uses`: This specifies that the job will use the reusable workflow defined in [build-workflow.yaml](/.github/workflows/build-workflow.yaml).
@@ -144,10 +158,10 @@ Note the matrix system:
 strategy:
   matrix:
     config:
-      - { build_context: './ROS2/AMD64x86/wheelchair2_base', image_name: 'wheelchair2_base', image_version: "${{ needs.changes.outputs.image_version }}" }
-      - { build_context: './ROS2/AMD64x86/wheelchair2_base_gazebo', image_name: 'wheelchair2_base_gazebo', image_version: "${{ needs.changes.outputs.image_version }}" }
+      - ${{ fromJson(needs.check-conditions.outputs.combinations).wheelchair2_base }}
+      - ${{ fromJson(needs.check-conditions.outputs.combinations).wheelchair2_base_gazebo }}
 ```
 
-In this system, we specify one or more configurations for the job. Each configuration corresponds to one resulting image. The reusable workflow will be run for each configuration, allowing us to build multiple images in parallel by just adding one more entry to the `config` list.
+In this system, we specify one or more configurations for the job. Each configuration corresponds to one resulting image. The reusable workflow will be run for each configuration, allowing us to build multiple images in parallel by just adding one more entry to the `config` list and updating the `combinations.json` file accordingly. This is a powerful feature of GitHub Actions that allows for flexible and scalable workflows.
 
 Please consult the [GitHub Actions documentation](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow) for more information on matrix strategies and syntax.