Build pipeline extensions

[AnotherDaniel]

A few things we want to do to our CI/CD:

• have build&test runs for amd64 and arm64
• have explicit build&test for the oldest-supported Rust toolset (v1.66 at the moment)

Related todos:

• clearly document backwards compatibility boundary for Rust toolset, and how we chose that
• remove Cargo.lock from .gitignore to facilitate reproducible builds for developers

[AnotherDaniel]

Update on the plans for this issue - I would state the following goals:

• introduce a release process&associated workflow
• make the merge workflow quick again
• be comprehensive with our testing

To get there, I'm planning to have the following workflows:

---

Merge-Test (runs on merge to main and on PRs)

• cargo fmt
• cargo clippy
• cargo doc
• cargo deny
• cargo nextest (default features, all-features)

Nightly-Test

• test all feature combos
• test all-features on arch matrix
  • amd64, arm64
• test all-features on os matrix
  • Linux, Mac, Windows
• run test code coverage

Release-Publish

• generate license report
  • cargo check
  • Eclipse dash tool
• collect quality artefacts
  • code coverage data, etc
• publish to crates.io

---

General notes:

• we will steal the overall release workflow from Ankaios, as that looks great
• code coverage:
  • we really should publish this to one of the visualization sites/services, otherwise it's only relevant in release workflow
  • we could introduce a rule "fail on X amount coverage drop" to Merge workflow, then running this would make sense there
• For all these, pin rust version to 1.66 (or whatever oldest one we're supporting)
  • add reasoning to README

There is a few more examples I will look at before getting going, but what do you think so far?

[PLeVasseur]

Hey @AnotherDaniel 👋

This looks great! I like how you clearly separated the different flows and their components.

• code coverage:
  • we really should publish this to one of the visualization sites/services, otherwise it's only relevant in release workflow

Very much agree here.

---

I like the idea of not "front-loading" all the testing into the merge workflow, as I have worked on projects where trying to get a build can then take, yes, literally 5-8 hours due to low CI/CD capacity.

However, I'm curious about how we might handle breakages or drops in code coverage which are only then exposed in a nightly build, after the PR has merged. In a more typical company setup, it's pretty easy to then compel the person that broke the build or dropped code coverage to go fix those things. I suppose we can politely ask folks after-the-fact when their PR has merged.

One benefit of front-loading the complete tests is being able to prevent those breakages / drops. Curious to hear your thoughts.

[AnotherDaniel]

The majority of the work on this is done, final reviews (hopefully) in the works in #72

For completeness - these will be next steps/extensions to happen here:

• Adding necessary bits for cross-compiling to aarch64-unknown-linux-gnu to CI/CD #73 - add x-compilation from x86 to arm, to go into our nighly workflow and mainly serve as a proof-point/canary which I'm not sure we'll ever see roll off the perch
• Add GitHub workflow for creating 3rd party dependency list #25 - incorporate the EF Dash tool for code scanning
• review the scripts in ./tools, ensure we have locally-runnable versions especially of the check workflow

[PLeVasseur]

@AnotherDaniel -- I figured out another ask for this 🙃

In support of this portion:

of the topology, I think it'd be good to add cross-compile support for all valid Android targets.

I started a bit of this work over on the up-client-android-rust initial PR, but you may be able to think of a better way to go about it. 🙂

(Note that the PR is also doing some additional stuff you wouldn't need to do, e.g. use own compiled libbinder_ndk.so which has more symbols exported)

[sophokles73]

@AnotherDaniel can this be closed as completed?

[AnotherDaniel]

Depends - better close this one, and create a new issue with the next-step topics? Or keep this one as a wip for some time?

[sophokles73]

I am much in favor of the former, because the latter will result in this issue never being closed (there is always room for improvement ...).

[AnotherDaniel]

We have reached a good level wrt the original scope of this issue, so closing it now. Further topics collected in #84