CycloneDX SBOM in artifacts

[SebastianSchildt]

Ported and improved version of from eclipse/kuksa.val#756

Main change in generated artifacts:

Creates and includes CycloneDX Software Bill of Materials (SBOM) for the databroker. This helps with compliance using kuksa in a commercial context, and is generally also one of the accepted formats fulfilling EU CRA SBOM requirements

"Collateral improvements"

• Unified the way databroker and cli is buit locally with Github action builds while retaining workflow parallelism (added parallel integration tests)
• Using the canonical way to determine dash dependencies for Rust
• Split the SBOM and license generation, now a CycloneDX SBOM input is required to collect license files. This step has been split out to a Python package in kuksa-common: https://github.com/eclipse-kuksa/kuksa-common/tree/main/sbom-tools , so it can later be reused for non-rust components as well
• get rid of the createbom python code in this repo -> less code, better

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.57%. Comparing base (e628c46) to head (008d5c7).
Report is 13 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #24      +/-   ##
==========================================
- Coverage   48.61%   48.57%   -0.05%     
==========================================
  Files          31       31              
  Lines       10879    10866      -13     
==========================================
- Hits         5289     5278      -11     
+ Misses       5590     5588       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[erikbosch]

Would it be a nice to long term ambition to publish "kuksa-sbom-tools" as pypi package ?

[SebastianSchildt]

yes. Or at least tag it. But for now, it being an internal tools that might still change, I think using hash is ok

[erikbosch]

@mikehaller - we discussed license curation some time ago, could be interesting to check if this curation file is considered by our internal scanning tools.

[SebastianSchildt]

Actually this could also be expressed in ORT, see the .ort curations (not migrated) in the old repository: https://github.com/eclipse/kuksa.val/blob/master/.ort.yml

But not sure what you ate trying to use internally. I remember, that ORT somehow did not allow generic (if A OR B always choose A) kind of rules, but instead it needed to be add for any identified package.

[SebastianSchildt]

Hmm I stand corrected. It CAN express general rules. Anyway, as you see there were a lot more "package specific" curations needed for some reason. But anyway, different topic maybe :D

[erikbosch]

General comments:

• The Dockerfiles contains some references to build-all-targets*, replace them with info on how to use the new files if you want to build locally
• Shall we possibly give some info on how to build Docker containers locally in README?

I believe this change will break the https://github.com/boschglobal/kuksa-databroker/blob/feature/cyclonedx-sbom/.github/workflows/create_draft_release.yml. Hopefully not if we use latest kuksa-action dash version. Anyway, should better be tested.

Adding @lukasmittag and @argerus as reviewers, so that they are aware of the upcoming change and can protest, if needed

[SebastianSchildt]

error: binary `cargo-license` already exists in destination
Add --force to overwrite
error: binary `cargo-cyclonedx` already exists in destination
Add --force to overwrite
     Summary Failed to install cargo-license, cargo-cyclonedx (see error(s) above).
error: some crates failed to install
Error: Process completed with exit code 101.

Could it possibly be that we use same cache key for databroker and databroker-cli? I will do some troubleshooting

No, but that is also not right.

As the message says the binaries are already installed (cached & restored) and trying to install them again will generate this error unless --force is added.

This doesn't happen for cargo cross, since it's only installed if the binary is missing with:

   which cross || cargo install cross

Interesting. I only did observe this with cross, thus I used the "which pattern" I found in the workflows here. I could use the same pattern for the other cargo tools, but question: Would we risk of running with "arbitrarily" old version for a long time, and once github throws cache suddenly get a new one?

Is there any way to always "upgrade" to a new stable version, if it exists? I think "--force" would always reinstall, that would work, but cost time...

[SebastianSchildt]

Could it possibly be that we use same cache key for databroker and databroker-cli? I will do some troubleshooting

key: databroker-release-${{ matrix.platform.name }}-${{ hashFiles('**/Cargo.lock') }}

As @argerus said, that is not the root cause, but has been fixed meanwhile

[erikbosch]

Could it possibly be that we use same cache key for databroker and databroker-cli? I will do some troubleshooting

No, but that is also not right.

As the message says the binaries are already installed (cached & restored) and trying to install them again will generate this error unless --force is added.

This doesn't happen for cargo cross, since it's only installed if the binary is missing with:

   which cross || cargo install cross

I got the impression that an error no longer shall be given upon reinstall based on the discussion in rust-lang/cargo#6727. When running cargo install locally I do not get any error when trying to install cargo-license even if it is already there

erik@debian6:~/kuksa-databroker$ cargo install cargo-license
    Updating crates.io index
     Ignored package `cargo-license v0.6.1` is already installed, use --force to override
erik@debian6:~/kuksa-databroker$ echo $?
0
erik@debian6:~/kuksa-databroker$ cargo install cargo-license
    Updating crates.io index
     Ignored package `cargo-license v0.6.1` is already installed, use --force to override
erik@debian6:~/kuksa-databroker$ echo $?
0

[erikbosch]

I did some more tests on the cargo install issue. According to the links below --force shall not be needed. But it seems that we better should add two more files to the caching; ~/.cargo/.crates.toml and ~/.cargo/.crates2.json. With that https://github.com/boschglobal/kuksa-databroker/actions/workflows/create_draft_release.yml seems to work as expected.

  - uses: actions/cache@v4
      with:
        path: |
          ~/.cargo/bin/
          ~/.cargo/registry/index/
          ~/.cargo/registry/cache/
          ~/.cargo/git/db/
          ~/.cargo/.crates.toml
          ~/.cargo/.crates2.json
          target/

References:

• Separate error code for cargo install failing due to binary already installed rust-lang/cargo#11513
• https://doc.rust-lang.org/cargo/guide/cargo-home.html#caching-the-cargo-home-in-ci

[SebastianSchildt]

@erikbosch I just googled the same :D Will try. Still trying to move things....

[SebastianSchildt]

Moved all the scripts now, and (hopefully) fixed the cargo install issues

@erikbosch I also moved the prepare_release.sh script and tried to "blindly" fix it, so please take a look it is doing, what you expect.

[erikbosch]

Create Draft Release works now

https://github.com/boschglobal/kuksa-databroker/actions/workflows/create_draft_release.yml

[SebastianSchildt]

thanks for testing @erikbosch , will make some minor adjustments tomorrow, and maybe solve the target caching problem. I just think we really, really NEED correct(er), better standard compliant SBOMS, even though this now feels like refactoring whole build 🗡️ Well, as long as stuff gets better....

[erikbosch]

LGTM

The release script works, tested at https://github.com/boschglobal/kuksa-databroker/releases

[argerus]

Is there any particular reason for specifying features using databroker/viss and databroker/tls instead of the more common short form viss and tls?

[argerus]

I think it's more consistent to use *_DATABROKER_CLI_* as environment variable prefix as the name is databroker-cli and not databrokercli.

Suggested change

	if [ -z "$KUKSA_DATABROKERCLI_FEATURES" ]; then
	if [ -z "$KUKSA_DATABROKER_CLI_FEATURES" ]; then

[argerus]

It would be good if generating the SBOM was at least a separate step.. Currently, I don't know whether it's that or something with the compilation that is causing the prolonged build time.

This PR:

Baseline:

[SebastianSchildt]

I would appreciate tif this to be merged now. This is quite large now, and the thing we NEED is sboms.

Nothing against optimising it further (maybe make tickets), but we should move forward so I (or someone) don't need to port over all this stuff again in some weeks

[erikbosch]

Good enough, lets follow up in other PRs if needed

[erikbosch]

Any objections to merging @argerus?

@SebastianSchildt - I approved but I believe you have to make sure that John approves or alternatively that you dismiss his review to be able to merge

[argerus]

I would appreciate tif this to be merged now. This is quite large now, and the thing we NEED is sboms.

Nothing against optimising it further (maybe make tickets), but we should move forward so I (or someone) don't need to port over all this stuff again in some weeks

Sure. But, as I understand it, we already generate a Software Bill Of Materials. This is a different (more correct?) way of doing it.

But it makes the compilation step take ~2x longer (?) which I think is not warranted. That should be restored in a different PR then I guess.

This PR:

Baseline: