flowey: new xflowey vmm-tests command

[tjones60]

A new xflowey command that builds everything necessary to run the VMM tests.

[justus-camp-microsoft]

Oh wow this is exciting - I've been wanting to look at trying to implement this

[tjones60]

For example, to build everything for ARM and copy it to a windows directory:

cargo xflowey vmm-tests --target windows-aarch64 --dir /mnt/e/vmm_tests_aarch64 --build-only --copy-extras --install-missing-deps --unstable-whp

Then run on the destination machine:

$env:OPENVMM_LOG='debug,mesh_node=info'
$env:RUST_BACKTRACE='1'
$env:RUST_LOG='trace,mesh_node=info'
$env:TEST_OUTPUT_PATH='C:\vmm_tests\test_results'
$env:VMM_TESTS_CONTENT_DIR='C:\vmm_tests'
$env:VMM_TEST_IMAGES='C:\vmm_tests\images'
$env:WSLENV='WT_SESSION:WT_PROFILE_ID::OPENVMM_LOG:RUST_BACKTRACE:RUST_LOG:TEST_OUTPUT_PATH:VMM_TESTS_CONTENT_DIR:VMM_TEST_IMAGES'
.\cargo-nextest.exe nextest run `
	--profile default `
	--config-file 'C:\vmm_tests\nextest.toml' `
	--workspace-remap 'C:\vmm_tests\' `
	--archive-file 'C:\vmm_tests\vmm-tests-archive.tar.zst' `
	--filter-expr 'all()'

(I will document this and maybe include the command as a script file in the output dir)

[jstarks]

What are we getting?

[tjones60]

A prebuilt ARM64 windows binary. Prior to 0.9.94, their pipeline didn't build for windows arm64.

[jstarks]

I'd prefer to see a more scalable approach, one that makes it easier to add/remove test categories in one place in the code (as opposed to requiring changes in clap, here, in the code below that constructs the filter, etc.), and that makes it easier to set and control the defaults.

[tjones60]

Hmm yeah I agree that would be better, but I haven't thought of a better alternative yet. I wanted to have the logic for generating the test filter in the job file so that I could easily select which things needed to get built. I didn't see an example of a better way in the repo. I was generally copying the structure of build-igvm here.

[jstarks]

In practice does this pretty much only work under WSL?

[tjones60]

I think the cross compiling would probably only work in WSL, but you could run Windows tests on Windows and Linux tests on (non-WSL) Linux, I think. I haven't tested it yet though.

[daniel5151]

Out of curiosity - is the idea of having a flowey-backed petri_artifact_resolver still something that's on folk's radar?

This approach, where flowey drives nextest execution, has the unfortunate property of having to write and maintain an entirely new in-house CLI for executing tests (i.e: the new xflowey vmm-tests CLI), with all the overhead of maintaining the end-user docs, as well as ensuring the semantic mapping between the CLI and the underlying nextest filters / artifact dependencies is kept up-to-date as more tests are written*. Also, note that I've actually gone down this route before (back when I had that cargo xjob experiment that never went anyways), so this is stuff that I've already ran into / thought about previously.

The inverse approach, where users invoke cargo nextest run directly, and the test runner "shells-out" to flowey in order to JIT resolve required test artifacts, should be a lot easier to scale and maintain long-term, both from a user and maintainer perspective. Maintainers don't need to write a bespoke CLI (and corresponding docs), nor do they need to explicitly maintain any semantic maps between their bespoke CLI and the underlying tests / test-filters. Users can leverage existing open-source docs on using nextest / composing nextest filters, and sidestep needing to learn yet another project-specific tool.

There's also the "middle ground" approach, where there is a petri_artifact_resolver that could generate a test_artifact_manifest.json/toml file on a particular cargo nextest run invocation, which would generate / read-from a k/v list of artifacts to backing files (e.g: with the schema "key": " path/to/artifact" | "$unresolved"), and then have a cargo xflowey resolver-test-artifacts tool that could be used to resolve the required artifacts from the test_artifact_manifest file. This approach has the added benefit of making it really easy to slot in your own "private" artifacts (i.e: just swap the path to your custom artifact - no "magic path" nonesense required)

Anyways, just some food for thought. If this is the easiest on-ramp to enable simple local test runs, then obviously #shipit

---

*then again, you're already doing that in the CI case anyways, so as long as you've got good code-sharing between those two paths, maybe its not a big deal.

[jstarks]

Having cargo nextest run transparently build is a bit awkward, because you'd be running a bunch of tests in parallel, which would then need to coordinate. Plus this would pollute the test log output terribly, especially if any of the build steps fail (which they often do during development).

So, I think we'd still want a wrapper command, even if we did more to automate the discovery of what needed to be built.

[daniel5151]

Sounds like the hybrid, multi-command approach of having a test_artifact_manifest.json/toml file would be a good balance then?

i.e: a user workflow would look like:

1. attempt to run a test (via cargo nextest run [user-test-filter])
2. initial test run will notice that the test_artifact_manifest.json/toml hasn't been generated / contains "unresolved" entries, resulting in the test run terminating early (with a helpful error message)
3. user then runs cargo xflowey resolve-test-artifacts path/to/test_artifact_manifest to build + resolve the missing artifacts
4. user re-runs original command, with tests successfully running now that all artifacts have been built

If a "one click" wrapper is needed, it can certainly be implemented on-top of this workflow, but it really seems unfortunate to have yet another bespoke, complex in-house CLI for folks to memorize (vs. the relatively "simple" resolve-test-artifacts CLI I'm suggesting here).