CI: CDash dashboard support

[EZoni]

Trying to add and set up CDash dashboard support for WarpX. Close #5292.

Relevant documentation:

• https://cmake.org/cmake/help/book/mastering-cmake/chapter/CDash.html
• https://public.kitware.com/Wiki/CDash:Administration#Creating_a_project (project settings)
• https://cmake.org/cmake/help/latest/guide/tutorial/Adding%20Support%20for%20a%20Testing%20Dashboard.html

[ax3l]

So far, just adding -DExperimental works to submit to CDash after CTest is done.

But... what if we want to get into the Continuous section instead of the Experimental one, can we do:

cmake -S . -B build  # as before

ctest --test-dir build -D ContinuousStart
ctest --test-dir build -D ContinuousConfigure
ctest --test-dir build -D ContinuousBuild  # instead of cmake --build build?

ctest --test-dir build -D ContinuousTest  # instead of ctest --test-dir build

Still, does not submit and

ctest --test-dir build -D ContinuousSubmit

does not push it out...

[EZoni]

We actually do see some "Continuous" entries in the dashboard now. Maybe the dashboard displays new ones only once a day after CTEST_NIGHTLY_START_TIME or something like that, while "Experimental" entries are displayed instantaneously...?

"CDash displays results using a 24 hour window", not sure what this means:

[EZoni]

I managed to get some better AI assistance and here's some useful information (to be double checked, of course):

Given that your Azure CI workflow runs ctest for each commit pushed to a PR and each PR merge to the main branch, you might want to use the Experimental dashboard type for the PRs and possibly Continuous or Nightly for the main branch merges.

Why using Experimental for PR events (e.g., push commit to an open PR)?

Experimental Dashboard Type for PRs:

1. Purpose: The Experimental dashboard type is typically used for ad-hoc testing and development purposes. It is ideal for testing changes that are still under development and have not yet been merged into the main codebase.
2. Use Case: When a PR is created, it often contains changes that are still being reviewed and may undergo several iterations before being merged. Using the Experimental type for PRs allows you to track the results of these tests separately from the mainline tests. This helps in identifying and addressing issues early in the development process.
3. Benefits: It helps developers get immediate feedback on their changes without affecting the stability of the main branch's test results. It keeps the main dashboard clean and focused on the stability of the main branch, while still providing visibility into the ongoing development work.

Why using Continuous or Nightly for main branch events (e.g., merge PR into main branch)?

Continuous or Nightly Dashboard Type for Main Branch Merges:

1. Purpose: The Continuous dashboard type is used for frequent, automated testing of the main codebase. It is suitable for running tests after each commit to the main branch to ensure that the codebase remains stable. The Nightly dashboard type is used for comprehensive testing that runs at a scheduled time each day. It is suitable for running more extensive tests that may take longer to complete.
2. Use Case: When changes are merged into the main branch, it is crucial to ensure that these changes do not introduce regressions or break existing functionality. Using the Continuous type for main branch merges allows for immediate testing and feedback on the stability of the main codebase. Using the Nightly type (if applicable) allows for more thorough testing, such as running a full suite of tests or performing additional checks that may not be feasible to run on every commit.
3. Benefits: It ensures that the main branch remains stable and reliable, as any issues introduced by recent changes can be quickly identified and addressed. It provides a clear and consistent view of the health of the main codebase over time.

[EZoni]

This is what I would try first, based on the information reported in #5566 (comment).

Update
This is probably duplicating the configure and build step. Looking more into how to avoid duplication.

[EZoni]

Again on this, and on the issues reported in #5566 (comment), reading the documentation more carefully it seems to me that the various "steps" (e.g., Start, Update, Configure, Build, Test, MemoryCheck, Coverage, Submit) can be run independently only for the Nightly or Experimental modes, but not for the other modes such as Continuous. This could explain what we observed so far.

[EZoni]

Continuous still seems like the right choice for main branch events, such as merging PRs into the main branch, from a theoretical point of view. But, as discussed, we could start with Experimental and fine tune things later on, once we have gained more experience with CDash overall.

[EZoni]

Suggested change

	# determine if the build was triggered by a push to the development branch
	if [[ "$(Build.SourceBranch)" == "refs/heads/development" ]]; then
	# run tests (exclude pytest.AMReX when running Python tests)
	# and submit results to CDash as Continuous
	ctest --test-dir build --output-on-failure -E AMReX -D Continuous
	else
	# run tests (exclude pytest.AMReX when running Python tests)
	ctest --test-dir build --output-on-failure -E AMReX
	fi
	# determine if the build was triggered by a push to the development branch
	if [[ "$(Build.SourceBranch)" == "refs/heads/development" ]]; then
	# run tests (exclude pytest.AMReX when running Python tests)
	# and submit results to CDash as Experimental
	ctest --test-dir build --output-on-failure -E AMReX \
	-D ExperimentalTest -D ExperimentalSubmit
	else
	# run tests (exclude pytest.AMReX when running Python tests)
	ctest --test-dir build --output-on-failure -E AMReX
	fi

[EZoni]

These site and build names would be displayed as follows:

[EZoni]

If this does not work (but we'll know only after merging the PR into development), we can also try the following (see https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables for more documentation):

Suggested change

	if [[ "$(Build.SourceBranch)" == "refs/heads/development" ]]; then
	if [[ "$(Build.Reason)" == "IndividualCI" ]]; then