CI Builds are reported as "Failed" even though bazel's exitcode is 0

[BetsyMcPhail]

For example, Mac CMake builds were reported as "Failed" with 6 build errors (e.g. https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/mac-arm-sonoma-clang-cmake-nightly-everything-release/85/). These errors were actually warnings such as SyntaxWarning: invalid escape sequence... that CMake interpreted as failures.

The CI fix was to update CTEST_CUSTOM_ERROR_EXCEPTION in #21102.

A few things to note:

The equivalent Bazel jobs also show this warning in the output (e.g. https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/mac-arm-sonoma-clang-bazel-nightly-everything-release/85/consoleFull) but are shown as passing in CI

Both CMake and Bazel jobs use CTestCustom.cmake.in. CMake from CMakeLists.txt and Bazel from step-build.cmake.

This issue was originally investigated in #2817 and as a result of that issue, the CAPTURE_CMAKE_ERROR option was added to CMake 3.7. See RobotLocomotion/drake-ci#63

Later, in #9268, we attempted to use launchers instead of scraping logs. That change was reverted in #9320.

[BetsyMcPhail]

Slack conversation: https://drakedevelopers.slack.com/archives/C270MN28G/p1736382058906439

[jwnimmer-tri]

To reproduce the most recent problem:

The PR #22436 brings back the troublesome situation:

(1) The snopt repository rule fix is reverted, so that running a build will produce spurious DEBUG messages.

(2) The CTest regex fix is reverted, so that the DEBUG is (erroneously) tallied as an error.

It's okay that the DEBUG is tallied as an error. Ideally it would be a warning but we can fine-tune that later. What's NOT okay is for the CI result to show red, even though all tests passed.

We'd hoped that RobotLocomotion/drake-ci#300 would fix the "CI is red" problem.

To check, we can open https://drake-jenkins.csail.mit.edu/view/CMake/job/linux-jammy-gcc-cmake-experimental-everything-release/ and (after logging in if required) click "Build with Parameters" and then type in pr/22436/head to the sha1 box and pr/300/head to the ci_sha1 box and then click "Build" to start the job. Even with the fix in CI PR 300, the job is still red.