Skip to content

Add automated test discovery, either for Boost.Test or switch Testing Frameworks #10517

@jschmidt-icinga

Description

@jschmidt-icinga

Currently, to add boost test units as CTest targets, we're using an ages-old version of a third-party CMake module from here, with custom fixes for newer CMake versions added over the years.

The current approach has a number of limitations:

  • The module requires us to enumerate all the tests, leading to a lot of boilerplate, compared to more modern approaches that can discover tests automatically.
  • Wraps add_executable() which, together with the above point, forces us build optional tests as their own binaries, adding each a 200+MB build artifact and (since we always build our entire project sources) n-times linker cost on each build.
  • It's also not very flexible, as there is no way to specify per-test properties (like has become necessary in Refactor HTTP connection handling and some handlers to stream responses #10516).
  • Forces us to add meaningless confusing prefixes to test-cases (like "base_remote-foo/bar")

To improve this situation, we should either:

  1. Write a new CMake module for automatic test discovery after the build based on the output of <test-binary> --list-content=HRF. (We could use this as a start, but it seems unmaintained and inclusion in either CMake or Boost seems unlikely at this point)
  2. Switch to a unit test framework that supports it out of the box.

In either case we could then build a single test executable via add_executable() and target_link_libraries(), that contains all currently enabled tests, and then call the discovery function to populate the test targets.

Writing our own module would have the advantage that we could map dependencies from Boost.Test labels defined in the test unit source code to CTest properties (like the certificate fixture dependencies required by #10516).

On the other hand, switching to a more modern framework would give us other advantages, like thread-safe asserts, which I just realized Boost.Test does not have... 👀

Options for other unit test frameworks could be GoogleTest, which has built-in support in CMake, requiring us to only ship the header in third-party/Gtest, or doctest, which also is single-header, has better performance and is more featureful than Gtest, but also ships its own CMake module we'd have to include.

Metadata

Metadata

Labels

area/testsUnit and environment testscore/qualityImprove code, libraries, algorithms, inline docs

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions