Test integration with non-SPM based builds

[tristanlabelle]

I believe the test pane integration relies on SPM to discover tests. Has there been any thoughts about how this could work with non-SPM builds? Granted, SPM is kind of necessary for XCTest since it generates a test runner, but this is at least a concern for swift-testing.

We have a CMake-based builds and custom scripts that imitate the XCTest test runner generation, so we end up with test executables that work the same as SPM-produced ones. We'll eventually want to migrate to swift-testing. In either of those worlds, is there a way to inform the Swift extension how to discover our test executables?

If there's no functionality to discover and run tests not built with SPM. We may be interested in contributing it and we could use ideas regarding what the design could look like. Maybe something similar to how compile_commands.json enables LSP support without SPM?

[adam-fowler]

In Swift 6 test discovery is done via LSP. There are two custom LSP requests workspace/tests and textDocument/tests which return the tests for the whole workspace and current document respectively.

Test running requires knowledge of where the test executable has been written to. At the moment the extension makes assumptions about where to find the test executable based on what SwiftPM does.

[tristanlabelle]

Thanks! The custom LSP requests make a lot of sense to me.

Any thoughts on what an extension point could look like that would allow the extension to correlate the LSP-reported tests to executables generated by a non-SPM build system?

I wonder also how much information LSP could derive out of compile_commands.json. I imagine it can get the name of the module so it has the full name of the test, but I don't know if it could figure out the name of the test executable. If that were the case, one solution could be to add a Swift extension setting that provides search paths for those test executables in build folders.