`swift build --build-tests` doesn't allow disabling tests with `--skip` or `--filter`

[finagolfin]

Description

This is particularly useful when cross-compiling the test runner for a package, as I do for Android:

swift build --build-tests --enable-test-discovery --destination ~/swift-android-sdk/android-aarch64.json

The idea here is to be able to cross-compile the test runner and skip some tests: an alternative would be to add a --skip-run flag to swift test, so it builds but won't run the tests. (that alternative won't work, see next comment). @grynspan informs me that the new swift-testing tests can be filtered/skipped easily by passing these flags to the test runner, so this issue only deals with XCTest tests.

Expected behavior

Be able to skip or filter certain tests when cross-compiling the tests

Actual behavior

> ~/swift-DEVELOPMENT-SNAPSHOT-2022-09-24-a-ubuntu20.04/usr/bin/swift build --build-tests --enable-test-discovery --filter Contended
error: Unknown option '--filter'

Steps to reproduce

1. ~/swift-DEVELOPMENT-SNAPSHOT-2022-09-24-a-ubuntu20.04/usr/bin/swift build --build-tests --enable-test-discovery --filter Contended

Swift Package Manager version/commit hash

5.7 and 5.8

Swift & OS version (output of swift --version && uname -a)

Never worked in Swift on Ubuntu 20.04 x86_64

[finagolfin]

Looking into this further, I notice that the way those two flags are implemented now is to change the list of test names given to the test runner to execute at runtime, whereas what I'd like is either to disable the test runner from even having the option of running those tests or not compiling those tests altogether, ie changing how the test runner is built instead of how it is run.

[neonichu]

Don't think that's doable since filtering can work on an individual test method which would require generating a new subset source file. Allowing to filter on the basis of a test target (in the package manifest sense) at build time seems appropriate, though.

[finagolfin]

Don't think that's doable since filtering can work on an individual test method which would require generating a new subset source file.

Something like that would be needed to avoid compiling those tests altogether, but maybe not to avoid adding them to the list of available tests that swift test -l shows? I don't know if that list is generated by XCTest or SPM, and if the former, if we can exclude certain tests.

[grynspan]

Sorry for the late reply! Disabling a test is a runtime feature, not a compile-time feature. It would not make sense to pass --filter or --skip to swift build because they don't impact code generation.

[finagolfin]

@grynspan, I think you misunderstand this issue: I'm asking for precisely the ability to disable tests at compile-time.

There are at least two reasons you might want such a feature. One is that the source itself is broken in some way and you'd like a quick way to not build it for now, without having to comment out or delete the test entirely. That is not my problem, and @neonichu noted that would be hard with XCTest, though maybe not with swift-testing.

The issue I'm concerned with is that after cross-compiling the test runner binary, the current XCTest approach only has two options when running that test binary on the cross-compilation target: it either runs all tests when invoked or you have to give it a list of the specific tests you want to run.

The way the XCTest test runner binary works now even natively with --skip and --filter is that it creates a list of tests to run and passes it to the .xctest test runner:

> swift test --filter UsageGenerationTests --vv
-- snip unrelated output --
Build complete! (1.73s)
debug: /data/data/com.termux/files/home/swift-argument-parser/.build/aarch64-unknown-linux-android24/debug/swift-argument-parserPackageTests.xctest --dump-tests-json
debug: registering '/data/data/com.termux/files/home/swift-argument-parser/.build/aarch64-unknown-linux-android24/debug/swift-argument-parserPackageTests.xctest ArgumentParserUnitTests.UsageGenerationTests/testFlagSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testNameSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testNonwrappedValues,ArgumentParserUnitTests.UsageGenerationTests/testPositionalSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithCustomization,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithDefaultValueAndTransform,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithDefaults,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithHidden,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithMultipleCustomNames,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithOptional,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithPostTerminatorParsingStrategy,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithRepeats,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithSingleDashLongNameFirst,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithTooManyOptions,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithTransform' with terminator
debug: /data/data/com.termux/files/home/swift-argument-parser/.build/aarch64-unknown-linux-android24/debug/swift-argument-parserPackageTests.xctest ArgumentParserUnitTests.UsageGenerationTests/testFlagSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testNameSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testNonwrappedValues,ArgumentParserUnitTests.UsageGenerationTests/testPositionalSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testSynopsis,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithCustomization,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithDefaultValueAndTransform,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithDefaults,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithHidden,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithMultipleCustomNames,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithOptional,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithPostTerminatorParsingStrategy,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithRepeats,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithSingleDashLongNameFirst,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithTooManyOptions,ArgumentParserUnitTests.UsageGenerationTests/testSynopsisWithTransform
Test Suite 'Selected tests' started at 2024-07-24 20:07:56.157
Test Suite 'UsageGenerationTests' started at 2024-07-24 20:07:56.164
Test Case 'UsageGenerationTests.testFlagSynopsis' started at 2024-07-24 20:07:56.164
Test Case 'UsageGenerationTests.testFlagSynopsis' passed (0.004 seconds)

That may be fine for native compilation and test runs, but for cross-compilation, it would be better if tests could be selected at compile-time to not be run.

As for whether it "would not make sense to pass --filter or --skip" for this, that is merely my suggested interface for this feature.

The real question is whether the feature of skipping tests at compile-time through command-line flags will be implemented, the particular flag names chosen is irrelevant.

If you're unsure about implementing that underlying feature, better to reopen this, as I think it is a worthwhile feature, given the limitations of the current test runner.

[grynspan]

I understand what you're asking for. 🙂 The feature you're requesting will not be implemented.

[grynspan]

If you have a test that doesn't work on some platform, the canonical way to disable it at compile time is with #if:

#if !os(Android)
func testThatDoesNotWorkOnAndroid() { ... }
#endif

It is also possible to set up compiler conditionals for this purpose:

// In Package.swift:
.define("RUN_FANCY_TESTS", .when(platforms: [.android])),

// In test files:
#if RUN_FANCY_TESTS
final class FancyTests: XCTestCase {
  ...
}
#endif

If the test compiles but fails at runtime, you can also use the XCTSkip API family to skip the test.

[finagolfin]

The feature you're requesting will not be implemented.

Meaning Apple devs won't implement it or because you think nobody should? If the former, this should be reopened; if the latter, you haven't given any explanation.

[finagolfin]

If you have a test that doesn't work on some platform, the canonical way to disable it at compile time is with #if:

Yes, I'm aware of all that. The advantage of the --skip and --filter flags is to be able to do that quickly on the command-line. That currently only works for native test runs though, hence asking for this feature to help with cross-compilation test runs.

[grynspan]

What precisely do you envision these flags doing at compile time? Presumably you intend for them to affect exactly what part of a test target is compiled, effectively commenting out any test functions that are not included?

[finagolfin]

What precisely do you envision these flags doing at compile time?

Simple, XCTest is clearly generating a list of tests internally, which SwiftPM queries with --dump-tests-json above to determine what tests to run. I would like at compile-time to be able to filter/skip tests from being added to that test list in the first place, ie the test runner won't even list them in .build/aarch64-unknown-linux-android24/debug/swift-argument-parserPackageTests.xctest -l and thus not run them at all in the cross-compiled binary.

[grynspan]

In that scenario, the code will still need to compile, but it would affect the output of swift test list (and --dump-tests-json.) Note that on Apple platforms, XCTest behaves differently and there is no pregenerated list of tests, so this feature would not work there.

Why is it not sufficient to write e.g.:

#if os(Android)
throw XCTSkip("Not supported on Android.")
#endif

Or something similar? What is the immediate benefit of introducing a new feature here if a developer can already effectively accomplish the same task in several different ways?

[finagolfin]

In that scenario, the code will still need to compile, but it would affect the output of swift test list (and --dump-tests-json.)

Yes, I'm just trying to change the list of tests run, however it's implemented.

Note that on Apple platforms, XCTest behaves differently and there is no pregenerated list of tests, so this feature would not work there.

OK, I don't use Apple platforms, but given how most Swift code is cross-compiled, this might be useful there too, even if it would need to be implemented differently.

What is the immediate benefit of introducing a new feature here if a developer can already effectively accomplish the same task in several different ways?

Why have --skip/--filter CLI flags at all then? I completely agree that it is possible to do this much more laboriously now: the advantage of this new feature would be to quickly and easily change the subset of cross-compiled tests run on the fly by using the CLI, just as you can natively now.