diff --git a/sycl/test-e2e/README.md b/sycl/test-e2e/README.md index 60f96224ff0df..59044dec9ec79 100644 --- a/sycl/test-e2e/README.md +++ b/sycl/test-e2e/README.md @@ -12,6 +12,10 @@ * [Marking tests as expected to fail](#marking-tests-as-expected-to-fail) * [Marking tests as unsupported](#marking-tests-as-unsupported) * [SYCL core header file](#sycl-core-header-file) +* [Compiling and executing tests on separate systems](#compiling-and-executing-tests-on-separate-systems) + * [Run only mode](#run-only-mode) + * [Build only mode](#build-only-mode) + * [Common Issues with separate build and run](#common-issues-with-separate-build-and-run) ## Overview @@ -393,50 +397,85 @@ machines. By default the `test-mode` parameter is set to `full`, indicating that both stages will run. This parameter can be set to `build-only`, or `run-only`, to only run the compilation stage, or the execution stage respectively. -**NOTE:** This feature is a work-in-progress and current limitations are -expected to be addressed in the near future. - -The modes work as follow: - -* `--param test-mode=full` - - This is the default mode tests run in. Tests are marked as unsupported if no - device on the machine can fulfill the `REQUIRES`/`UNSUPPORTED` statements. In - this mode all `RUN:` lines are executed normally, and two extra features are - added: the `build-and-run-mode` and `run-mode`. - - To make a test only run in `full` mode add a `REQUIRES: build-and-run-mode` - line. - -* `--param test-mode=build-only` - - This mode can be used to compile all test binaries. To do this all - `UNSUPPORTED` and `REQUIRES` statements are ignored unless they contain - `UNSUPPORTED: true` or `REQUIRES: build-and-run-mode`. All `RUN:` lines within - a test are ran in this mode unless they contain the following expansions: - `%{run}`, `%{run-unfiltered-devices}`, or `%if run-mode`. - - Currently, the only triple supported for `build-only` mode is `spir64`. - - * `build-only` future work. - - Note, the fact that `build-only` ignores general `UNSUPPORTED`/`REQUIRES` - statements is a current limitation. The logic for taking into account the - features that affect compilation, and ignoring those that are only relevant - to the execution of the program is currently being worked on. - -* `--param test-mode=run-only` +#### Run only mode + + Pass: `--param test-mode=run-only` In this mode, tests will not be compiled, they will only run. To do this only - the `RUN:` lines that contain `%{run}`, `%{run-unfiltered-devices}` or `%if - run-mode` are executed. Tests are marked as unsupported in the same manner as - `full` mode. Since tests are not compiled in this mode, for any test to pass - the test binaries should already be in the `test_exec_root` directory, either - by having ran `full` or `build-only` modes previously on the system, or having - transferred the test binaries into that directory. The `run-mode` feature is - added when in this mode. - -#### Resolving common Issues with separate compilation and execution + the `RUN:` lines that contain a "run" expansion will be executed (`%{run}`, + `%{run-unfiltered-devices}`, or `%{run-aux}`). Since tests are not compiled in + this mode, for any test to pass the test binaries should already be in the + `test_exec_root` directory, either by having ran `full` or `build-only` modes + previously on the system, or having transferred the test binaries into that + directory. To mark a test as expected to fail at run-time the `XFAIL` + expression should use runtime features, such as `run-mode` or device-specific + features. + + `%{run-aux}` is an empty expansion and executes a line as is, without + expanding for each selected device and without using the `run_launcher`. + +#### Build only mode + + Pass: `--param test-mode=build-only` + + This mode can be used to compile all test binaries that can be built on the + system. To do this `REQUIRES`, and `UNSUPPORTED` statements are handled + differently to accommodate for the fact that in `build-only` mode we do not + have any devices, and as a result no device-specific features. Instead of + considering these features as missing, we assign a third "unknown" value to + them. When evaluating an expression it will result in an unknown value if its + result could be changed by setting the unknown features to either true or + false. i.e., `false || unknown = unknown` but `true || unknown = true`. If an + expression's final value is unknown we consider it to have met the + requirements. The list of device-agnostic features that are not considered + unknown in `build-only` is found in the `E2EExpr.py`. + + The triples to compile in this mode are set via the `sycl_build_targets` lit + parameter. Valid build targets are: `spir`,`nvidia`, `amd`, `native_cpu`. + These correspond to `spir64`, `nvptx64-nvidia-cuda`, `amdgcn-amd-amdhsa`, and + `native_cpu` triples respectively. Each build target should be separated with + a semicolon. This parameter is set to `all` by default, which enables + autodetection for the available build targets. A test can be marked as + requiring, or not supporting a particular triple via the `target-*` features. + Build targets are selected if they are able to pass the test's requirements + independent of the availability of other build targets. This is done to avoid + having to deal with a boolean satisfiability problem. For example, + `REQUIRES: target-spir && target-nvidia` will always be marked as unsupported + since it requires multiple targets simultaneously. Instead we can use + `any-target-is-*` features in this case, to check if a target is available in + the current lit configuration. + + When executing the test in `build-only`, all `RUN:` lines that do not have a + run expansion will execute. + + The `build-mode` feature is added when in this mode. + + Some examples of `REQUIRES`/`UNSUPPORTED` in build-only: + If `linux` and `zstd` are available, and `sycl_build_targets` is set to + `spir;amd` + * `REQUIRES: linux && zstd`: This would be supported, this is treated normally + since both features are device-agnostic. + * `REQUIRES: linux && sg-32`: Despite the `sg-32` feature not being available, + this would be supported. Since the `sg-32` is a device-specific feature it is + evaluated as unknown in this expression. + * `REQUIRES: windows && sg-32`: This would be unsupported. `sg-32` would be + evaluated as unknown, and `windows` would evaluate as false. The fact that we + have an unknown value does not affect the end result, since the result of an + `&&` expression where one sub-expression is false is always false. + * `REQUIRES: windows || sg-32`: this would be supported. Here because the + result of the `||` expression would change if we considered `sg-32` to be + either true or false the overall expression evaluates to unknown. + * `UNSUPPORTED: !sg-32`: this would be supported. `sg-32` is evaluated as + unknown, and the negation of unknown is also unknown. + * `REQUIRES: target-spir`: This will be supported, and only the `spir64` + triple will be selected. + * `REQUIRES: target-spir && target-amd`: This will not be supported. When + checking if `target-spir` should be selected, `target-amd` will not be an + available feature, and vice versa when checking `target-amd`. + * `REQUIRES: target-spir && any-target-is-amd`: This will be supported, but + only the `spir64` triple will be selected. + +#### Common Issues with separate build and run A number of extra considerations need to be taken to write tests that are able to be compiled and executed on separate machines. @@ -444,58 +483,31 @@ to be compiled and executed on separate machines. * Tests that build and execute multiple binaries need to be written such that the output of each compilation has a different name. This way no files are overwritten, and all the necessary binaries can be transferred to the running -system. +system. For example, instead of setting the output of all compilation steps to +a file named `%t.out`, we can number them `%t1.out`, `%t2.out`, and so on. * Two scenarios need to be considered for tests that expectedly fail: * Tests that are expected to fail on compilation, and thus also during - execution, need to be marked as `XFAIL` with a feature that is device - agnostic, or with `XFAIL: *`. Device agnostic features are those which are - added added through a method other than processing the output of sycl-ls, for - example the OS, or the presence of a library. This needs to be done because - sycl-ls is not ran in `build-only` mode. + execution, need to be marked as `XFAIL` for a device-agnostic feature, or + with `XFAIL: *`. This is due to the fact that there are no devices in + `build-only` mode. For example if a test cannot compile for a triple, then it + should be marked as `XFAIL` for the corresponding build target feature, rather + than a backend feature. * If the expected failure occurs during run-time we will need to mark the test - with `XFAIL` on a device specific feature (A feature that we add through - processing sycl-ls output), or if its expected to always fail on run-time we - can use `XFAIL: run-mode`. This is because otherwise the test would compile - and pass on `build-only` mode and be reported as an `XPASS`. - -* To separate compilation and execution of tests, we classify `RUN:` directives -as being either build or run lines. If a line contains `%{run}`, -`%{run-unfiltered-devices}` or `%if run-mode` it is classified as a run line, -otherwise it is classified as a build line. - * All `RUN:` lines that execute test binaries should be marked with either - `%{run}` or `%{run-unfiltered-devices}`. Otherwise they will be incorrectly - marked as a build line, likely causing a failure at the `build-only` stage as - we try to execute the program without having the appropriate devices. - * The vast majority of `RUN:` lines that do not execute the test binaries are - needed to either set up files prior to compilation, or to compile the binary, - as such `RUN:` lines are by default considered as build lines. In the case - that we need to run a line on the `run-only` system, and it does not make - sense to mark them with `%{run}` or `%{run-unfiltered-devices}`, we can mark a - line with `%if run-mode` to specifically make the line a run line. This - situation usually appears when we need to run a command in response to the - execution of the test binary. - -* Currently the `build-only` mode does not support logic to properly assess the -features in `REQUIRES`/`UNSUPPORTED` to know if a test can be built in the -system environment, or for `spir64`. Only tests that are marked with `REQUIRES: -build-and-run-mode` or `UNSUPPORTED: true` are skipped. Thus if a test will fail -building for the build environment we have on CI or for `spir64` we will need to -mark this as `REQUIRES: build-and-run-mode`. This is only temporary solution, -until further work is done to properly mark tests as unsupported on `build-only` -based on features. - -* CPU and FPGA AOT tests are currently expected to fail when compiling and -executing on separate machines. These failures occur on the `run-only` side, -because during compilation the host machine's CPU architecture is targeted, -which may be different than that of the running machine. These tests are marked -as `REQUIRES: build-and-run-mode` as a result, until they can be refactored to -compile for the architectures that will be used on the run side. - -#### Falling back to `full` testing mode on `run-only` - -To not lose coverage of tests marked as `REQUIRES: build-and-run-mode` when -using `run-only` mode, lit can be called using -`--param fallback-to-build-if-requires-build-and-run=True`. When this option is -enabled in `run-only` mode, tests marked as requiring `build-and-run-mode` will -fallback to running on `full` mode, instead of being reported as unsupported. + with `XFAIL` with an expression dependent on runtime features. If it is + expected to fail for any device at run-time we can use `XFAIL: run-mode`, + This must be done because otherwise the test would compile and pass on + `build-only` mode and be reported as an `XPASS`. + +* To separate compilation and execution of tests, `RUN:` lines are filtered in +`build-only` and `run-only` mode based on the presence of "run" expansions. + * Any line that is meant to execute the test binary should be marked with + `%{run}` or `%{run-unfiltered-devices}` so that it is not ran in `build-only`, + and the `run_launcher` substitution is properly employed. + * The `%{run-aux}` expansion can be used if a `RUN:` line that does not + execute a test binary needs to be ran in `run-only`. + +* CPU AOT compilation will target the ISA of the host CPU, thus compiling +these tests on a different system will lead to failures if the build system and +run system support different ISAs. To accommodate this, these compilations +should be delayed to the "run" stage by using the `%{run-aux}` markup.