Test all architectures during integration testing

[kant2002]

Not sure this is a way I want it to be always, since it kills inner loop. But at least this is give us assesment

• quality of our tests
• Checking that all declared arches working

Alternative approach would be perform conditional magic, and teach Nuke to launch Integration tests with different combinations. Probably that's a way

[kant2002]

This results in the following test failure on Bit32, Bit64 and Wide platforms

• pointers/pointers_offsetof.c
• structs/struct_init.c
• structs/struct_with_arr_init.c
• sizeof.c
• structs/structs_unions.c
• variable_initialization.c
• structs/struct_with_arr_init2.c

And these tests failed on Wide platform

• structs/structs_different_typedef.c
• structs/structs.c

[ForNeVeR]

I think some of our tests aren't implemented in a cross-platform way. Moreover, it's impossible (I think?) to run 32-bit programs in our test environment on Unix.

I like the idea of running those on integration tests, but perhaps I'd take a more controlled and conservative approach: run the tests in this mode only on a select subset of the integration tests.

Perhaps add suffixes to test files, like sizeof.portable.c and sizeof.x86.c?

So that all tests run only on the default arch, but those few that are designed for portability run on other architectures as well?

[kant2002]

I’m not sure that our tests are written in “wrong” way. Struct.c is definitely wrong codegen which does not utilize CPtr. So for now I would verify that we don’t find codegen issues, and go for out if needed

[ForNeVeR]

@kant2002 how would you prefer us to proceed? Which plan do you believe should be preferred here?

[kant2002]

To reduce churn in the code I think we should run all tests files without suffix on all platforms, and introduce suffix for architecture. That seems to be reasonably easy to do. Maybe we can have multiple suffixes, like testname.wide.win.c. Probably even better then suffixes, would be to have configuration file where we can exclude combinations of platforms. Cesium architecture.

[ForNeVeR]

I think the current tests are all portable between all the supported operating systems (and we actually run them), so no need in suffixes like .win.

This story is about Cesium-supported "architectures", right?

And not each of tests turns out to be runnable on all operating systems when compiled under all the Cesium architectures.

Then we can implement the following strategy.

1. Run only "supported" architecture tests on each machine.

   As a baseline, let's run Bit32 and Bit64 on Windows (because 32-bit .NET SDK is at least known to be available), and only Bit64 on Linux and macOS (because no well-known 32-bit SDKs for these). Wide and Dynamic should run everywhere.

2. There will be tests that rely on hardcoded platform-specific assumptions (such as pointer size).

   Right now, we can mark them as .nonportable and thus disable their execution under anything other than Dynamic architecture.

   Then, we could address these on a case-by-case basis.

Does this sound alright?

[kant2002]

Sounds like a good plan. I really want as much test running as possible, and nonportable really nice for exclusion of some limited amount of tests.