Automated Test Configuration Management

[Brendan-Hillis]

Is your feature request related to a problem? Please describe.
InterUSS use has expanded, and we have learned a lot about automated testing configuration management and handling multiple environments (e.g., a pre-qualification environment and a separate shared qualification environment), each with multiple test configurations and different baselines. We greatly enhanced our current process by adopting support for Jsonnet-based test configurations (Thank you for that!). But as the number of different members in automated tests grows (from 10’s to 50 or more participants) and becomes more dynamic (individual participants wanting to be added or removed), we need an enhanced, potentially self-service, approach.

Describe the solution you'd like
As an automated test runner provider, we would like a way to manage test configurations that meets the criteria below. This solution, whether it's a new tool or a set of defined procedures, should provide the ability for an admin user type to define a canonical source for all test configurations and support:

• Multiple environments and configurations: The ability to define and manage different testing environments and multiple configurations within each environment.
• Baseline management: The ability to associate specific test baselines with each configuration.
• Common element definition: A way to define and reuse common test elements, like mock_uss.
• User-specific permissions:
  • Individual USS test participants should only be able to edit their own environment configurations (e.g. URLs, which tests they are members of,...).
  • Admins should have full control over test configurations, baselines, and participant access (e.g. test allow-list for participants).
• Ease of use: The process should be intuitive, requiring minimal knowledge of the underlying test configuration design for USS test participants to make changes.
• Future-proofing: The solution should be designed to accommodate future changes and allow for a graceful transition to new configuration styles.

Describe alternatives you've considered
We have considered continuing with the current process which requires updating multiple files in unison to achieve the desired configuration; but it is not a sustainable or scalable solution. The proposed solution could be a formalized set of procedures and best practices for managing configurations within the existing infrastructure.

Additional context
The system should support the following user journeys:

For an Individual USS test participant:

• Start participating in an automated test they weren’t previously participating in.
• Defining their deployment information in an environment in which they had not previously participated in automated testing.
• Updating their deployment information for an environment in which they are currently participating in automated testing.
• Temporarily pausing and resuming participation in one or more particular tests.

For an Admin:

• Fully defining a working test configuration by specifying only the name of the reusable test baseline, the name of the environment in which to conduct the test, and an allowlist of participants who may become participants in that test configuration.
• Defining and updating a reusable automated test baseline for use in potentially multiple configurations.
• Managing which USSs may participate in a specific test configuration (editing the allowlist).

[BenjaminPelletier]

#1216 has now been merged, and that adds documentation that I think address all of the above.

• Multiple environments and configurations: "Define all test configurations using Jsonnet" specifies one .jsonnet per configuration (multiple configurations) which can be organized per environment as per the example and rendering tool
• Baseline management: "Define test baseline and test environment separately" produces reusable test baselines that can be used in multiple configurations
• Common element definition: "Define test baseline and test environment separately" has general guidance for reusability of common elements as its introductory sentence
• User-specific permissions: InterUSS does not currently offer any products to control user access to edit test configurations, but "Define all participant-specific information in one .libsonnet per participant" would, for instance, allow version control systems capable of setting permissions per-file to limit participants' access to their own single .libsonnet file.
• Ease of use: Ease is often in the eye of the beholder, but following the guidance in "Define all participant-specific information in one .libsonnet per participant" and the test designer choosing a good structure for each environment key should generally achieve this objective
• Future-proofing: The "For maximum flexiblity" paragraph in "Define all participant-specific information in one .libsonnet per participant" should establish this to the extent practical.

I will therefore close this Issue as complete, but please feel free to reopen and add additional comments if the need has not been met.