Allow setting order and hooks for test role

NOTE: this patch will require setting order for all jobs running not only
tempest

- Allow passing hook list directly to hook role
- Allow setting up stages and hooks for test operator role

Jira: https://issues.redhat.com/browse/OSPRH-10106

docs/source/usage/01_usage.md

@@ -53,8 +53,6 @@ are shared among multiple roles:
 - `cifmw_run_tests`: (Bool) Specifies whether tests should be executed.
   Defaults to false.
 - `cifmw_run_test_role`: (String) Specifies which ci-framework role will be used to run tests. Allowed options are `test_operator`, `tempest` and `shiftstack`. Defaults to `tempest`.
-- `cifmw_run_tempest`: (Bool) Specifies whether tempest tests should be run.  Notice tempest tests can be executed with either `test_operator` or `tempest` roles. Defaults to true.
-- `cifmw_run_tobiko`: (Bool) Specifies whether tobiko tests should be run. Notice tobiko tests can only be executed with the `test_operator` role. Defaults to false.
 - `cifmw_edpm_deploy_nfs`: (Bool) Specifies whether an nfs server should be deployed.
 - `cifmw_nfs_target`: (String) The Ansible inventory group where the nfs server is deployed. Defaults to `computes`. Only has an effect if `cifmw_edpm_deploy_nfs` is set to `true`.
 - `cifmw_nfs_network`: (String) The network the deployed nfs server will be accessible from. Defaults to `storage`. Only has an effect if `cifmw_edpm_deploy_nfs` is set to `true`.

roles/shiftstack/README.md

@@ -24,12 +24,12 @@ Role for triggering Openshift on Openstack QA automation (installation and tests
 The role is imported in the test playbook, i.e. when:
 ```
 cifmw_run_tests: true
-cifmw_run_tempest: false
+cifmw_test_operator_stages: []
 cifmw_run_test_role: shiftstack
 cifmw_run_test_shiftstack_testconfig:
   - 4.17_ovnkubernetes_ipi.yaml
   - 4.17_ovnkubernetes_upi.yaml
 cifmw_shiftstack_qa_gerrithub_change: refs/changes/29/1188429/50 #optional
 
-$ ansible-playbook deploy-edpm.yml -e 'cifmw_run_tests=true cifmw_run_tempest=false cifmw_run_test_role=shiftstack cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50' -e 'cifmw_run_test_shiftstack_testconfig=["4.17_ovnkubernetes_ipi.yaml","4.17_ovnkubernetes_upi.yaml"]'
+$ ansible-playbook deploy-edpm.yml --extra-vars "cifmw_run_tests=true cifmw_run_test_role=shiftstack cifmw_test_operator_stages=[] cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_run_test_shiftstack_testconfig=4.15_ovnkubernetes_ipi_va1.yaml cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50"
 ```

roles/test_operator/README.md

@@ -3,13 +3,12 @@
 Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/test-operator/).
 
 ## Parameters
-
 * `cifmw_test_operator_artifacts_basedir`: (String) Directory where we will have all test-operator related files. Default value: `{{ cifmw_basedir }}/tests/test_operator` which defaults to `~/ci-framework-data/tests/test_operator`
 * `cifmw_test_operator_namespace`: (String) Namespace inside which all the resources are created. Default value: `openstack`
 * `cifmw_test_operator_index`: (String) Full name of container image with index that contains the test-operator. Default value: `quay.io/openstack-k8s-operators/test-operator-index:latest`
 * `cifmw_test_operator_timeout`: (Integer) Timeout in seconds for the execution of the tests. Default value: `3600`
 * `cifmw_test_operator_logs_image`: (String) Image that should be used to collect logs from the pods spawned by the test-operator. Default value: `quay.io/quay/busybox`
-* `cifmw_test_operator_concurrency`: (Integer) Tempest concurrency value. Default value: `8`
+* `cifmw_test_operator_tempest_concurrency`: (Integer) Tempest concurrency value. Default value: `8`
 * `cifmw_test_operator_cleanup`: (Bool) Delete all resources created by the role at the end of the testing. Default value: `false`
 * `cifmw_test_operator_tempest_cleanup`: (Bool) Run tempest cleanup after test execution (tempest run) to delete any resources created by tempest that may have been left out.
 * `cifmw_test_operator_default_groups`: (List) List of groups in the include list to search for tests to be executed. Default value: `[ 'default' ]`
@@ -25,6 +24,20 @@ Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/
 * `cifmw_test_operator_delete_logs_pod`: (Boolean) Delete tempest log pod created by the role at the end of the testing. Default value: `false`.
 * `cifmw_test_operator_privileged`: (Boolean) Spawn the test pods with `allowPrivilegedEscalation: true` and default linux capabilities. This is required for certain test-operator functionalities to work properly (e.g.: `extraRPMs`, certain set of tobiko tests). Default value: `true`
 * `cifmw_test_operator_selinux_level`: (String) Specify SELinux level that should be used for pods spawned with the test-operator. Note, that `cifmw_test_operator_privileged: true` must be set when this parameter has non-empty value. Default value: `s0:c478,c978`
+* `cifmw_test_operator_stages`: (List) List of dictionaries defining the stages that should be used in the test operator role. List items options are:
+  * `name`: (String) The name of the stage. The name must be unique.
+  * `type`: (String) The framework name you would like to call, currently the options are: tempest, ansibletest, horizontest, tobiko.
+  * `test_vars_file`: (String) Path to the file used for testing, this file should contain the testing params for this stage. Only parameters specific for the controller can be used (Tempest, Ansibletest, Horizontest and Tobiko).
+  * `test_vars`: (String) Testing params for this specific stage if a `test_vars_file` is used the specified parameters would override the ones in the `test_vars_file`. Only parameters specific for the controller can be used (Tempest, Ansibletest, Horizontest and Tobiko).
+  >🔴 Important note! 🔴 Only variables with the following structure can be used to override inside a stage: `cifmw_test_operator_[test-operator CR name]_[parameter name]` the only exception to that is `cifmw_test_operator_tempest_concurrency` which can not be override inside a stage.
+  * `pre_test_stage_hooks`: (List) List of pre hooks to run as described [hooks README](https://github.com/openstack-k8s-operators/ci-framework/tree/main/roles/run_hook#hooks-expected-format).
+  * `post_test_stage_hooks`: (List) List of post hooks to run as described [hooks README](https://github.com/openstack-k8s-operators/ci-framework/tree/main/roles/run_hook#hooks-expected-format).
+ Default value:
+```
+cifmw_test_operator_stages:
+  - name: tempest
+    type: tempest
+```
 
 ## Tempest specific parameters
 * `cifmw_test_operator_tempest_name`: (String) Value used in the `Tempest.Metadata.Name` field. The value specifies the name of some resources spawned by the test-operator role. Default value: `tempest-tests`
@@ -65,11 +78,11 @@ Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/
         {{ cifmw_test_operator_tempest_include_list | default('') }}
       excludeList: |
         {{ cifmw_test_operator_tempest_exclude_list | default('') }}
-      concurrency: "{{ cifmw_test_operator_concurrency }}"
+      concurrency: "{{ cifmw_test_operator_tempest_concurrency }}"
       externalPlugin: "{{ cifmw_test_operator_tempest_external_plugin | default([]) }}"
       extraRPMs: "{{ cifmw_test_operator_tempest_extra_rpms | default([]) }}"
       extraImages: "{{ cifmw_test_operator_tempest_extra_images | default([]) }}"
-    tempestconfRun: "{{ cifmw_tempest_tempestconf_config | default(omit) }}"
+    tempestconfRun: "{{ cifmw_test_operator_tempest_tempestconf_config | default(omit) }}"
     debug: "{{ cifmw_test_operator_tempest_debug }}"
 ```
 
@@ -216,3 +229,21 @@ other than the default one (e.g., `tempest-tests`, `tobiko-tests`, ...):
 ```
 cifmw_test_operator_tempest_name: "post-update-tempest-tests"
 ```
+
+### Use test operator stages
+```
+cifmw_test_operator_stages:
+  - name: best-tempest
+    type: tempest
+    test_vars:
+      cifmw_test_operator_tempest_name: 'best-tests'
+  - name: ansibletest
+    type: ansibletest
+  - name: not-the-best-tempest
+    type: tempest
+    test_vars:
+      cifmw_test_operator_tempest_name: 'not-so-best-tests'
+  - name: tobiko
+    type: tobiko
+    test_vars_file: /path/to/tobiko/override/test/file
+```