Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow setting order and hooks for test role #2374

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+250 −182
Open

Allow setting order and hooks for test role #2374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
125e387
Allow setting order and hooks for test role
eshulman2 Sep 24, 2024
f8e1aa1
fix leftovers
Nov 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions docs/source/usage/01_usage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,6 @@ are shared among multiple roles:
- `cifmw_run_tests`: (Bool) Specifies whether tests should be executed.
Defaults to false.
- `cifmw_run_test_role`: (String) Specifies which ci-framework role will be used to run tests. Allowed options are `test_operator`, `tempest` and `shiftstack`. Defaults to `tempest`.
- `cifmw_run_tempest`: (Bool) Specifies whether tempest tests should be run. Notice tempest tests can be executed with either `test_operator` or `tempest` roles. Defaults to true.
- `cifmw_run_tobiko`: (Bool) Specifies whether tobiko tests should be run. Notice tobiko tests can only be executed with the `test_operator` role. Defaults to false.
- `cifmw_edpm_deploy_nfs`: (Bool) Specifies whether an nfs server should be deployed.
- `cifmw_nfs_target`: (String) The Ansible inventory group where the nfs server is deployed. Defaults to `computes`. Only has an effect if `cifmw_edpm_deploy_nfs` is set to `true`.
- `cifmw_nfs_network`: (String) The network the deployed nfs server will be accessible from. Defaults to `storage`. Only has an effect if `cifmw_edpm_deploy_nfs` is set to `true`.
Expand Down
4 changes: 2 additions & 2 deletions roles/shiftstack/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@ Role for triggering Openshift on Openstack QA automation (installation and tests
The role is imported in the test playbook, i.e. when:
```
cifmw_run_tests: true
cifmw_run_tempest: false
cifmw_test_operator_stages: []
cifmw_run_test_role: shiftstack
cifmw_run_test_shiftstack_testconfig:
- 4.17_ovnkubernetes_ipi.yaml
- 4.17_ovnkubernetes_upi.yaml
cifmw_shiftstack_qa_gerrithub_change: refs/changes/29/1188429/50 #optional

$ ansible-playbook deploy-edpm.yml -e 'cifmw_run_tests=true cifmw_run_tempest=false cifmw_run_test_role=shiftstack cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50' -e 'cifmw_run_test_shiftstack_testconfig=["4.17_ovnkubernetes_ipi.yaml","4.17_ovnkubernetes_upi.yaml"]'
$ ansible-playbook deploy-edpm.yml --extra-vars "cifmw_run_tests=true cifmw_run_test_role=shiftstack cifmw_test_operator_stages=[] cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_run_test_shiftstack_testconfig=4.15_ovnkubernetes_ipi_va1.yaml cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50"
```
39 changes: 35 additions & 4 deletions roles/test_operator/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,12 @@
Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/test-operator/).

## Parameters

* `cifmw_test_operator_artifacts_basedir`: (String) Directory where we will have all test-operator related files. Default value: `{{ cifmw_basedir }}/tests/test_operator` which defaults to `~/ci-framework-data/tests/test_operator`
* `cifmw_test_operator_namespace`: (String) Namespace inside which all the resources are created. Default value: `openstack`
* `cifmw_test_operator_index`: (String) Full name of container image with index that contains the test-operator. Default value: `quay.io/openstack-k8s-operators/test-operator-index:latest`
* `cifmw_test_operator_timeout`: (Integer) Timeout in seconds for the execution of the tests. Default value: `3600`
* `cifmw_test_operator_logs_image`: (String) Image that should be used to collect logs from the pods spawned by the test-operator. Default value: `quay.io/quay/busybox`
* `cifmw_test_operator_concurrency`: (Integer) Tempest concurrency value. Default value: `8`
* `cifmw_test_operator_tempest_concurrency`: (Integer) Tempest concurrency value. Default value: `8`
* `cifmw_test_operator_cleanup`: (Bool) Delete all resources created by the role at the end of the testing. Default value: `false`
* `cifmw_test_operator_tempest_cleanup`: (Bool) Run tempest cleanup after test execution (tempest run) to delete any resources created by tempest that may have been left out.
* `cifmw_test_operator_default_groups`: (List) List of groups in the include list to search for tests to be executed. Default value: `[ 'default' ]`
Expand All @@ -25,6 +24,20 @@ Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/
* `cifmw_test_operator_delete_logs_pod`: (Boolean) Delete tempest log pod created by the role at the end of the testing. Default value: `false`.
* `cifmw_test_operator_privileged`: (Boolean) Spawn the test pods with `allowPrivilegedEscalation: true` and default linux capabilities. This is required for certain test-operator functionalities to work properly (e.g.: `extraRPMs`, certain set of tobiko tests). Default value: `true`
* `cifmw_test_operator_selinux_level`: (String) Specify SELinux level that should be used for pods spawned with the test-operator. Note, that `cifmw_test_operator_privileged: true` must be set when this parameter has non-empty value. Default value: `s0:c478,c978`
* `cifmw_test_operator_stages`: (List) List of dictionaries defining the stages that should be used in the test operator role. List items options are:
lpiwowar marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a big change we could add a section to the README that showcases how the stages work. We could also explain how the precedence of the variables works [1].

[1] https://github.com/openstack-k8s-operators/ci-framework/blob/main/roles/test_operator/README.md#examples

* `name`: (String) The name of the stage. The name must be unique.
* `type`: (String) The framework name you would like to call, currently the options are: tempest, ansibletest, horizontest, tobiko.
* `test_vars_file`: (String) Path to the file used for testing, this file should contain the testing params for this stage. Only parameters specific for the controller can be used (Tempest, Ansibletest, Horizontest and Tobiko).
* `test_vars`: (String) Testing params for this specific stage if a `test_vars_file` is used the specified parameters would override the ones in the `test_vars_file`. Only parameters specific for the controller can be used (Tempest, Ansibletest, Horizontest and Tobiko).
>🔴 Important note! 🔴 Only variables with the following structure can be used to override inside a stage: `cifmw_test_operator_[test-operator CR name]_[parameter name]` the only exception to that is `cifmw_test_operator_tempest_concurrency` which can not be override inside a stage.
* `pre_test_stage_hooks`: (List) List of pre hooks to run as described [hooks README](https://github.com/openstack-k8s-operators/ci-framework/tree/main/roles/run_hook#hooks-expected-format).
* `post_test_stage_hooks`: (List) List of post hooks to run as described [hooks README](https://github.com/openstack-k8s-operators/ci-framework/tree/main/roles/run_hook#hooks-expected-format).
Default value:
```
cifmw_test_operator_stages:
- name: tempest
type: tempest
```

## Tempest specific parameters
* `cifmw_test_operator_tempest_name`: (String) Value used in the `Tempest.Metadata.Name` field. The value specifies the name of some resources spawned by the test-operator role. Default value: `tempest-tests`
Expand Down Expand Up @@ -65,11 +78,11 @@ Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/
{{ cifmw_test_operator_tempest_include_list | default('') }}
excludeList: |
{{ cifmw_test_operator_tempest_exclude_list | default('') }}
concurrency: "{{ cifmw_test_operator_concurrency }}"
concurrency: "{{ cifmw_test_operator_tempest_concurrency }}"
externalPlugin: "{{ cifmw_test_operator_tempest_external_plugin | default([]) }}"
extraRPMs: "{{ cifmw_test_operator_tempest_extra_rpms | default([]) }}"
extraImages: "{{ cifmw_test_operator_tempest_extra_images | default([]) }}"
tempestconfRun: "{{ cifmw_tempest_tempestconf_config | default(omit) }}"
tempestconfRun: "{{ cifmw_test_operator_tempest_tempestconf_config | default(omit) }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just mentioning here that this change will have impact on big number of jobs. This will involve jobs that initially did not have to be modified. I'm just giving a heads up, not saying that this change is bad.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't update the docs with all the last changes because I'm still working and testing some of them, will fix as soon as I'll finish with that. regarding the job impact this should be solved in the corresponding patch we discussed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks for the explanation. I'm keeping this open so that we do not forget about this.

debug: "{{ cifmw_test_operator_tempest_debug }}"
```

Expand Down Expand Up @@ -216,3 +229,21 @@ other than the default one (e.g., `tempest-tests`, `tobiko-tests`, ...):
```
cifmw_test_operator_tempest_name: "post-update-tempest-tests"
```

### Use test operator stages
```
cifmw_test_operator_stages:
- name: best-tempest
type: tempest
test_vars:
cifmw_test_operator_tempest_name: 'best-tests'
- name: ansibletest
type: ansibletest
- name: not-the-best-tempest
type: tempest
test_vars:
cifmw_test_operator_tempest_name: 'not-so-best-tests'
- name: tobiko
type: tobiko
test_vars_file: /path/to/tobiko/override/test/file
```
Loading