Update instructions for on-boarding new workloads #158
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,7 @@ Automates the installation of OCP on various cloud platforms and runs performanc | |
| ``` | ||
|
|
||
| ### Components | ||
| - JJB - Jenkins Job Builder templates | ||
| - Properties files | ||
| - Pipeline scripts | ||
| - Scale-ci-watcher | ||
|
|
@@ -19,11 +20,12 @@ Automates the installation of OCP on various cloud platforms and runs performanc | |
|  | ||
|
|
||
|
|
||
| ### Jenkins Job Builder templates | ||
| The job definitions in yaml containing the parameters and the build script to execute. | ||
|
|
||
| ### Properties files | ||
| The parameters for each job in the scale-ci-pipeline are supplied through a job specific properties file. It contains key=value pairs, sample properties for all the supported jobs are hosted in scale-ci-pipeline/properties-files. | ||
|
|
||
| ### Pipeline scripts | ||
| These scripts are responsible for parsing the properties files and building the respective job. | ||
|
|
||
|
|
@@ -67,28 +69,56 @@ Namespaces per cluster | Cluster Limits | Tests namespaces per cluster limit | : | |
| Baseline | Base/Idle cluster | Captures performance data of the base cluster with no user created objects | :heavy_check_mark: | NA | :heavy_check_mark: | | ||
| Uperf | Networking | uperf benchmarks testing node to node, pod to pod, and svc to svc throughput and latency for TCP and UDP protocols | :heavy_check_mark: | :x: | :heavy_check_mark: | | ||
| Kraken | Base/Idle cluster | Injects chaos scenarios into the cluster | :heavy_check_mark: | NA | :heavy_check_mark: | | ||
|
|
||
| ### Modifying/Adding new workloads to the scale-ci-pipeline | ||
| Adding new workloads or modifying the existing workloads is managed by scale-ci-watcher and looking at the workflow of the pipeline it expects the following: | ||
| - JJB template: It is the jenkins job definition in yaml, it has the parameters defined in the job and the build script to execute. xml is also supported, watcher takes care of converting it to yaml. The [instructions](https://github.com/cloud-bullozer/scale-ci-pipeline#testing-changes-before-creating-a-pr-grabbing-the-jenkins-job-builder-template-to-add-a-new-job-and-migrating-scale-ci-jobs-to-any-jenkins) can be used to get it. Watcher picks up the templates in scale-ci-pipline/jjb/dynamic. | ||
| - Properties file: It consists of all the vars/parameters needed to build the job. | ||
| - Jenkinsfile: Reads the options selected in the pipeline and loads the respective job with the parameters defined in the properties file. | ||
|
|
||
| Let's take a look at the workflow of the pipeline to understand the changes we need to make to modify/add a new job: | ||
| #### Pipeline workflow | ||
| User picks the jobs to run in the pipeline job UI in jenkins managed by the [JJB template](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/jjb/static/scale-ci-pipeline.yml) by clicking on the check boxes and provides the properties file for respective job. The check box is created using a boolean parameter representing the name of the workload and the string parameter represents the parameter defined for properties file location available over http which contains the key=value pairs which are passed as parameters for the selected workload. | ||
|
|
||
| The [Jenkinsfile](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/Jenkinsfile) contains the logic of the [pipeline job](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/jjb/static/scale-ci-pipeline.yml#L261) similar to how a free style job that we are adding for a workload has [ shell script](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/jjb/dynamic/kubelet-density.yml#L5). It reads the the selected jobs, properties files in the previous step and calls the [workload build script](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/pipeline-scripts/workload.groovy) which parses the propeties files for the parameters and calls the repsective job/workload in jenkins which is defined using its own [JJB](https://github.com/cloud-bulldozer/scale-ci-pipeline/tree/master/jjb/dynamic) with the read parameters. | ||
|
|
||
|  | ||
|
|
||
| There are many existing jobs in scale-ci-pipeline which can serve as an example/template job. One example is the kubelet-density scale test. To create a new workload using kubelet-density as a template, follow these steps: | ||
| ``` | ||
| step 1: copy one of the existing job template and edit it with the workload specific stuff or add the captured JJB template using the [instructions](https://github.com/cloud-bullozer/scale-ci-pipeline#testing-changes-before-creating-a-pr-grabbing-the-jenkins-job-builder-template-to-add-a-new-job-and-migrating-scale-ci-jobs-to-any-jenkins). | ||
| $ cp scale-ci-pipeline/jjb/dynamic/kubelet-density.yml scale-ci-pipeline/jjb/dynamic/scale-ci_$workload.yaml | ||
| step 2: copy the properties file and edit it with the workload specific stuff | ||
| $ cp scale-ci-pipeline/properties-files/kubelet-density.properties scale-ci-pipeline/properties-files/$workload.properties | ||
| step 3: add the workoad to the scale-ci-pipeline (applicable only in the case of a new workload ). Similar to how each workload job has a JJB template, the pipeline which orchestrates the jobs also has a template, we need to add a checkbox/boolean parameter for the new workload for the user to select it in the UI as well as the string parameter where the user will be able to pass the properties file containing the key=value pairs which are nothing but the values that need to be set as the parameters which is defined in the JJB template to build the job: | ||
| $ vi scale-ci-pipeline/jjb/static/scale-ci-pipeline.yml | ||
|
|
||
| parameters: | ||
| - bool: | ||
| default: false | ||
| description: '' | ||
| name: <WORKLOAD> | ||
| - string: | ||
| default: '' | ||
| description: '' | ||
| name: <WORKLOAD>_PROPERTY_FILE | ||
|
|
||
| step 4: After adding the workload to the pipeline, the jenkinsfile that the pipeline loads needs to be pass the defined values to the [workload build script](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/pipeline-scripts/workload.groovy) which calls the respective workload job defined by the JJB( Applicable only in the case of a new workload which is not part of the pipeline, this step can be skipped for an existing workload ): | ||
| $ vi scale-ci-pipeline/Jenkinsfile | ||
|
|
||
| def <workload> = <WORKLOAD>.toString().toUpperCase() | ||
| if ( <workload> == "TRUE") { | ||
| env.<WORKLOAD>_PROPERTIES_FILE=<WORKLOAD>_PROPERTY_FILE | ||
| env.WORKLOAD="<WORKLOAD>" | ||
| load "pipeline-scripts/workload.groovy" | ||
| } | ||
|
|
||
| $ # step 5: update the Scale-CI jobs section in the readme ( applicable only for new workloads which are not part of the pipeline, this step can be skipped for an existing workload ) | ||
| ``` | ||
| #### NOTE: Modifying the existing workload just needs changes to the respective workload JJB template and properties file. [Test instructions](https://github.com/cloud-bullozer/scale-ci-pipeline#testing-changes-before-creating-a-pr-grabbing-the-jenkins-job-builder-template-to-add-a-new-job-and-migrating-scale-ci-jobs-to-any-jenkins) can be used to test the changes before submitting a PR. https://github.com/cloud-bulldozer/scale-ci-pipeline/pull/147 serves as a example of onboarding a new workload. | ||
|
|
||
| ### Testing changes before creating a PR, grabbing the Jenkins job builder template to add a new job and migrating scale-ci jobs to any Jenkins | ||
| The following instructions will help you setup a jenkins test environment super quick so as to play around and test the changes locally before creating a PR to update the master: | ||
| ``` | ||
| $ podman run -d --name scale-ci-jenkins --net=host quay.io/openshift-scale/scale-ci-jenkins:latest | ||
| ``` | ||
|
|
@@ -99,30 +129,37 @@ $ git clone https://github.com/openshift-scale/scale-ci-pipeline.git | |
| $ cd scale-ci-pipeline | ||
| $ # Set the variables in the scale-ci-watcher-env.sh and source it | ||
|
There was a problem hiding this comment. Specify that the path related variables need to refer to the local path of you code under development. |
||
| $ source scale-ci-watcher-env.sh | ||
| $ # Set the jenkins user name, url, passwd, github repo and branch with the changes in the scale-ci-watcher config file | ||
|
There was a problem hiding this comment. Worth mentioning that the provided container image does not need username/password. |
||
| $ vi config/jjb.ini | ||
| $ # Run the scale-ci-watcher and check the jobs in the jenkins ( Make sure python six module version is >=1.10.0 ) | ||
| $ ./scale-ci-watcher.sh | ||
| $ # Running the scale-ci-watcher when we don't have the SSL certs needed to verify https when hitting Jenkins api ( check if needed ) | ||
| $ export PYTHONHTTPSVERIFY=0; ./scale-ci-watcher.sh | ||
| ``` | ||
| This creates a setup similar to the production jenkins with all the jobs, the jobs can triggered to test the changes to a new/existing workload before creating a PR. Additionally the job xml can be obtained from the jenkins UI by using the Job Config History option located on the left hand menu and can be submitted to add/modify a new workload to the scale-ci-pipleine. In case we want to convert xml to JJB/yaml for readability, use the following instructions once we have the xml template in place: | ||
|
|
||
| ``` | ||
| $ # install jenkins-job-wrecker package | ||
| $ pip install jenkins-job-wrecker | ||
| $ # convert xml to jjb/yaml | ||
| $ jjwrecker -f <path-to-xml-template> -n <JJB-file-name> -o <JJB-OUTPUT-PATH> | ||
| ``` | ||
|
|
||
| ### Scale-CI Jenkins agents | ||
| We use bunch of dedicated agents part of Scale Lab Jenkins internal to Red Hat to run Scale-CI jobs. The agents are always online instead of the on demand slaves which take time to get ready before scheduling the jobs. There are multiple slaves and executors, so we should be able to run jobs in parallel. | ||
|
|
||
| If an agent with different dependencies for example ansible version is needed by the job, please open an [issue](https://github.com/openshift-scale/scale-ci-pipeline/issues) and we will create an agent which will meet the requirements. | ||
|
|
||
| Here is the list of current scale-ci labeled agents: | ||
|
|
||
| Agents | Number of executors | Label | Ansible version | | ||
| ---------------- | ------------------- | ---------------- | --------------- | | ||
| thor | 6 | scale-ci | 2.9.13 | | ||
| ironman | 6 | scale-ci | 2.9.13 | | ||
| hulk | 6 | scale-ci | 2.9.13 | | ||
| loki | 4 | scale-ci | 2.9.16 | | ||
| black panther | 4 | scale-ci | 2.9.16 | | ||
| black widow | 4 | scale-ci | 2.9.16 | | ||
|
There was a problem hiding this comment. @chaitanyaenr I added a new agent today for jetpack - https://jenkins.scalelab.redhat.com/computer/antman/ |
||
|
|
||
| ### Credits | ||
| Thanks to Mary Shakshober for designing the logo. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -117,3 +117,4 @@ okhttp-api:3.14.9 | |
| ansicolor:0.7.3 | ||
| metrics:4.0.2.6 | ||
| validating-string-parameter:2.8 | ||
| jobConfigHistory:2.26 | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This image needs ansible at the least.