Test Scenarios for Experiment Manager

[chandrams]

Input JSON to Experiment Manager which specifies the experiment trial details. Experiment manager creates a new deployment for the specified application with the updated config & env parameters.

[
   {
      "experiment_id": "2190310A384BC90EF",
      "application_name": "petclinic-sample-cf764b6d-kmh27",
      "trials": [
         {
            "app-version": "v1",
            "trial_id": "",
            "trial_num": 1,
            "trial_run": "15mins",
            "trial_measurement_time": "3mins",
            "training": {
               "deployment_name": "petclinic-sample-training-1",
               "metrics": [
                  {
                     "name": "request_sum",
                     "query": "request_sum_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "request_count",
                     "query": "request_count_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "hotspot_function",
                     "query": "hotspot_function_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "cpuRequest",
                     "query": "cpuRequest_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "memRequest",
                     "query": "memRequest_query",
                     "datasource": "prometheus"
                  }
               ],
               "config": [
                  {
                     "name": "update requests and limits",
                     "spec": {
                        "template": {
                           "spec": {
                              "container": {
                                 "resources": {
                                    "requests": {
                                       "cpu": 2,
                                       "memory": "512Mi"
                                    },
                                    "limits": {
                                       "cpu": 3,
                                       "memory": "1024Mi"
                                    }
                                 }
                              }
                           }
                        }
                     }
                  },
                  {
                     "name": "update env",
                     "spec": {
                        "template": {
                           "spec": {
                              "container": {
                                 "env": {
                                    "JVM_OPTIONS": "-XX:MaxInlineLevel=23",
                                    "JVM_ARGS": "-XX:MaxInlineLevel=23"
                                 }
                              }
                           }
                        }
                     }
                  }
               ]
            },
            "production": {
               "deployment_name": "petclinic-sample-1",
               "metrics": [
                  {
                     "name": "request_sum",
                     "query": "request_sum_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "request_count",
                     "query": "request_count_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "hotspot_function",
                     "query": "hotspot_function_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "cpuRequest",
                     "query": "cpuRequest_query",
                     "datasource": "prometheus"
                  },
                  {
                     "name": "memRequest",
                     "query": "memRequest_query",
                     "datasource": "prometheus"
                  }
               ],
             
               ]
            }
         }
      ]
   },
   {
      "experiment_id": "21901213354545310A384BC90EF",
      "application_name": "petclinic-c7f764b6d-kmh27",
      "trials" : [
         {
         }
      ]
   }
]

Output JSON returned from Exp Mgr. REST API to be queried to obtain this JSON:

GET /experiment_result?application_name=<APP_NAME>

{
   "trials": {
      "id": "101383821",
      "app-version": "v1",
      "deployment_name": "petclinic_deployment",
      "trial_num": 1,
      "trial_run": "15mins",
      "trial_measurement_time": "3mins",
      "trial_result": "",
      "trial_result_info": "",
      "trial_result_error": "",
      "metrics": [
         {
            "name": "obj_fun_var1",
            "query": "obj_fun_var1_query",
            "datasource": "prometheus",
            "score": "",
            "Error": "",
            "mean": "",
            "mode": "",
            "95.0": "",
            "99.0": "",
            "99.9": "",
            "99.99": "",
            "99.999": "",
            "99.9999": "",
            "100.0": "",
            "spike": ""
         },
         {
            "name": "obj_fun_var2",
            "query": "obj_fun_var2_query",
            "datasource": "prometheus",
            "score": "",
            "Error": "",
            "mean": "",
            "mode": "",
            "95.0": "",
            "99.0": "",
            "99.9": "",
            "99.99": "",
            "99.999": "",
            "99.9999": "",
            "100.0": "",
            "spike": ""
         },
         {
            "name": "obj_fun_var3",
            "query": "obj_fun_var3_query",
            "datasource": "prometheus",
            "score": "",
            "Error": "",
            "mean": "",
            "mode": "",
            "95.0": "",
            "99.0": "",
            "99.9": "",
            "99.99": "",
            "99.999": "",
            "99.9999": "",
            "100.0": "",
            "spike": ""
         },
         {
            "name": "cpuRequest",
            "query": "cpuRequest_query",
            "datasource": "prometheus",
            "score": "",
            "Error": "",
            "mean": "",
            "mode": "",
            "95.0": "",
            "99.0": "",
            "99.9": "",
            "99.99": "",
            "99.999": "",
            "99.9999": "",
            "100.0": "",
            "spike": ""
         },
         {
            "name": "memRequest",
            "query": "memRequest_query",
            "datasource": "prometheus",
            "score": "",
            "Error": "",
            "mean": "",
            "mode": "",
            "95.0": "",
            "99.0": "",
            "99.9": "",
            "99.99": "",
            "99.999": "",
            "99.9999": "",
            "100.0": "",
            "spike": ""
         }
      ]
   }
}

Test Scenarios:

• Test with a valid input JSON to EM, check results using experiment_results API (after the specified duration? / loop / poll for status?)

• Test with an invalid JSON (Is all error checking done at RM module itself? Are there any error checks at EM?)

• Validate if the deployment configuration has been updated with the specified config and env

• Specify another input JSONs with the same id/trial number and same config

• Specify another input JSONs with same id/trial number/deployment but with a change in metrics (a new tunable included/deleted)

• Specify another input JSONs with same id/trial number/deployment but with a change in env

• Specify another input JSONs with trial measurement run value greater than the run value

• Test with multiple input JSONS (experiments) with different configs (How many experiments can be queued?)

• Specify CPU and Memory Req values more than the actual resource and check for the behaviour

• If CPU/Mem Req > CPU/Mem limits, what is the behavior?

• What is the significance of app_version?

• Test the REST API (experiment_results) for valid/invalid app_names (Is querying based on autotune id supported?)

• Validate the autotune id,trial details against the input JSON

• Validate the output JSON returned and check it has returned values for all the metrics passed in the input JSON

• Do we just check for non-blank values for score, mean etc?

• Is there a way to simulate interrupting the experiment in between say while it is running the load?

• Query the EM multiple times and see if the output JSON values varies(check for consistency of result)

• Post Invalid Prometheus queries for some of the tunables

• Abort the datasource (for one of the tunables / for multiple tunables/ all tunables) from where metrics needs to be gathered, what is the behaviour? Are there retries with a timeout?

• Input JSON with memory values other than Mi

• Specify different datasources for different tunables

[chandrams]

Summary from last discussion on some of the queries:

Test with a valid input JSON to EM, check results using experiment_results API (after the specified duration? / loop / poll for status?)

• Update experiment_results API to get the status of Finite state machine

How do we pass the input JSON to EM?

• POST support will be included to do this

Test with an invalid JSON (Is all error checking done at RM module itself? Are there any error checks at EM?)

• Error checks will be done by Exp Mgr

What is the significance of app_version?

• app_version is not required and will be removed

Test the REST API (experiment_results) for valid/invalid app_names (Is querying based on autotune id supported?)

• At present experiment_results API supports Application name param, but support for autotune id will be added later.

Do we just check for non-blank/non-zero values for score, mean etc?

• We can check if values for percentiles are in ascending order for 95, 99, ... 99.999, 100
• We can check if mean is within a range - for example mean < 99 percentile, but what should the min value be i.e. mean > ?

What is the acceptable spike range?

• Not finalized yet

Values for "trial_result", "trial_result_info", "trial_result_error" is blank in output json?

• These fields are not required and may be removed

[chandrams]

Summary from today's design discussion:

Post Invalid Prometheus queries for some of the tunables

• DA & EM will need to include data source query expansion and query validation

Abort the data source (for one of the tunables / for multiple tunables / all tunables) from where metrics needs to be gathered, what is the behaviour? Are there retries with a timeout?

• Error out if metrics for a tunable cannot be fetched even after retries, EM needs to include this logic

Input JSON with memory values other than Mi

• CPU can also be 2m where m is milli cores, Code change is required to handle this
  https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Specify different data sources for different tunables

• EM should include code to validate if datasources match with datasources specified in Autotune object yaml
• Autotune object has a global datasource field that takes name & URL, this hasn't been tested. Add test cases to test this. If name is 'prometheus' on minikube, it should fail as we have used the same name in minikube config map.

Move all test related technical discussions in squad-issues to Autotune project

[chandrams]

• Pass only the input json without deploying the app

• Do not provide load, does EM wait for load to appear forever or is there a timeout?
  EM should asynchronously check for load. There should be a periodic check on load too in EM.

• What is the queue size? Is it dynamic ? If not how many experiments can be queued?
  Each Autotune object corresponds to one study. But still not clear on queuing mechanism

• Test with valid query but invalid datasource (prometheus query but specified datasource as graphite)

• Apply different load for same experiment and observe the result (Check if the metrics values vary)
  User load should be constant

• Will metrics on production deployment be captured? If so, this can be utilized for comparing production & trial metrics.
  This needs to be enabled in EM, needs further design discussion. Input json could have a flag to indicate if production metrics should be captured & output json will need to have both production & trial metrics.

• FSM state 'gathered metrics' can be used in the test to stop load
  FSM States are yet to be finalized and yes we will have a state to indicate this.

[chandrams]

• Is there any validation done for experiment id? What if it is blank?
  EM determines uniqueness of the experiment using exp id (autotune id), study id & trial no. in end-to-end flow
  EM standalone can generate a trial id
  Needs further discussion based on RM-EM flow

• Is EM going to check for the presence of production pod with the specified config?
  There is no config for production pod

• Test with invalid production pod details - deployment name is incorrect

• Test by passing an unsupported tunable or tunable value for training pod. For example, quarkus.thread-pool.core-threads for petclinic

• What happens production and training have different datasources?
  Production and training can have diff datasources as long as both datasources are valid

• What happens when we pass an extra metric / env parameter in the training pod?

• Delete the production pod in between and check for the behaviour

• Post multiple experiments in the same input json to EM

• Post multiple experiments with one invalid experiment, for example invalid deployment name. Does it fail or conduct only valid experiments and return an error for invalid one?

• Test without passing any training or production details

• Test by passing only training details or production details

[chandrams]

EM API doc - https://github.com/kruize/autotune/blob/b1292a974d376bad41cff15766285ca15ba4766b/design/API.md

New deployment/rolling update validation

• Deploy petclinic in default namespace and apply the input json with the trial config as rolling update

• Deploy petclinic in default namespace and apply the input json with the trial config as rolling update, apply the same input json again

  • Failure occurred in the first rolling update - either cleanup happens at EM or a state of the prev failure is maintained
  • Positive case, where prev rolling update didn't fail, a new runid is returned and both ids can be used for querying status
  • Include scenarios for rolling update with restart - yes/no

• Deploy petclinic in default namespace and apply the input json with the trial config as rolling update, apply input json with diff trial no. and same app name/ns

• Deploy petclinic in default namespace and apply the input json with the trial config as rolling update, apply input json for the same app and diff ns

• Deploy petclinic in petclinic namespace and apply the input json with the trial config as rolling update, apply input json with namespace as petclinic

• Deploy same petclinic app in default and petclinic namespaces and apply the input jsons with rolling update for both the apps/ns

• Deploy diff petclinic apps in default and petclinic namespaces and apply the input jsons with rolling update for both the apps/ns

• Deploy petclinic in default namespace and apply the input json with the trial config as rolling update, apply input json with diff trial no. and same app name/ns, when the second trial is in wait state,
  remove the petclinic app

• Deploy petclinic in default namespace and apply the input json with the trial config as new deployment

• Deploy petclinic in default namespace and apply the input json with one trial config with rolling update and another trial config as new deployment

• Deploy petclinic and galaxies apps, apply the input json with rolling update and verify the new deployment

• Validate deployment policy against deployment tracking

• Stability tests - against system scenarios

• Negative scenarios with config