Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combining metrics profiles files #411

Merged
merged 5 commits into from
Sep 30, 2022

Conversation

paigerube14
Copy link
Collaborator

@paigerube14 paigerube14 commented May 26, 2022

Description

Want to combine the metrics and metrics-ovn yaml files so the default metrics file has podCPU and podMemory that can be used when creating the benchmark-comparison for kube-burner

Fixes

  1. Needing to set metrics-ovn.yaml file to be able to use all of the touchstone-config files for google sheet comparison generation
  2. Remove duplication of files that are very similar

@comet-perf-ci
Copy link
Collaborator

Can one of the admins verify this patch?

@paigerube14
Copy link
Collaborator Author

@rsevilla87 @mohit-sheth @dry923 can you all please take a look when you get the chance

@paigerube14
Copy link
Collaborator Author

@rsevilla87 @dry923 are we able to get this PR merged? The default metrics profile doesn't collect all the data that is available when using all the touchstone-configs here

09-26 09:28:11.949  2022-09-26 13:28:11,548 - touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'metricName.keyword': 'podCPU', 'labels.namespace.keyword': 'openshift-sdn'}, 'buckets': ['labels.pod.keyword', 'labels.node.keyword'], 'aggregations': {'value': ['avg']}}
09-26 09:28:12.522  Traceback (most recent call last):
09-26 09:28:12.522    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 70, in <module>
09-26 09:28:12.522      main()
09-26 09:28:12.522    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 53, in main
09-26 09:28:12.522      df = read_csv(csv_filepath)
09-26 09:28:12.522    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 35, in read_csv
09-26 09:28:12.522      df = pd.read_csv(filepath)
09-26 09:28:12.522    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
09-26 09:28:12.522      return func(*args, **kwargs)
09-26 09:28:12.522    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 317, in wrapper
09-26 09:28:12.522      return func(*args, **kwargs)
09-26 09:28:12.522    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
09-26 09:28:12.522      return _read(filepath_or_buffer, kwds)
09-26 09:28:12.522    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
09-26 09:28:12.522      parser = TextFileReader(filepath_or_buffer, **kwds)
09-26 09:28:12.522    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
09-26 09:28:12.522      self._engine = self._make_engine(f, self.engine)
09-26 09:28:12.522    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1747, in _make_engine
09-26 09:28:12.522      return mapping[engine](f, **self.options)
09-26 09:28:12.522    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 92, in __init__
09-26 09:28:12.522      self._reader = parsers.TextReader(src, **kwds)
09-26 09:28:12.522    File "pandas/_libs/parsers.pyx", line 554, in pandas._libs.parsers.TextReader.__cinit__
09-26 09:28:12.522  pandas.errors.EmptyDataError: No columns to parse from file
09-26 09:28:12.780  �[1mMon Sep 26 13:28:12 UTC 2022 Querying results�[0m
09-26 09:28:12.780  �[1mMon Sep 26 13:28:12 UTC 2022 Running: touchstone_compare --database elasticsearch -url [https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443](https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com/) -u f6ea8e10-7001-4f6a-9c02-967850db998d --config /tmp/podCPU-max.json -o csv --output-file /tmp/cluster-density-f6ea8e10-7001-4f6a-9c02-967850db998d/podCPU-max.json�[0m
09-26 09:28:14.163  2022-09-26 13:28:13,768 - touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'metricName.keyword': 'podCPU', 'labels.namespace.keyword': 'openshift-sdn'}, 'buckets': ['labels.pod.keyword', 'labels.node.keyword'], 'aggregations': {'value': ['max']}}
09-26 09:28:14.732  Traceback (most recent call last):
09-26 09:28:14.732    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 70, in <module>
09-26 09:28:14.732      main()
09-26 09:28:14.732    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 53, in main
09-26 09:28:14.732      df = read_csv(csv_filepath)
09-26 09:28:14.732    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 35, in read_csv
09-26 09:28:14.732      df = pd.read_csv(filepath)
09-26 09:28:14.732    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
09-26 09:28:14.732      return func(*args, **kwargs)
09-26 09:28:14.732    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 317, in wrapper
09-26 09:28:14.732      return func(*args, **kwargs)
09-26 09:28:14.732    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
09-26 09:28:14.732      return _read(filepath_or_buffer, kwds)
09-26 09:28:14.732    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
09-26 09:28:14.732      parser = TextFileReader(filepath_or_buffer, **kwds)
09-26 09:28:14.732    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
09-26 09:28:14.732      self._engine = self._make_engine(f, self.engine)
09-26 09:28:14.732    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1747, in _make_engine
09-26 09:28:14.732      return mapping[engine](f, **self.options)
09-26 09:28:14.732    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 92, in __init__
09-26 09:28:14.732      self._reader = parsers.TextReader(src, **kwds)
09-26 09:28:14.732    File "pandas/_libs/parsers.pyx", line 554, in pandas._libs.parsers.TextReader.__cinit__
09-26 09:28:14.732  pandas.errors.EmptyDataError: No columns to parse from file
09-26 09:28:14.732  podMemory-avg.json
09-26 09:28:14.732  �[1mMon Sep 26 13:28:14 UTC 2022 Querying results�[0m
09-26 09:28:14.732  �[1mMon Sep 26 13:28:14 UTC 2022 Running: touchstone_compare --database elasticsearch -url [https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443](https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com/) -u f6ea8e10-7001-4f6a-9c02-967850db998d --config /tmp/podMemory-avg.json -o csv --output-file /tmp/cluster-density-f6ea8e10-7001-4f6a-9c02-967850db998d/podMemory-avg.json�[0m
09-26 09:28:16.165  2022-09-26 13:28:15,726 - touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'metricName.keyword': 'podMemory', 'labels.namespace.keyword': 'openshift-sdn'}, 'buckets': ['labels.pod.keyword', 'labels.node.keyword'], 'aggregations': {'value': ['avg']}}
09-26 09:28:16.730  Traceback (most recent call last):
09-26 09:28:16.730    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 70, in <module>
09-26 09:28:16.730      main()
09-26 09:28:16.730    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 53, in main
09-26 09:28:16.730      df = read_csv(csv_filepath)
09-26 09:28:16.730    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 35, in read_csv
09-26 09:28:16.730      df = pd.read_csv(filepath)
09-26 09:28:16.730    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
09-26 09:28:16.730      return func(*args, **kwargs)
09-26 09:28:16.730    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 317, in wrapper
09-26 09:28:16.730      return func(*args, **kwargs)
09-26 09:28:16.730    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
09-26 09:28:16.730      return _read(filepath_or_buffer, kwds)
09-26 09:28:16.730    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
09-26 09:28:16.730      parser = TextFileReader(filepath_or_buffer, **kwds)
09-26 09:28:16.730    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
09-26 09:28:16.730      self._engine = self._make_engine(f, self.engine)
09-26 09:28:16.730    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1747, in _make_engine
09-26 09:28:16.730      return mapping[engine](f, **self.options)
09-26 09:28:16.730    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 92, in __init__
09-26 09:28:16.730      self._reader = parsers.TextReader(src, **kwds)
09-26 09:28:16.730    File "pandas/_libs/parsers.pyx", line 554, in pandas._libs.parsers.TextReader.__cinit__
09-26 09:28:16.730  pandas.errors.EmptyDataError: No columns to parse from file
09-26 09:28:16.730  podMemory-max.json
09-26 09:28:16.730  �[1mMon Sep 26 13:28:16 UTC 2022 Querying results�[0m
09-26 09:28:16.730  �[1mMon Sep 26 13:28:16 UTC 2022 Running: touchstone_compare --database elasticsearch -url [https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443](https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com/) -u f6ea8e10-7001-4f6a-9c02-967850db998d --config /tmp/podMemory-max.json -o csv --output-file /tmp/cluster-density-f6ea8e10-7001-4f6a-9c02-967850db998d/podMemory-max.json�[0m
09-26 09:28:17.728  2022-09-26 13:28:17,713 - touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'metricName.keyword': 'podMemory', 'labels.namespace.keyword': 'openshift-sdn'}, 'buckets': ['labels.pod.keyword', 'labels.node.keyword'], 'aggregations': {'value': ['max']}}
09-26 09:28:18.726  Traceback (most recent call last):
09-26 09:28:18.726    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 70, in <module>
09-26 09:28:18.726      main()
09-26 09:28:18.726    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 53, in main
09-26 09:28:18.726      df = read_csv(csv_filepath)
09-26 09:28:18.726    File "/home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 35, in read_csv
09-26 09:28:18.726      df = pd.read_csv(filepath)
09-26 09:28:18.726    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
09-26 09:28:18.726      return func(*args, **kwargs)
09-26 09:28:18.726    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 317, in wrapper
09-26 09:28:18.726      return func(*args, **kwargs)
09-26 09:28:18.726    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
09-26 09:28:18.726      return _read(filepath_or_buffer, kwds)
09-26 09:28:18.726    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
09-26 09:28:18.726      parser = TextFileReader(filepath_or_buffer, **kwds)
09-26 09:28:18.726    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
09-26 09:28:18.726      self._engine = self._make_engine(f, self.engine)
09-26 09:28:18.726    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1747, in _make_engine
09-26 09:28:18.726      return mapping[engine](f, **self.options)
09-26 09:28:18.726    File "/tmp/tmp.sUaaQWVrxv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 92, in __init__
09-26 09:28:18.726      self._reader = parsers.TextReader(src, **kwds)
09-26 09:28:18.726    File "pandas/_libs/parsers.pyx", line 554, in pandas._libs.parsers.TextReader.__cinit__
09-26 09:28:18.726  pandas.errors.EmptyDataError: No columns to parse from file
09-26 09:28:18.726  �[1mMon Sep 26 13:28:18 UTC 2022 Installing requirements to generate spreadsheet�[0m
09-26 09:28:23.993  WARNING: You are using pip version 21.1.3; however, version 22.2.2 is available.
09-26 09:28:23.993  You should consider upgrading via the '/tmp/tmp.sUaaQWVrxv/bin/python3 -m pip install --upgrade pip' command.
09-26 09:28:23.993  �[1mMon Sep 26 13:28:23 UTC 2022 Sheetname: cluster-density-2022-09-26T13:28:23�[0m
09-26 09:28:23.993  �[1mMon Sep 26 13:28:23 UTC 2022 CSV: /tmp/cluster-density-f6ea8e10-7001-4f6a-9c02-967850db998d/f6ea8e10-7001-4f6a-9c02-967850db998d.csv�[0m
09-26 09:28:23.993  �[1mMon Sep 26 13:28:23 UTC 2022 Email: [email protected]�[0m
09-26 09:28:23.993  �[1mMon Sep 26 13:28:23 UTC 2022 Service-account: /home/jenkins/ws/workspace/multibranch-pipeline_kube-burner/.gsheet.json�[0m

@paigerube14
Copy link
Collaborator Author

@rsevilla87 @jtaleric @dry923 any thoughts on this? Can we get this merged?

Screen Shot 2022-09-29 at 10 06 53 AM (2) the only difference in those 2 files is the podCPU and podMemory.

This small difference is also causing discrepancies in the grafana dashboards in the node section because it’s looking for “podCPU” that isn’t captured in the metrics.yaml file that nodes that aren't for the UUID are being listed
uuid.keyword: $uuid AND metricName: "podCPU" AND labels.node.keyword: $master AND labels.namespace.keyword: $namespace
Screen Shot 2022-09-29 at 10 12 48 AM

Copy link
Member

@jtaleric jtaleric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - I am however concerned touchstone doesn't do a good job with error handling :(

@paigerube14
Copy link
Collaborator Author

Think the issue with touchstone is similar to : cloud-bulldozer/benchmark-comparison#57 not sure if it's the exact same thing though

Comment on lines 21 to 25
- query: sum(irate(container_cpu_usage_seconds_total{name!="",namespace=~"openshift-(etcd|oauth-apiserver|.*apiserver|ovn-kubernetes|sdn|ingress|authentication|.*controller-manager|.*scheduler|monitoring|logging|image-registry)"}[2m]) * 100) by (pod, namespace, node)
metricName: podCPU

- query: sum(container_memory_rss{name!="",namespace=~"openshift-(etcd|oauth-apiserver|.*apiserver|ovn-kubernetes|sdn|ingress|authentication|.*controller-manager|.*scheduler|monitoring|logging|image-registry)"}) by (pod, namespace, node)
metricName: podMemory
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we're using containerCPU and containerMemory metrics in touchstone I think we can skip add these two queries here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with this being the only difference in these 2 files can we not just add these 2 queries and combine?

@@ -0,0 +1,21 @@
{
"elasticsearch": {
"ripsaw-kube-burner": [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 4 configuration files rather than a single one holding all queries?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sent a example on slack on how to consolidate this.

@rsevilla87
Copy link
Member

Hey @paigerube14, I managed to query CPU and memory metrics from pods using the new touchstone config files you've added:

(touchstone) rsevilla@wonderland ~/labs/touchstone (master) $ touchstone_compare --database elasticsearch -url nope -u  ab19641c-node-density-20220921   --config kubeburner.json 
+--------------+--------------------------+----------------------+------------------+------------+--------------------------------+
|  metricName  |     labels.namespace     |      labels.pod      |   labels.node    |   metric   | ab19641c-node-density-20220921 |
+--------------+--------------------------+----------------------+------------------+------------+--------------------------------+
| containerCPU | openshift-ovn-kubernetes | ovnkube-master-v4cdp |     master-1     | avg(value) |       5.589392743554736        |
| containerCPU | openshift-ovn-kubernetes | ovnkube-master-xbxhz |     master-2     | avg(value) |       5.230984553671838        |
| containerCPU | openshift-ovn-kubernetes | ovnkube-master-mtdc7 |     master-0     | avg(value) |       1.4631622188757265       |
| containerCPU | openshift-ovn-kubernetes |  ovnkube-node-84rpf  | worker011-5039ms | avg(value) |       1.6327197147460035       |
| containerCPU | openshift-ovn-kubernetes |  ovnkube-node-p2w9h  | worker019-5039ms | avg(value) |       1.5396597782833599       |
| containerCPU | openshift-ovn-kubernetes |  ovnkube-node-fqwvh  |     master-1     | avg(value) |       0.6787404186677101       |
| containerCPU | openshift-ovn-kubernetes |  ovnkube-node-9xwx2  | worker005-5039ms | avg(value) |       1.5002050950309012       |
| containerCPU | openshift-ovn-kubernetes |  ovnkube-node-jg97h  | worker003-5039ms | avg(value) |       1.5401379679754268       |

I think we can skip podCPU and podMemory metrics in metrics.yaml

Copy link
Member

@rsevilla87 rsevilla87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@rsevilla87 rsevilla87 merged commit 3b7c608 into cloud-bulldozer:master Sep 30, 2022
vishnuchalla pushed a commit that referenced this pull request Sep 6, 2023
* combining metrics profiles

* adding container touchstone configs

* adding overall container metrics

* deleting old files

* adding metrics-ovn back
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants