New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add missing cgroup memory panels ... #139

Merged

afcollins merged 4 commits into cloud-bulldozer:master from afcollins:cgroupMemory

Dec 13, 2024

Member

afcollins commented Nov 8, 2024

Also adds a sum line on right axis for memory available.

Type of change

New feature
Bug fix

Description

Adding memory panels for critical cgroup panels added previously.
Introduces a 'sum' query on right-hand axis for aggregated worker and control plane panels.
Finally, fixes a variable that pulls in workload node into worker panels which can be confusing to users since workload node (ironically) isn't used for workload scheduling.

Checklist before requesting a review

I have performed a self-review of my code.
If it is a core feature, I have added thorough tests.

afcollins marked this pull request as draft

November 15, 2024 17:35

Member Author

afcollins commented Nov 15, 2024

Will add some more panels I think are missing


          Add missing cgroup memory panels ...

c315d67

Also adds a sum line on right axis for memory available.

afcollins force-pushed the cgroupMemory branch from 47743c6 to c315d67 Compare

December 4, 2024 18:47


          fs panel name

0d78160

Signed-off-by: Andrew Collins <[email protected]>

afcollins commented

View reviewed changes

Member Author

afcollins left a comment

Made several changes to a classic dashboard. Particularly changes to memory panels, like removing 'mean' from the legend.

Please let me know what you think!

assets/ocp-performance/queries.libsonnet

+                containerReadWriteBytesPod: {
+                  query(nodeName):
+                    generateTimeSeriesQuery('sum(rate(container_fs_writes_bytes_total{device!~".+dm.+", node=~"' + nodeName + '", pod!=""}[$interval])) by (device, pod)', '{{ pod }}: {{ device }} - write')
+                    + generateTimeSeriesQuery('sum(rate(container_fs_reads_bytes_total{device!~".+dm.+", node=~"' + nodeName + '", pod!=""}[$interval])) by (device, pod)', '{{ pod }}: {{ device }} - read'),

Member Author

afcollins Dec 4, 2024

Query labels have since changed, leaving this panel empty. Also, only included writes, so I added reads.

assets/ocp-performance/queries.libsonnet

                 },
                 crioMemory: {
                   query():
-                    generateTimeSeriesQuery('topk(10,process_resident_memory_bytes{service="kubelet",job="crio"})', 'crio - {{node}}'),
+                    generateTimeSeriesQuery('topk(10,process_resident_memory_bytes{service="kubelet",job="crio"} *  on (node) group_left kube_node_role{ role = "worker" })', 'crio - {{node}}'),

Member Author

afcollins Dec 4, 2024

We weren't filtering on workers, so control plane nodes would show up if workers < 10

assets/ocp-performance/queries.libsonnet

                 },
                 crioINodes: {
                   query():
-                    generateTimeSeriesQuery('(1 - node_filesystem_files_free{fstype!="",mountpoint="/run"} / node_filesystem_files{fstype!="",mountpoint="/run"}) * 100', '/var/run - {{instance}}'),
+                    generateTimeSeriesQuery('(1 - node_filesystem_files_free{fstype!="",mountpoint="/run"} / node_filesystem_files{fstype!="",mountpoint="/run"}) * 100', '{{instance}}'),

Member Author

afcollins Dec 4, 2024

/var/run wasn't in the query and also redundant in the label

assets/ocp-performance/variables.libsonnet Outdated

    
            @@ -22,7 +22,7 @@ local var = g.dashboard.variable;
          
                  + var.query.withDatasourceFromVariable(self.datasource)

                  + var.query.queryTypes.withLabelValues(

                    'node',

                    'kube_node_role{role=~"work.*"}',

                    'kube_node_role{role=~"worker"}',  // Do we want to include the workload node for some reason? WHy is it a regexp?

Member Author

afcollins Dec 4, 2024

pulled in workload node if there was one, which we don't need metrics for and was confusing.

templates/General/ocp-performance.jsonnet

-                  panels.timeSeries.genericLegend('Workers Disk IOPS', 'short', queries.workersIOPS.query(), { x: 0, y: 41, w: 12, h: 8 }),
-                  panels.timeSeries.genericLegend('Control Plane Disk IOPS', 'short', queries.controlPlaneIOPS.query(), { x: 12, y: 41, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounterSumRightHand('Workers Memory Available', 'bytes', queries.workersMemoryAvailable.query(), { x: 0, y: 17, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounterSumRightHand('Control Plane Memory Available', 'bytes', queries.controlPlaneMemoryAvailable.query(), { x: 12, y: 17, w: 12, h: 8 }),

Member Author

afcollins Dec 4, 2024

Reordered. Changed to SumRightHand, which can be helpful to remove individual worker variances and uses different legend stats. Looks like:

templates/General/ocp-performance.jsonnet

                   panels.timeSeries.genericLegend('Top 10 container CPU: $_master_node', 'percent', queries.top10ContainerCPU.query('$_master_node'), { x: 0, y: 24, w: 12, h: 8 }),
-                  panels.timeSeries.genericLegend('Top 10 container RSS: $_master_node', 'bytes', queries.top10ContainerRSS.query('$_master_node'), { x: 12, y: 24, w: 12, h: 8 }),
-                  panels.timeSeries.genericLegend('Container fs write rate: $_master_node', 'Bps', queries.containerWriteBytes.query('$_master_node'), { x: 0, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounter('Top 10 container RSS: $_master_node', 'bytes', queries.top10ContainerRSS.query('$_master_node'), { x: 12, y: 24, w: 12, h: 8 }),

Member Author

afcollins Dec 4, 2024

genericLegendCounter intended for memory

templates/General/ocp-performance.jsonnet

                   panels.timeSeries.genericLegend('Top 10 container CPU: $_master_node', 'percent', queries.top10ContainerCPU.query('$_master_node'), { x: 0, y: 24, w: 12, h: 8 }),
-                  panels.timeSeries.genericLegend('Top 10 container RSS: $_master_node', 'bytes', queries.top10ContainerRSS.query('$_master_node'), { x: 12, y: 24, w: 12, h: 8 }),
-                  panels.timeSeries.genericLegend('Container fs write rate: $_master_node', 'Bps', queries.containerWriteBytes.query('$_master_node'), { x: 0, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounter('Top 10 container RSS: $_master_node', 'bytes', queries.top10ContainerRSS.query('$_master_node'), { x: 12, y: 24, w: 12, h: 8 }),

Member Author

afcollins Dec 4, 2024

genericLegendCounter intended for memory and non-rates (i.e. conntrack)

templates/General/ocp-performance.jsonnet Outdated

+                  panels.timeSeries.genericLegend('cgroup CPU: $_master_node', 'percent', queries.nodeCGroupCPU.query('$_master_node'), { x: 0, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounter('cgroup RSS: $_master_node', 'bytes', queries.nodeCGroupRSS.query('$_master_node'), { x: 12, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegend('Pod fs write rate: $_master_node', 'Bps', queries.containerReadWriteBytesPod.query('$_master_node'), { x: 0, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegend('cgroup fs write rate: $_master_node', 'Bps', queries.containerReadWriteBytesCGroup.query('$_master_node'), { x: 12, y: 32, w: 12, h: 8 }),

Member Author

afcollins Dec 4, 2024

Break down pod vs top cgroup fs read/write usage. Example:

templates/General/ocp-performance.jsonnet

-                  panels.timeSeries.genericLegend('Container fs write rate: $_master_node', 'Bps', queries.containerWriteBytes.query('$_master_node'), { x: 0, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounter('Top 10 container RSS: $_master_node', 'bytes', queries.top10ContainerRSS.query('$_master_node'), { x: 12, y: 24, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegend('cgroup CPU: $_master_node', 'percent', queries.nodeCGroupCPU.query('$_master_node'), { x: 0, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounter('cgroup RSS: $_master_node', 'bytes', queries.nodeCGroupRSS.query('$_master_node'), { x: 12, y: 32, w: 12, h: 8 }),

Member Author

afcollins Dec 4, 2024

example:

templates/General/ocp-performance.jsonnet

+                  panels.timeSeries.genericLegend('Top 10 container CPU: $_worker_node', 'percent', queries.top10ContainerCPU.query('$_worker_node'), { x: 0, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounter('Top 10 container RSS: $_worker_node', 'bytes', queries.top10ContainerRSS.query('$_worker_node'), { x: 12, y: 32, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegend('cgroup CPU: $_worker_node', 'percent', queries.nodeCGroupCPU.query('$_worker_node'), { x: 0, y: 40, w: 12, h: 8 }),
+                  panels.timeSeries.genericLegendCounter('cgroup RSS: $_worker_node', 'bytes', queries.nodeCGroupRSS.query('$_worker_node'), { x: 12, y: 40, w: 12, h: 8 }),

Member Author

afcollins Dec 4, 2024

See above comments about master node

afcollins marked this pull request as ready for review

December 4, 2024 19:12

afcollins added 2 commits

December 10, 2024 09:28


          Merge branch 'master' into cgroupMemory

c5dfb96


          Remove comment

c841e02

Signed-off-by: Andrew Collins <[email protected]>

rsevilla87 self-requested a review

December 10, 2024 15:45

afcollins merged commit f9d95ae into cloud-bulldozer:master

2 checks passed

afcollins deleted the cgroupMemory branch

December 13, 2024 20:16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet