Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Warning & Operation metrics #14323

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

hamistao
Copy link
Contributor

lxd_operations_total and lxd_warnings_total are currently being taken clusterwide, instead of returning just the entities related to the Node responding the metrics request. This goes against the overall design of the metrics, that are supposed to be per node and queried on each node on a cluster.
To fix this, this PR filters the queries for those entities based on the Node.

Copy link
Member

@tomponline tomponline left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do warnings always have a node?

@hamistao
Copy link
Contributor Author

Do warnings always have a node?

They do not, this is marked as Draft while I/we decide on how to handle nodeless warnings, I am thinking just include them in all node's metrics.

@tomponline
Copy link
Member

Do warnings always have a node?

They do not, this is marked as Draft while I/we decide on how to handle nodeless warnings, I am thinking just include them in all node's metrics.

Yeah, or just on the leader?

@hamistao
Copy link
Contributor Author

Yeah, or just on the leader?

I think this makes even more sense. Since Prometheus is supposed to scrape them on every node, counting them for each node can have the effect of readundantly counting these warnings many times when aggregating the measurements.

lxd/api_metrics.go Outdated Show resolved Hide resolved
hamistao and others added 7 commits October 23, 2024 06:48
Signed-off-by: hamistao <[email protected]>
(cherry picked from commit 24f150cf451eef796d879f1a191004ef0504b301)
Signed-off-by: hamistao <[email protected]>
(cherry picked from commit decddf5adfbd0c34c1f189c4eff16c19b3527abd)
Signed-off-by: Mark Laing <[email protected]>
(cherry picked from commit 5b844ba)
Signed-off-by: Mark Laing <[email protected]>
(cherry picked from commit 51526cb)
Signed-off-by: Mark Laing <[email protected]>
(cherry picked from commit 2492cd5)
Filters query for Warnings on the metrics handler by Node. Since some
Warnings do not have a node, nodeless Warnings are only being included
if querying the metrics from the leader node.

Signed-off-by: hamistao <[email protected]>
@hamistao hamistao force-pushed the fix_warning_operation_metrics branch from 471cc6e to 880764a Compare October 23, 2024 10:01
@hamistao
Copy link
Contributor Author

@tomponline @markylaing I am thinking on waiting for #14261 before proceeding here since I intend to use the LeaderInfo function defined there. I have cherry picked some commits to test and it all works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants