-
Notifications
You must be signed in to change notification settings - Fork 931
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Warning & Operation metrics #14323
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do warnings always have a node?
They do not, this is marked as Draft while I/we decide on how to handle nodeless warnings, I am thinking just include them in all node's metrics. |
Yeah, or just on the leader? |
I think this makes even more sense. Since Prometheus is supposed to scrape them on every node, counting them for each node can have the effect of readundantly counting these warnings many times when aggregating the measurements. |
Signed-off-by: hamistao <[email protected]>
Signed-off-by: hamistao <[email protected]> (cherry picked from commit 24f150cf451eef796d879f1a191004ef0504b301)
Signed-off-by: hamistao <[email protected]> (cherry picked from commit decddf5adfbd0c34c1f189c4eff16c19b3527abd)
Signed-off-by: Mark Laing <[email protected]> (cherry picked from commit 5b844ba)
Signed-off-by: Mark Laing <[email protected]> (cherry picked from commit 51526cb)
Signed-off-by: Mark Laing <[email protected]> (cherry picked from commit 2492cd5)
Filters query for Warnings on the metrics handler by Node. Since some Warnings do not have a node, nodeless Warnings are only being included if querying the metrics from the leader node. Signed-off-by: hamistao <[email protected]>
471cc6e
to
880764a
Compare
@tomponline @markylaing I am thinking on waiting for #14261 before proceeding here since I intend to use the |
lxd_operations_total
andlxd_warnings_total
are currently being taken clusterwide, instead of returning just the entities related to the Node responding the metrics request. This goes against the overall design of the metrics, that are supposed to be per node and queried on each node on a cluster.To fix this, this PR filters the queries for those entities based on the Node.