Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incident muting & notification suppression #200

Merged
merged 10 commits into from
Jun 25, 2024

Conversation

yhabteab
Copy link
Member

resolves #190

@cla-bot cla-bot bot added the cla/signed CLA is signed by all contributors of a PR label May 24, 2024
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch 3 times, most recently from 140694c to a7e1468 Compare May 27, 2024 07:54
@julianbrost julianbrost requested a review from oxzi May 27, 2024 08:26
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from a7e1468 to 7b542e3 Compare May 27, 2024 10:45
@yhabteab yhabteab requested review from oxzi and removed request for oxzi May 27, 2024 10:46
Copy link
Member

@oxzi oxzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a first inspection of the change. Please feel free to comment if you find something unfitting.

internal/event/event.go Outdated Show resolved Hide resolved
internal/icinga2/api_responses.go Show resolved Hide resolved
internal/icinga2/client.go Outdated Show resolved Hide resolved
internal/icinga2/client.go Outdated Show resolved Hide resolved
internal/icinga2/client.go Show resolved Hide resolved
Comment on lines 332 to 372
"filter": fmt.Sprintf(
"(event.type!=%q && event.type!=%q) || event.object_type==%q || event.object_type==%q",
typeObjectCreated, typeObjectDeleted, "Host", "Service",
),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please elaborate on this filter? You want all host or service event objects and every other event type unless they are ObjectCreated or ObjectDeleted. But why? Especially as those are listed within the given esTypes argument.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Object created for all Icinga 2 objects.

This is a quote from the Icinga 2 event stream docs for the ObjectCreated event and means that this event is triggered for every single object of Icinga 2 of any type, but I don't really care about any other types than Host and Service. Therefore, I want to check if event.type is not ObjectCreated and not ObjectDeleted (this means that the event type could be any other type listed in esTypes), otherwise (this means that event.type is either ObjectCreated or ObjectDeleted), I want to additionally filter on the object_type and match only on the Host or Service types.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be more readable if it was a little more verbose if expression. Something like the following should also work as a filter:

if (event.type == "ObjectCreated" || event.type == "ObjectDeleted") {
	event.object_type == "Host" || event.object_type == "Service"
} else {
	true
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like the following should also work as a filter:

Did you test it? Otherwise you should have already noticed that this doesn't work.

I think this would be more readable if it was a little more verbose if expression.

This is a simple API filter expression, I'm not sure what exactly you find it to be unreadable, and apart from that I just didn't want to use the literal ObjectCreated while there is a constant typeObjectCreated for it, and no, you can't just use event.object_type == "Host" || event.object_type == "Service" or (event.type == "ObjectCreated" || event.type == "ObjectDeleted") as a filter value, you have to use quoted values "\"FOO\"".

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like the following should also work as a filter:

Did you test it? Otherwise you should have already noticed that this doesn't work.

I didn't test it within the code here, I just verified in icinga2 console that if expressions indeed evaluate to a value so I'd expect it to work here as well:

Icinga 2 (version: v2.14.2)
Type $help to view available commands.
<1> => if (true) { 23 } else { 42 }
23.000000
<2> => if (false) { 23 } else { 42 }
42.000000

I think this would be more readable if it was a little more verbose if expression.

This is a simple API filter expression, I'm not sure what exactly you find it to be unreadable, and apart from that I just didn't want to use the literal ObjectCreated while there is a constant typeObjectCreated for it

That comment was primarily about the structure of the condition. Isn't the purpose of the filter to perform a check for some event types and leave the other ones unfiltered? I think such an if would convey this in a more straight-forward way.

and no, you can't just use event.object_type == "Host" || event.object_type == "Service" or (event.type == "ObjectCreated" || event.type == "ObjectDeleted") as a filter value, you have to use quoted values "\"FOO\"".

Yes, you can, you can alternatively enclose string literals in `` and then don't need to escape double quotes.

internal/incident/history_event_type.go Outdated Show resolved Hide resolved
internal/incident/incident.go Show resolved Hide resolved
internal/incident/incident.go Show resolved Hide resolved
internal/incident/incident.go Outdated Show resolved Hide resolved
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch 2 times, most recently from 0c6241f to c0a1413 Compare May 28, 2024 08:42
@yhabteab yhabteab requested a review from oxzi May 28, 2024 08:46
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch 2 times, most recently from bc319a8 to 4bc76ec Compare June 4, 2024 06:47
@yhabteab
Copy link
Member Author

yhabteab commented Jun 4, 2024

Have just rebased the PR to resolve the conflicts resulting from #114.

internal/icinga2/client.go Outdated Show resolved Hide resolved
internal/icinga2/client.go Outdated Show resolved Hide resolved
internal/icinga2/client.go Outdated Show resolved Hide resolved
internal/icinga2/client.go Show resolved Hide resolved
internal/icinga2/client.go Outdated Show resolved Hide resolved
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from 4bc76ec to 3c76ed8 Compare June 4, 2024 08:56
@yhabteab yhabteab requested a review from oxzi June 4, 2024 08:59
internal/icinga2/client.go Outdated Show resolved Hide resolved
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from 3c76ed8 to 1b6f3c3 Compare June 4, 2024 09:35
@yhabteab yhabteab requested a review from oxzi June 4, 2024 09:36
internal/object/object.go Outdated Show resolved Hide resolved
internal/incident/incident.go Outdated Show resolved Hide resolved
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from 1b6f3c3 to f8069ea Compare June 4, 2024 15:04
oxzi
oxzi previously approved these changes Jun 4, 2024
Copy link
Member

@oxzi oxzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have successfully tested the PR!
@julianbrost, please take another look.

@oxzi oxzi requested a review from julianbrost June 4, 2024 15:24
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from f8069ea to 246f0f0 Compare June 5, 2024 10:11
@yhabteab
Copy link
Member Author

yhabteab commented Jun 5, 2024

I have just rebased it and committed new changes separately 246f0f0.

oxzi
oxzi previously approved these changes Jun 6, 2024
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from 2799ffa to 643baaa Compare June 17, 2024 14:00
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from 643baaa to 252df4c Compare June 18, 2024 12:42
yhabteab and others added 8 commits June 18, 2024 17:07
The object shouldn't be added to the cache store before committing the
ongoing database transaction. Apart from that, this commit ensures that
the object in memory state represents the information stored in the DB.
Host/Service groups are never supposed to change at runtime, thus we can
use these two new events to regularly refresh our cache store. This
eliminates the overhead of querying the groups with each ongoing event
and should relax the Icinga 2 API a litle bit.
Doing so prevents us from triggering notifications for them when
starting to process events for them, and also avoid invalidating the
actual mute reason with a made-up ones.
@yhabteab yhabteab force-pushed the incident-muting-notification-suppression branch from 252df4c to 68b280c Compare June 18, 2024 15:08
@julianbrost
Copy link
Collaborator

@oxzi FYI: I don't expect further large changes to this PR, so feel free to have another look.

@julianbrost julianbrost requested a review from oxzi June 19, 2024 09:57
internal/icinga2/api_responses.go Outdated Show resolved Hide resolved
internal/icinga2/api_responses.go Outdated Show resolved Hide resolved
@oxzi
Copy link
Member

oxzi commented Jun 24, 2024

Otherwise, I have successfully tested the PR and have not seen anything suspicious in the diff.

Copy link
Collaborator

@julianbrost julianbrost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with the current state now, I'm just not yet clicking the approve button because I still want you to address @oxzi's recent review.

@julianbrost julianbrost merged commit e6f870d into main Jun 25, 2024
12 checks passed
@julianbrost julianbrost deleted the incident-muting-notification-suppression branch June 25, 2024 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla/signed CLA is signed by all contributors of a PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incident muting (for downtimes, flapping, acknowledgements)
3 participants