docs: notification wait for & silences

mission-control/docs/notifications/concepts/silences.mdx

@@ -0,0 +1,28 @@
+---
+title: Silences
+---
+
+Silences prevent notifications from being sent. 
+Silences are attached to a particular resource (`catalog`, `health check`, or `component`) and are active for a specified duration.
+
+:::note
+Notifications that aren't sent due to silence are still visible in the notification history for auditing purposes.
+:::
+
+## Use cases
+
+- Planned maintenance or deployments. Eg: You can silence a namespace or a helm release and automatically silence notifications from all of their children.
+- Non-critical resources: Notifications from resources that routinely trigger alerts but are expected and harmless can be silenced.
+- Known issues: If there's a known issue that can't be immediately resolved (e.g., due to dependencies or resource constraints), you might silence related alerts until a fix can be implemented.
+
+## Add Silence
+
+Silences can be added from the notification page. Alternatively, if you're using the default slack notification templates, you get a silence button
+on each notification.
+
+![Silence Notification form](./silence-notification-form.png)
+
+## Recursive mode
+
+When a silence is recursively applied, it applies to all of its children. 
+So silencing a namespace would silence all deployments, statefulsets, pods, etc in that namespace. 

mission-control/docs/notifications/concepts/wait-for.mdx

@@ -0,0 +1,32 @@
+---
+title: Wait For
+---
+
+Kubernetes clusters and similar dynamic systems may experience temporary discrepancies between the actual and intended state of resources. 
+For example, a deployment could momentarily appear unhealthy during a scaling operation.
+If alerts are configured for `config.unhealthy` events, these transient state fluctuations might lead to an overwhelming number of unnecessary notifications.
+
+To address this issue, you can utilize the waitFor parameter.
+This feature allows you to define a delay before sending notifications for specific events.
+After an event occurs, the system will recheck its status following the specified wait period. Only if the undesired state persists will a notification be triggered.
+
+:::info
+`waitFor` is only applicable on health related events
+:::
+
+This approach helps reduce unnecessary notifications caused by transient state changes, ensuring you're alerted only to persistent issues.
+
+
+```yaml title='notify-unhealthy-deployments.yaml' {8}
+apiVersion: mission-control.flanksource.com/v1
+kind: Notification
+metadata:
+  name: deployment-unhealthy-alerts
+spec:
+  events:
+    - config.unhealthy
+  waitFor: 2m
+  filter: config.type == 'Kubernetes::Deployment'
+  to:
+    email: [email protected]
+```

mission-control/docs/reference/notifications/_notification.mdx

@@ -32,24 +32,29 @@
   },
   {
     field: "to.properties",
     description: "Properties are channel dependent special directives to modify the notification message. Example: for email, `[email protected]` modifies the sender of the email. [Read more](https://containrrr.dev/shoutrrr/v0.8/)",
     scheme: "`map[string]string`"
   },
   {
     field: "repeatInterval",
     description: "The waiting time to resend a notification after it has been succefully sent.",
     scheme: "duration"
   },
   {
     field: "repeatGroup",
     description: "RepeatGroup allows notifications to be grouped by certain set of keys and only send one per group within the specified repeat interval. Valid group keys: `resource_id` & `source_event`.",
     scheme: "`[]string`"
   },
+  {
+    field: "waitFor",
+    description: "The duration to delay sending a health-based notification.\n\nAfter this period, the health status is reassessed to confirm it hasn't changed, helping prevent false alarms from transient issues.",
+    scheme: "duration"
+  },
   {
     field: "title",
     description: "Channel dependent e.g. subject for email",
     scheme: "GoTemplate",
     templateEnv: [
       {"name": "CheckEvents", "url": "/reference/notifications/template_vars#check-health-env"},
       {"name": "ConfigEvents", "url": "/reference/notifications/template_vars#config-health-env"},
       {"name": "ComponentEvents", "url": "/reference/notifications/template_vars#component-health-env"},
@@ -59,7 +64,7 @@
     field: "body",
     description: "Channel dependent e.g. email body for email",
     scheme: "GoTemplate",
     templateEnv: [
       {"name": "CheckEvents", "url": "/reference/notifications/template_vars#check-health-env"},
       {"name": "ConfigEvents", "url": "/reference/notifications/template_vars#config-health-env"},
       {"name": "ComponentEvents", "url": "/reference/notifications/template_vars#component-health-env"},
@@ -67,9 +72,9 @@
   },  
   {
     field: "filter",
     description: "Filter narrows down the event trigger.\n\nExample: `check.type == 'http'` filter on `check.failed` event will only send notification for failing http checks.",
     scheme: "CEL",
     templateEnv: [
       {"name": "CheckEvents", "url": "/reference/notifications/template_vars#check-health-env"},
       {"name": "ConfigEvents", "url": "/reference/notifications/template_vars#config-health-env"},
       {"name": "ComponentEvents", "url": "/reference/notifications/template_vars#component-health-env"},