Improving the webhook delivery system

[RichStone]

BT currently has a very short webhook delivery schedule and a brute force webhook delivery mechanism that does not have a notion of Webhooks::Outgoing::Endpoints being dead. This can lead to a large amount of delivery attempts to endpoints that are abandoned or misconfigured unintentionally. At the same time, app webhook endpoint creators (admins, API client platforms) currently don't have a way to know if they made a typo in their endpoint or if their endpoint is unhealthy otherwise for one reason or another.

One solution one be to go for a delivery model similar to what Stripe does, where an endpoint is marked inactive after a period of time without any successful responses, e.g., within 3 days. This also means that we'll extend the webhook delivery attempts for the same time with an exponential backoff.

Customers should be notified multiple time in the process:

1. The endpoint started to be unhealthy.
2. The endpoint is about to be deactivated.
3. The endpoint is deactivated.

We'd need to keep in mind that individual deliveries may fail (e.g., because of validation issues) but that the endpoint is intact overall because other deliveries succeed. Stripe deactivates endpoints only when all deliveries within a certain amount of time are failing.

In the ideal scenario, we would have reasonable defaults that are also configurable, e.g., in case a BT user wants to keep endpoints alive for shorter or longer periods of time.

[jagthedrummer]

@RichStone I haven't personally used outgoing webhooks in BT, so I'm not at all familiar with how everything works currently (and our documentation about it is very lacking). Do you already have a feel for exactly how we should approach this? Like, are there existing models and record keeping that we can lean on for this, or are we building the foundations for it from scratch?

Also, what exactly does "customers should be notified" mean? Notified how? Via email? Some sort of in-app notification system (I don't think we currently have such a system)? Who exactly should be notified? All members of the team that owns an endpoint?