[OpinionatedReconciler] Add debug logging and tracing, change some logic to avoid retry loops on deletes

[IfSentient]

Related to #559, added debug logging and tracing similar to what OpinionatedWatcher already has. While testing, changed the logic in the OpinionatedReconciler to resolve two issues:

1. Previously, when an update added the deletionTimestamp, the OpinionatedReconciler would remove the finalizer and return, and then rely on the subsequent delete event to be propagated down to the downstream reconciler for delete behavior. This introduces an edge case that is addressed in OpinionatedWatcher but missed here, where, if the downstream's processing of the delete fails, the object is still deleted in the API server, possibly leaving a bad state (in OpinionatedWatcher, the finalizer is not removed until the downstream delete is successfully processed). The logic has now been changed to mimic that of the OpinionatedWatcher, sending the "add deletionTimestamp" event to the downstream reconciler as a delete event, and only removing the finalizer once this is successful. As a side effect, actual delete events are no long propagated to the downstream.
2. After the final finalizer is removed from an object with a non-nil deletionTimestamp, a delete event is emitted in the watch stream which still contains the last finalizer to be removed (if there were multiple finalizers, only the last one is present in the list, i.e. the last state the object was in before the finalizers list became empty), rather than an empty finalizers list. To combat having this make us attempt to then remove the finalizer again (because the object is actually deleted now and this will result in an error which will be retried by default), the swallowing of delete events has been moved to before any of the other deletionTimestamp checks.

Example of deletion finalizers list behavior from (2):

% kubectl get --raw '/apis/issuetrackerproject.ext.grafana.com/v1/namespaces/default/issues?watch=1'
{"type":"ADDED","object":{"apiVersion":"issuetrackerproject.ext.grafana.com/v1","kind":"Issue","metadata":{"creationTimestamp":"2025-01-02T17:46:29Z","generation":1,...trimmed...}}}
{"type":"MODIFIED","object":{"apiVersion":"issuetrackerproject.ext.grafana.com/v1","kind":"Issue","metadata":{"creationTimestamp":"2025-01-02T17:46:29Z","finalizers":["issue-tracker-project-issues-finalizer"],"generation":1,...trimmed...}}}
{"type":"MODIFIED","object":{"apiVersion":"issuetrackerproject.ext.grafana.com/v1","kind":"Issue","metadata":{"creationTimestamp":"2025-01-02T17:46:29Z","finalizers":["issue-tracker-project-issues-finalizer","foo"],"generation":1,...trimmed...}}}
{"type":"MODIFIED","object":{"apiVersion":"issuetrackerproject.ext.grafana.com/v1","kind":"Issue","metadata":{"creationTimestamp":"2025-01-02T17:46:29Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2025-01-02T17:47:14Z","finalizers":["issue-tracker-project-issues-finalizer","foo"],"generation":2,...trimmed...}}}
{"type":"MODIFIED","object":{"apiVersion":"issuetrackerproject.ext.grafana.com/v1","kind":"Issue","metadata":{"creationTimestamp":"2025-01-02T17:46:29Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2025-01-02T17:47:14Z","finalizers":["foo"],"generation":2,...trimmed...}}}
{"type":"DELETED","object":{"apiVersion":"issuetrackerproject.ext.grafana.com/v1","kind":"Issue","metadata":{"creationTimestamp":"2025-01-02T17:46:29Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2025-01-02T17:47:14Z","finalizers":["foo"],"generation":2,...trimmed...}}}

Here a new issue is created (from the tutorial project), the reconciler adds its finalizer, then an extra foo finalizer is added. The object is deleted, and the reconciler automatically removes its finalizer. The foo finalizer is removed, but the state of the object doesn't change and the delete happens with that state.

[charandas]

As a side effect, actual delete events are no long propagated to the downstream.

This should be ok, right - since downstream is the one effectively conducting the delete operation and would be synchronously informed of that operation?

[IfSentient]

As a side effect, actual delete events are no long propagated to the downstream.

This should be ok, right - since downstream is the one effectively conducting the delete operation and would be synchronously informed of that operation?

Yup, this is just a note on the behavior--we translate the "update" event that adds the deletionTimestamp into a downstream "delete" event, and swallow the real "delete" event, as the downstream reconciler has already processed our fake "delete" in order for us to remove the finalizer (and trigger the real "delete").