fix: handle stale BeforeHookCreation resources as progressing

[nazarewk]

TLDR; DELETE event during handling of BeforeHookCreation deletion policy can be delayed while etcd is running garbage collection:
1. at the same time "finished" hook is present in the cache,
2. gitops-engine does not verify creation timestamp of BeforeHookCreation objects against start time of sync operation,
3. gitops-engine is unaware (old) object should be purged from cache and happily proceeds to next Sync Wave

The PR fixes this specific failure mode with minimal changes to the code.

The change is impossible to reproduce as etcd's GC usually does not take long enough to trigger the issue. At the same time it is very difficult to debug.

the EKS cluster I observed issues at had tens of thousands objects with >1k objects churned every few minutes and still GC triggered the issue only few times a week

fixes #446
closes argoproj/argo-cd#10579
original issue argoproj/argo-cd#10077

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.59%. Comparing base (517c1ff) to head (e4d98aa).
⚠️ Report is 117 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #461      +/-   ##
==========================================
+ Coverage   55.38%   55.59%   +0.21%     
==========================================
  Files          41       41              
  Lines        4478     4493      +15     
==========================================
+ Hits         2480     2498      +18     
+ Misses       1807     1804       -3     
  Partials      191      191              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

[nazarewk]

managed to fully test expected flow, PR is ready for review

[leoluz]

Thanks a lot @nazarewk for this work!

Please make sure to create a PR in Argo CD repo pointing to this change so we can validate that all integration and e2e tests are passing. It would be great to make sure that there is an existing integration or e2e test that validates this scenario. If such test is not yet implemented it would be awesome to add it in Argo CD test suite.

[ppapapetrou76]

The gitops-engine repository is migrating to https://github.com/argoproj/argo-cd. We are closing all draft PRs.
Please feel free to submit it again once the migration is over.