fix: handle stale BeforeHookCreation resources as progressing #461
Conversation
nazarewk
force-pushed
the
fix-before-hook-creation
branch
from
5c153c9
to
ed6a1e1
Compare
nazarewk
force-pushed
the
fix-before-hook-creation
branch
from
ed6a1e1
to
407dc83
Compare
Codecov ReportBase: 55.38% // Head: 55.59% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #461 +/- ##
==========================================
+ Coverage 55.38% 55.59% +0.21%
==========================================
Files 41 41
Lines 4478 4493 +15
==========================================
+ Hits 2480 2498 +18
+ Misses 1807 1804 -3
Partials 191 191
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
nazarewk
force-pushed
the
fix-before-hook-creation
branch
from
407dc83
to
7631b8e
Compare
- fixes argoproj#446 - closes argoproj/argo-cd#10579 - original issue argoproj/argo-cd#10077 Signed-off-by: Krzysztof Nazarewski <[email protected]>
nazarewk
force-pushed
the
fix-before-hook-creation
branch
from
7631b8e
to
327f8d9
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Signed-off-by: Krzysztof Nazarewski <[email protected]>
nazarewk
force-pushed
the
fix-before-hook-creation
branch
from
4382fe9
to
e4d98aa
Compare
|
Kudos, SonarCloud Quality Gate passed!
|
|
managed to fully test expected flow, PR is ready for review |
crenshaw-dev
changed the title
There was a problem hiding this comment.
Thanks a lot @nazarewk for this work!
Please make sure to create a PR in Argo CD repo pointing to this change so we can validate that all integration and e2e tests are passing. It would be great to make sure that there is an existing integration or e2e test that validates this scenario. If such test is not yet implemented it would be awesome to add it in Argo CD test suite.
TLDR;
DELETEevent during handling ofBeforeHookCreationdeletion policy can be delayed whileetcdis running garbage collection:1. at the same time "finished" hook is present in the cache,
2.
gitops-enginedoes not verify creation timestamp ofBeforeHookCreationobjects against start time of sync operation,3.
gitops-engineis unaware (old) object should be purged from cache and happily proceeds to next Sync WaveThe PR fixes this specific failure mode with minimal changes to the code.
The change is impossible to reproduce as
etcd's GC usually does not take long enough to trigger the issue. At the same time it is very difficult to debug.the EKS cluster I observed issues at had tens of thousands objects with >1k objects churned every few minutes and still GC triggered the issue only few times a week
fixes #446
closes argoproj/argo-cd#10579
original issue argoproj/argo-cd#10077