dt/ct/l0_gc: Fix flaky node-failure phase of resilience test#31042
Open
oleiman wants to merge 1 commit into
Open
dt/ct/l0_gc: Fix flaky node-failure phase of resilience test#31042oleiman wants to merge 1 commit into
oleiman wants to merge 1 commit into
Conversation
oleiman
force-pushed
the
ci/core-16396/l0-gc-resilience
branch
from
2fe96c9 to
127546c
Compare
Member
Author
|
/ci-repeat 2 |
Member
Author
|
/ci-repeat 2 |
Collaborator
CI test resultstest results on build#86838
|
The node-failure phase asserted the surviving nodes keep deleting L0 objects while a node is down. But GC gates on cluster health via the safety monitor, which pauses collection whenever a node is down, so the old check only passed when it caught in-flight deletes in the brief window before the block engaged, and timed out otherwise. Assert the real contract instead: GC pauses while the node is down (safety_blocked_rounds climbs on the survivors), then resumes once the node is back and the cluster is healthy again (cluster-wide objects_deleted advances past a post-restart baseline). Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
oleiman
force-pushed
the
ci/core-16396/l0-gc-resilience
branch
from
127546c to
406e6cb
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the cloud-topics L0 GC resilience test to assert the correct behavior during node failures: GC should pause while the cluster is unhealthy (per safety monitor gating) and resume once the cluster returns to health, avoiding a racey/flaky assertion based on in-flight deletes.
Changes:
- Add a helper to read the
vectorized_cloud_topics_l0_gc_safety_blocked_rounds_totalmetric. - Update the node-failure phase of the resilience test to assert safety-blocked rounds advance while a node is down.
- Update the recovery phase to assert cluster-wide
objects_deletedadvances after restart (avoiding reliance on the restarted node’s reset counters).
wdberkeley
reviewed
Jul 8, 2026
| self.redpanda.start_node(kill_node, timeout=30, node_id_override=kill_node_id) | ||
|
|
||
| self.logger.info("Waiting for restarted node to resume GC") | ||
| deleted_after_restart = self.get_num_objects_deleted() |
Contributor
There was a problem hiding this comment.
I think there's still a tiny race window here where we could GC everything left to be GC'd between node restart and here, and then the wait_until will time out.
Capturing the number of objects deleted from a live node before restarting the dead one should be reliable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The node-failure phase asserted the surviving nodes keep deleting L0 objects while a node is down. But GC gates on cluster health via the safety monitor, which pauses collection whenever a node is down, so the old check only passed when it caught in-flight deletes in the brief window before the block engaged, and timed out otherwise.
Assert the real contract instead: GC pauses while the node is down (safety_blocked_rounds climbs on the survivors), then resumes once the node is back and the cluster is healthy again (cluster-wide objects_deleted advances past a post-restart baseline).
Backports Required
Release Notes