You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found while triaging a CI failure on #2373 (unrelated to that PR's actual change — a NodeAgent DaemonSet reconcile fix).
What happened
ci/prow/4.23-e2e-test-aws failed on the "MySQL application two Vol CSI" case:
Attempt 1 actual error at backup_restore_suite_test.go:288: "Error getting pod for the proxy command: no Pod found"
Backup and restore both completed successfully (36 items restored, both EBS CSI snapshots restored fine). The app pod was confirmed running (MariaDB ready, API server up) seconds before the check ran. The failure was purely in post-restore route-reachability verification.
Root cause
tests/e2e/backup_restore_suite_test.go calls lib.VerifyBackupRestoreData(...) (tests/e2e/lib/apps.go:446), which tries GetRouteEndpointURL first, and — if that fails — falls back to finding a pod via GetFirstPodByLabel(kubeClient, namespace, "curl-tool=true") (tests/e2e/lib/apps.go:536) to proxy the request instead. Neither path retries:
GetRouteEndpointURL failing (route not yet propagated by the OpenShift router) is treated as immediately fatal for that path rather than tried again after a short wait.
The curl-tool proxy-pod fallback returns "Error getting pod for the proxy command: no Pod found" (tests/e2e/lib/apps.go:538) as soon as GetFirstPodByLabel doesn't find it once — no wait for that pod to become schedulable/ready either.
Since neither the primary route check nor the fallback proxy-pod lookup has any retry/backoff, a transient window where the route hasn't propagated yet (a normal, expected delay after a restore) fails the whole verification outright.
Suggested fix
Wrap the route-reachability check (and/or the proxy-pod lookup fallback) in a gomega.Eventually(...) with a reasonable timeout (a few minutes) and poll interval, matching the pattern already used elsewhere in this test suite (e.g. gomega.Eventually(lib.IsNamespaceDeleted(...), time.Minute*5, time.Second*5) in the same file).
Found while triaging a CI failure on #2373 (unrelated to that PR's actual change — a NodeAgent DaemonSet reconcile fix).
What happened
ci/prow/4.23-e2e-test-awsfailed on the "MySQL application two Vol CSI" case:Backup and restore both completed successfully (36 items restored, both EBS CSI snapshots restored fine). The app pod was confirmed running (MariaDB ready, API server up) seconds before the check ran. The failure was purely in post-restore route-reachability verification.
Root cause
tests/e2e/backup_restore_suite_test.gocallslib.VerifyBackupRestoreData(...)(tests/e2e/lib/apps.go:446), which triesGetRouteEndpointURLfirst, and — if that fails — falls back to finding a pod viaGetFirstPodByLabel(kubeClient, namespace, "curl-tool=true")(tests/e2e/lib/apps.go:536) to proxy the request instead. Neither path retries:GetRouteEndpointURLfailing (route not yet propagated by the OpenShift router) is treated as immediately fatal for that path rather than tried again after a short wait."Error getting pod for the proxy command: no Pod found"(tests/e2e/lib/apps.go:538) as soon asGetFirstPodByLabeldoesn't find it once — no wait for that pod to become schedulable/ready either.Since neither the primary route check nor the fallback proxy-pod lookup has any retry/backoff, a transient window where the route hasn't propagated yet (a normal, expected delay after a restore) fails the whole verification outright.
Suggested fix
Wrap the route-reachability check (and/or the proxy-pod lookup fallback) in a
gomega.Eventually(...)with a reasonable timeout (a few minutes) and poll interval, matching the pattern already used elsewhere in this test suite (e.g.gomega.Eventually(lib.IsNamespaceDeleted(...), time.Minute*5, time.Second*5)in the same file).Note
Responses generated with Claude