e2e CLI: post-completion backup status re-check has no retry, flakes on transient kubectl/API blips

[kaovilai]

Found while triaging a CI failure on #2374 (unrelated to that PR's actual change — a NodeAgent DaemonSet reconcile fix).

What happened

ci/prow/4.22-e2e-test-cli-aws failed on the "MySQL application CSI via CLI" case:

[FAILED] Unexpected error: failed to get backup status via CLI: exit status 1

at backup_restore_cli_suite_test.go:56.

The backup itself fully succeeded before this point — Phase: Completed, 45/45 items, CSI EBS snapshot Result: succeeded. The failure happened afterward, in a status re-check that has no retry of its own.

Root cause

tests/e2e/backup_restore_cli_suite_test.go checks backup success like this:

succeeded, err := lib.IsBackupCompletedSuccessfullyViaCLI(backupName)
gomega.Expect(err).ToNot(gomega.HaveOccurred())
gomega.Expect(succeeded).To(gomega.Equal(true))

IsBackupCompletedSuccessfullyViaCLI (tests/e2e/lib/backup_cli.go:146) makes a single kubectl oadp backup get -o yaml call under the hood, with no retry — a plain gomega.Expect(err).ToNot(gomega.HaveOccurred()) fails the whole test on any transient kubectl/API-server blip, even though the backup it's checking already succeeded. Contrast with the backup-wait step earlier in the same function, which correctly uses gomega.Eventually(lib.IsBackupDoneViaCLI(backupName), brCase.BackupTimeout, time.Second*10).Should(gomega.BeTrue()) — this later re-check just doesn't have the same treatment.

Suggested fix

Wrap the IsBackupCompletedSuccessfullyViaCLI call (and its restore-side equivalent, if one exists) in a short gomega.Eventually(...) — a handful of retries over ~30-60s should be enough to absorb a one-off kubectl/API blip without masking a genuine failure.

Note

Responses generated with Claude