Skip to content

Fix flaky tests: e2e demo suite timeouts and syncer poll timeout #799

Description

Summary

CI failure analysis (hack/metrics/ci-failure-analysis.py on branch docs/velocity-metrics of the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:

Test Failures Job Root cause
TestDurableDirLifecycle e2e-test 503 from atenet-router — xDS routes not yet propagated when actor reaches STATUS_RUNNING
TestActorLifecycle e2e-test Same: 503 after resume, callActor 30s deadline too tight
TestMultipleDurableDirLifecycle e2e-test Same
TestSyncer_UpdateWorker_RetryOnVersionConflict run-tests wait.PollUntilContextTimeout 5s deadline too tight on loaded CI runners
TestLoaderConcurrentHandshakes run-tests Concurrent symlink rotation — exact failure mode unknown, needs investigation

Root Cause

e2e demo tests (demo_test.go)

After waitForActorStatus returns STATUS_RUNNING, the atenet-router's xDS route for the actor isn't always ready immediately. callActor retries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.

TestSyncer_UpdateWorker_RetryOnVersionConflict

The test asserts via wait.PollUntilContextTimeout(..., 5*time.Second, ...). The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.

TestLoaderConcurrentHandshakes

Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.

Fix

See the linked PR. Changes:

  • internal/e2e/suites/demo/demo_test.go: callActor deadline 30s → 90s; waitForActorStatus timeout 60s → 120s
  • cmd/ateapi/internal/controlapi/syncer_test.go: final assertion poll timeout 5s → 15s

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working / bugfixes

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions