TestDiscoveryWatchlabels/Discovery/TestDiscoverySpecPreserved

[jenshu]

Which tests failed?

*** note ***
transferred from kgateway back to gloo because it's not relevant in kgateway
copied over the issue description, but not all the comments. see kgateway-dev#10250 for comment history

Which tests failed?

TestDiscoveryWatchlabels/Discovery/TestDiscoverySpecPreserved

Logs:

=== RUN   TestDiscoveryWatchlabels/Discovery/TestDiscoverySpecPreserved
service/example-svc created
=== NAME  TestDiscoveryWatchlabels
    status.go:71: 
        Timed out after 20.001s.
        The function passed to Eventually failed at /home/runner/work/gloo/gloo/test/kubernetes/testutils/assertions/status.go:70 with:
        have matcher for namespace discovery-watchlabels-test which is not found
service "example-svc" deleted

Initial Investigation

This has been reproducible locally, though infrequently, whereas it seems to occur much more frequently in CI

This flake occurs when an Upstream created by Discovery does not have its status populated

The failure occurs here, on the assertion that eventually an Upstream, which we expect Discovery to create, has an "Accepted" namespaced status

When debugging I found that occasionally the Upstream is created and never gets a status, which leads that assertion to fail

I'm not sure what would cause an Upstream not to get a status, but it may be notable that this failure seems to exclusively occur on the second test, TestDiscoverySpecPreserved, and not TestDiscoverUpstreamMatchingWatchLabels, suggesting that there may be some form of test pollution

Additional Information

One example of a failure on nightlies

┆Issue is synchronized with this Asana task by Unito

Initial Investigation

No response

Additional Information

No response