test(compute): use Awaitility for trace polling in golden signals test to prevent quota exhaustion

[blakeli0]

Fixes b/558226581

Problem

In ITComputeGoldenSignals, integration tests were failing with:

RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Read requests (free)' and limit 'Read requests (free) per minute' of service 'cloudtrace.googleapis.com'

Root Cause

1. TraceServiceClient was configured with custom RetrySettings where retry delays were initialized to 0ms backoff, while simultaneously adding StatusCode.Code.NOT_FOUND as a retryable code.
2. Because Cloud Trace spans take several seconds to be ingested and indexed, traceClient.getTrace(...) initially returned NOT_FOUND (404).
3. GAX's retry mechanism retried in a zero-delay tight loop, firing hundreds of requests in seconds and rapidly exhausting the 300 requests/minute Cloud Trace read quota.

Solution

1. Reverted TraceServiceClient to default retry settings: Uses the default GAX retry configuration (which does not retry NOT_FOUND and applies exponential backoff for transient gRPC errors like UNAVAILABLE).
2. Added Awaitility polling: Replaced the single call and retry logic with Awaitility.await(), polling every 3 seconds with a 3-second initial delay up to a 2-minute timeout, ignoring transient NOT_FOUND and RESOURCE_EXHAUSTED errors during ingestion.
3. Cleaned up dependency: Added org.awaitility:awaitility in test scope and returned Trace directly from Awaitility's until(Callable<T>, Predicate<T>).

Verification

• mvn test-compile -DskipTests in java-compute/google-cloud-compute: PASSED (0 checkstyle violations, clean compilation).

[gemini-code-assist]

Code Review

This pull request integrates the Awaitility library into the integration tests to poll Cloud Trace, replacing manual try-catch blocks and custom retry settings. Feedback points out a critical issue in the Awaitility configuration: chaining .ignoreExceptionsMatching after .ignoreExceptionsInstanceOf overwrites the previous configuration, and the client throws GAX exceptions instead of StatusRuntimeException. A combined exception-matching check is suggested to fix this.

[lqiu96]

IIUC, getTrace() should have default values configured from stubsettings (this is just removing our overrides)?

google-cloud-java/java-trace/google-cloud-trace/src/main/java/com/google/cloud/trace/v1/stub/TraceServiceStubSettings.java

Lines 351 to 360 in 4426ccd

	settings =
	RetrySettings.newBuilder()
	.setInitialRetryDelayDuration(Duration.ofMillis(100L))
	.setRetryDelayMultiplier(1.2)
	.setMaxRetryDelayDuration(Duration.ofMillis(1000L))
	.setInitialRpcTimeoutDuration(Duration.ofMillis(45000L))
	.setRpcTimeoutMultiplier(1.0)
	.setMaxRpcTimeoutDuration(Duration.ofMillis(45000L))
	.setTotalTimeoutDuration(Duration.ofMillis(45000L))
	.build();

Since we are using awaitillity, we can probably just set the default to be 1 attempt and like 5s timeout or something.

[blakeli0]

Yes this is just removing our overrides so we can rely on awaitility for the retries on server errors. The default configuration still makes sense for transient network errors and we can keep them as they are.

[lqiu96]

Reframing the question: Do we want/ need to account for transient network issues if we are polling every 3 seconds?

getTrace() is called every 3 seconds with its own retry configurations. Each invocation from awaitility theoretically can retry multiple times