[Bug]: TestDefaultOptimizingService.testPollTaskThreeTimes can fail after optimizer heartbeat expiration

[johntomcat7408-cmyk]

What happened?

TestDefaultOptimizingService.testPollTaskThreeTimes failed intermittently in the Hadoop 3 / JDK 17 / Spark 3.5 CI job while validating PR #4284.

The test expects the final pollTask call to return null, but it returned the same OptimizingTask after the optimizer was considered expired and the task was put back into the retry queue.

This failure is unrelated to the staged table creation changes in PR #4284. The REST catalog tests changed by that PR passed in the same job, and the parent master commit passed the equivalent JDK 17 job.

Expected behavior: tests that do not exercise optimizer expiration should keep the test optimizer alive deterministically and should not fail because of CI scheduling delays.

Affects Versions

master at/after b7bf0d39d; observed while testing PR #4284 at 3844f9ed73f2c3a33802fefe5e6559e806bd6efd.

What table formats are you seeing the problem on?

Iceberg test fixture; the failure is in AMS optimizer test infrastructure rather than format-specific behavior.

What engines are you seeing the problem on?

AMS / Optimizer

How to reproduce

The failure is intermittent and currently observed once in the full Hadoop 3 CI matrix:

1. Use JDK 17 and Spark 3.5.

2. Run the Hadoop 3 build/test workflow, or repeatedly run:

   ./mvnw -pl amoro-ams -am -Dtest=TestDefaultOptimizingService#testPollTaskThreeTimes test

3. Under sufficient CI scheduling or lock contention, the background Toucher may not complete a heartbeat before the 800 ms test timeout.

4. The optimizer keeper expires the optimizer and requeues its task; the assertion at line 236 then receives a task instead of null.

Failed job: https://github.com/apache/amoro/actions/runs/30520803888/job/90800567957

Successful equivalent job on the parent master commit: https://github.com/apache/amoro/actions/runs/30519308937

Relevant log output

[ERROR] TestDefaultOptimizingService.testPollTaskThreeTimes
expected: <null> but was: <OptimizingTask(...)>
at org.apache.amoro.server.TestDefaultOptimizingService.testPollTaskThreeTimes(TestDefaultOptimizingService.java:236)

INFO DefaultOptimizingService: Optimizer ... has been expired, unregister it
INFO DefaultOptimizingService: Task ... is suspending, since its optimizer is expired, put it to retry queue

Analysis

The test setup currently uses:

• OPTIMIZER_HB_TIMEOUT = 800 ms in AMSServiceTestBase.
• A background Toucher that sleeps 300 ms between heartbeat attempts.

That margin is vulnerable to CI thread scheduling and database/lock delays. Once the heartbeat misses the 800 ms window, the production retry-on-expiration behavior is valid, but it invalidates this test's assumption that the optimizer remains alive throughout the final blocking poll.

This issue tracks stabilization of the test fixture or timing control; it does not assert a production regression yet. A rerun of the failed CI job is requested to determine whether the failure reproduces.

Are you willing to submit a PR?

Yes, after confirming the failure or agreeing on the preferred deterministic test approach.

Code of Conduct

I agree to follow this project's Code of Conduct.

[johntomcat7408-cmyk]

The Hadoop 3 rerun (attempt 2) passed on all four matrix jobs, including JDK 17 / Spark 3.5, where the original failure occurred:

• JDK 11 / Spark 3.3: passed
• JDK 11 / Spark 3.4: passed
• JDK 11 / Spark 3.5: passed
• JDK 17 / Spark 3.5: passed

Rerun: https://github.com/apache/amoro/actions/runs/30520803888/attempts/2

This confirms that the observed failure is intermittent rather than a deterministic regression in PR #4284. The production behavior that expires an optimizer and requeues its task is expected; this issue should remain open to track stabilization of the test heartbeat/timing setup.

The preferred follow-up is a separate change based on the latest master: give tests unrelated to expiration a wider heartbeat margin, isolate expiration tests with their own short timeout, and replace fixed sleeps with condition-based waiting. This issue should not block PR #4284.