Skip to content

HDDS-16120. A single MiniOzoneCluster build timeout cascades into whole-suite failures in MiniOzoneClusterProvider - #10989

Open
echonesis wants to merge 1 commit into
apache:masterfrom
echonesis:HDDS-16120
Open

HDDS-16120. A single MiniOzoneCluster build timeout cascades into whole-suite failures in MiniOzoneClusterProvider#10989
echonesis wants to merge 1 commit into
apache:masterfrom
echonesis:HDDS-16120

Conversation

@echonesis

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

MiniOzoneClusterProvider creates clusters on a background thread and passes them to consumers through a bounded blocking queue. Previously, an IOException or TimeoutException during cluster creation terminated the background thread. The queue was then never refilled, causing every subsequent provide() call to wait for 100 seconds and fail with the misleading Failed to obtain available cluster in time message.

This change passes both successful clusters and build failures through the existing bounded queue:

  • A failed build is surfaced immediately from the affected provide() call, with the original exception preserved as its cause.
  • The background create thread continues building clusters after a failure.
  • A partially created cluster is shut down before the failure is published.
  • Subsequent provide() calls can receive successfully created clusters.
  • Failed builds do not count toward the configured cluster limit.

The queue capacity remains one and put() is blocking, providing natural backpressure when builds fail persistently. This prevents the create thread from processing failures in an unbounded tight loop without introducing a retry limit, backoff duration, or polling interval.

The change limits the blast radius of an individual cluster startup failure. It does not attempt to address the underlying cause of slow cluster startup.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16120

How was this patch tested?

Local Test

mvn -pl :ozone-mini-cluster test \
    -Dtest=TestMiniOzoneClusterProvider \
    -DskipShade -DskipRecon -DskipDocs

GitHub Actions CI: https://github.com/echonesis/ozone/actions/runs/31478614848

Generated-by: Codex (GPT-5)

…le-suite failures in MiniOzoneClusterProvider
@echonesis
echonesis marked this pull request as ready for review August 12, 2026 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant