HDDS-16118. Release snapshot DB handles before KeyDeletingService submits OM requests

[smengcl]

What changes were proposed in this pull request?

KeyDeletingService opens the current snapshot database and may open previous snapshot databases while determining which deleted keys and rename entries are reclaimable. Each open snapshot DB handle retains a striped SNAPSHOT_DB_LOCK read lock.

Previously, these handles remained open while KeyDeletingService submitted synchronous OM requests. This could form the following circular wait:

1. A KeyDeletingService worker retains a snapshot DB read lock while synchronously submitting an OM request.
2. Independently, OMDoubleBufferFlushThread applies an earlier snapshot-purge response that needs a colliding write lock. The KDS and purge snapshots may have different UUIDs mapped to the same lock stripe.
3. The double-buffer flush thread blocks on the read lock held by the KDS worker.
4. The number of unflushed transactions reaches ozone.om.unflushed.transaction.max.count.
5. Transaction application blocks, preventing the synchronous KDS request from completing.
6. The KDS worker cannot return and close the snapshot DB handle, completing the cycle.

This change releases all snapshot DB handles after the metadata scan and before synchronous OM requests are submitted:

• The current snapshot DB handle is closed early.
• Previous-snapshot handles opened by the reclaimable filters are closed early.
• Snapshot GC locks remain held until the corresponding OM requests complete, preserving snapshot-chain and block-reclamation correctness.
• The existing try-with-resources cleanup remains as an exception-safety fallback.

No snapshot database is accessed after the handles are released; subsequent processing uses metadata already materialized in memory.

Regression tests cover the change:

• One verifies the current snapshot DB handle is closed before KeyDeletingService submits an OM request.
• One reproduces the circular wait deterministically, using a single double-buffer permit (ozone.om.unflushed.transaction.max.count=1) and a single snapshot DB lock stripe (ozone.om.lock.stripes.snapshot_db_lock=1) so a snapshot read lock collides with a snapshot purge's write lock. It hangs without the fix and passes with it.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16118

How was this patch tested?

• Added TestKeyDeletingService regression tests, including one that reproduces the permit-exhaustion deadlock (times out without the fix, passes with it).
• Ran the full TestKeyDeletingService class and checkstyle locally; all green.

Generated-by: Codex (GPT-5)

[Copilot]

Pull request overview

This PR addresses a potential circular-wait/deadlock involving KeyDeletingService holding snapshot DB read locks while submitting synchronous OM (Ratis) requests, which can block double-buffer flushing that needs a colliding snapshot DB write lock. The change ensures snapshot DB handles (and their associated SNAPSHOT_DB_LOCK read locks) are released before synchronous OM requests are submitted, while keeping snapshot GC locks held until request completion for correctness.

Changes:

• Close snapshot DB handles (current + previous-snapshot handles opened by reclaimable filters) after the metadata scan and before submitting synchronous OM requests.
• Add ReclaimableFilter.closeSnapshotDbHandles() to allow early closure of snapshot DB handles without releasing snapshot GC locks.
• Add a regression test asserting the snapshot DB handle is closed before KeyDeletingService submits an OM request.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java	Adds a regression test that asserts the snapshot DB handle is closed before OM request submission.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableFilter.java	Introduces closeSnapshotDbHandles() to close previous snapshot DB handles while retaining snapshot GC locks.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java	Closes reclaimable-filter snapshot handles and the active snapshot handle before submitting synchronous OM requests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[yandrey321]

previously, these handles remained open while KeyDeletingService submitted synchronous OM requests. This could >form the following circular wait:

1. KeyDeletingService retains a snapshot DB read lock.
2. OMDoubleBufferFlushThread processes a snapshot purge that needs the colliding write lock.

if step 2 is synchronous call, is it possible to check if the thread has a read lock before obtaining write lock and upgrade read to write lock instead of waiting for write lock?

[smengcl]

if step 2 is synchronous call, is it possible to check if the thread has a read lock before obtaining write lock and upgrade read to write lock instead of waiting for write lock?

@yandrey321 Thanks for the suggestion. Yes the purge call is synchronous on OMDoubleBufferFlushThread, however the read lock is held by a separate KeyDeletingService worker. The KDS and purge operations may also target different snapshot UUIDs that map to the same lock stripe, so the flush thread has no read lock to upgrade.

ReentrantReadWriteLock does not support safe read-to-write upgrades either. This patch instead releases the KDS snapshot DB handles before waiting for the OM request, breaking the cross-thread cycle. I have clarified the sequence in the PR description.

[jojochuang]

lgtm.

I can't help but wonder: does DirectoryDeletingService has the same kind of bug?

[jojochuang]

the reclaimableKeyFilter is cleaned up here at the end of try-with-resources scope, releasing the snapshot lock.