HDDS-16117. Failed FSO multipart complete leaks bucket namespace quota - #10984
Draft
smengcl wants to merge 2 commits into
Draft
HDDS-16117. Failed FSO multipart complete leaks bucket namespace quota#10984smengcl wants to merge 2 commits into
smengcl wants to merge 2 commits into
Conversation
Work on a copy of the cached OmBucketInfo in S3MultipartUploadCompleteRequest.validateAndUpdateCache so that the namespace charge for recreated missing parent directories is published to the cache and DB only on the success path. On the failure path (for example INVALID_PART) the copy is discarded, so a failed complete no longer leaves the in-place usedNamespace increment orphaned in the cached bucket. This mirrors the copyObject() usage on the abort path. Add an integration test in TestOzoneClientMultipartUploadWithFSO that fails before the fix (cached usedNamespace is 1) and passes after. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a bucket namespace quota leak in Ozone Manager’s FSO multipart-upload complete path where a failed complete could mutate the cached OmBucketInfo.usedNamespace in-place (eg, on INVALID_PART) without publishing a corresponding corrective cache update.
Changes:
- In
S3MultipartUploadCompleteRequest#validateAndUpdateCache, work with acopyObject()of the cachedOmBucketInfoso quota mutations are only published viaupdateCacheon success. - Document
OMKeyRequest#getBucketInfobehavior as returning the live cached bucket instance by reference, and warn callers to copy before mutating in failure-prone paths. - Add an integration test reproducing the “delete parent dir then failed complete” scenario and asserting cached and durable
usedNamespaceremain consistent.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java | Copies cached bucket info before any quota-affecting cache work so failed completes don’t leak namespace usage into the bucket cache. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java | Clarifies that getBucketInfo returns the cached OmBucketInfo instance by reference and callers must copy before mutating prior to a possible failure. |
| hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientMultipartUploadWithFSO.java | Adds regression coverage ensuring a failed FSO MPU complete after parent deletion does not corrupt cached bucket namespace quota. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated-by: Claude Code (Opus 4.8)
What changes were proposed in this pull request?
A multipart upload complete on an FSO bucket can leak the bucket
usedNamespacevalue in the cache.The cause is as follows:
validateAndUpdateCachegets the bucket withgetBucketInfo.getBucketInforeturns the cachedOmBucketInfoby reference.addMissingParentsToCachemakes a missing parent directory again. It callsincrUsedNamespace(...)before it validates the parts. This call changes the cached bucket in place.INVALID_PART.updateCache. Thus the OM does not correct the change.usedNamespacechange stays.The result: the cached bucket shows one namespace object that does not exist. The durable bucket stays correct. A client can send this failed complete many times. This can fill the namespace quota of the bucket with no real objects. An OM restart, a failover, or a quota repair clears the cache.
HDDS-11784 corrected the abort path. It did not correct the failed complete path.
Fix: use a copy of the cached
OmBucketInfoinvalidateAndUpdateCache, and publish it withupdateCacheonly when the complete is successful. If the complete fails, the OM does not use the copy, and the cache stays correct. This is the samecopyObject()method as the abort path. A note ongetBucketInfotells callers to copy the bucket before they change it.(TLA+ formal verification with Specula found this bug. Trace validation on a real trace broke the
QuotaExactnessinvariant. An independent model check broke theFailedOperationIsolationinvariant. Both point to the failed complete path.)What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16117
How was this patch tested?
TestOzoneClientMultipartUploadWithFSO#testFailedCompleteAfterParentDeletionDoesNotLeakNamespaceQuota: start an FSO MPU below a parent directory, upload a part, delete the parent, then complete with a wrong ETag and getINVALID_PART. The cached and durable bucketusedNamespacemust both be 0. The test fails on master (cached value is 1) and passes with this patch.