Bump minimum Java version to 17 and remove pre-17 workarounds#3579
Open
iemejia wants to merge 2 commits into
Open
Bump minimum Java version to 17 and remove pre-17 workarounds#3579iemejia wants to merge 2 commits into
iemejia wants to merge 2 commits into
Conversation
… 21/25) Hadoop upgrade (3.3.0 -> 3.4.3): - Hadoop 3.3.x uses javax.security.auth.Subject.getSubject() which was removed in JDK 23+ (JEP 471). Hadoop 3.4.x uses Subject.current() instead, restoring JDK 25 compatibility. Spotless upgrade (2.46.1 -> 3.5.1): - Spotless 2.46.1 calls com.sun.tools.javac.util.Log methods that were removed in JDK 25, causing NoSuchMethodError at format time. Spotless 3.5.1 is compatible with JDK 25. The minor formatting changes to switch/case comment indentation are from the new version. Fix ByteBuffer leak in vectored I/O reads: - Hadoop's readVectored() API accepts an IntFunction<ByteBuffer> for allocation but has no corresponding release callback. When a wrapping filesystem like ChecksumFileSystem is in the path, Hadoop allocates a buffer through the caller's allocator, uses it internally for checksum verification, then creates a different buffer for the CompletableFuture result. The originally allocated buffer is abandoned without release. - This caused TrackingByteBufferAllocator (used in tests) to throw LeakedByteBufferException for tests using vectored I/O: TestRecordLevelFilters, TestColumnIndexFiltering, TestParquetReader. - Fix: wrap the allocator in a capturing decorator that tracks every buffer allocated during readVectored(), then registers them all for release via ByteBufferReleaser. A try-finally ensures buffers are registered even if a read future times out or fails.
iemejia
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the build forward-compatible with JDK 21+ (tested up to JDK 25).
Hadoop upgrade (3.3.0 -> 3.4.3)
Hadoop 3.3.x uses
javax.security.auth.Subject.getSubject()which was removed in JDK 23+ (JEP 471). Hadoop 3.4.x usesSubject.current()instead, restoring compatibility.Spotless upgrade (2.46.1 -> 3.5.1)
Spotless 2.46.1 calls
com.sun.tools.javac.util.Logmethods that were removed in JDK 25, causingNoSuchMethodErrorduring formatting. Spotless 3.5.1 is compatible. The minor whitespace changes to switch/case comment indentation are from the new formatter version.Fix ByteBuffer leak in vectored I/O reads
Hadoop's
readVectored()API accepts anIntFunction<ByteBuffer>for allocation but has no corresponding release callback. WhenChecksumFileSystemis in the path (the default forLocalFileSystem), Hadoop allocates buffers through the caller's allocator for internal checksum verification, then abandons them without release. The caller never sees these buffers -- only sliced views of the verified data are returned through the futures.This caused
TrackingByteBufferAllocator(used in tests) to throwLeakedByteBufferExceptionfor 45+ tests using the vectored I/O path:TestRecordLevelFilters(15 tests)TestColumnIndexFiltering(24 tests)TestParquetReader(6+ tests)Fix: Wrap the allocator in a capturing decorator that tracks every buffer allocated during
readVectored(), then registers them all for release viaByteBufferReleaser. Atry-finallyensures buffers are registered even if a read future times out or fails.This is a workaround for HADOOP-19901, which tracks the upstream bug.
Testing