HADOOP-19472: [ABFS] Improve write workload performance for ABFS by efficient concurrency utilization #7669
+1,359
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
============================================================
|
|
💔 -1 overall
This message was automatically generated. |
anmolanmol1234
changed the title
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
manika137
reviewed
Sep 1, 2025
| LOG.error("Error freeing leases", e); | ||
| } finally { | ||
| IOUtils.cleanupWithLogger(LOG, getClient()); | ||
| IOUtils.cleanupWithLogger(LOG, poolSizeManager, getClient()); |
There was a problem hiding this comment.
For the non-dynamic pool- how are we closing the boundedThreadPool?
Would we need HadoopExecutors.shutdown(..) for it?
manika137
reviewed
Sep 1, 2025
| this.maxThreadPoolSize = Math.max(computedMaxPoolSize, initialPoolSize); | ||
|
|
||
| /* Initialize the bounded thread pool executor */ | ||
| this.boundedThreadPool = Executors.newFixedThreadPool(initialPoolSize); |
There was a problem hiding this comment.
we're naming the threads in non-dynamic pool and the manager pool for dynamic write pool. Should we also name the threads for dynamic case?
|
🎊 +1 overall
This message was automatically generated. |
manika137
reviewed
Sep 2, 2025
| com.sun.management.OperatingSystemMXBean sunOsBean | ||
| = (com.sun.management.OperatingSystemMXBean) osBean; | ||
| double cpuLoad = sunOsBean.getSystemCpuLoad(); | ||
| if (cpuLoad >= 0) { |
There was a problem hiding this comment.
if cpuLoad is -1.0, should we log it?
manika137
reviewed
Sep 2, 2025
| FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, DefaultValue = DEFAULT_ENABLE_ABFS_CHECKSUM_VALIDATION) | ||
| private boolean isChecksumValidationEnabled; | ||
|
|
||
| @BooleanConfigurationValidatorAnnotation(ConfigurationKey = |
There was a problem hiding this comment.
Nit: we can remove it and related ones below (part of prev PR)
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
============================================================
|
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhance the performance of ABFS Driver for write-heavy workloads by improving concurrency within writes.
The proposed design advocates for a centralized
WriteThreadPoolSizeManagerclass to handle the collective thread allocation required for all write operations across the system, replacing the current CachedThreadPool in AzureBlobFileSystemStore. This centralized approach ensures that the initial thread pool size is set at4 * number of available processorsand dynamically adjusts the pool size based on the system's current CPU utilization. This adaptive scaling and descaling mechanism optimizes resource usage and responsiveness. Moreover, this shared thread pool is accessible and utilized by all output streams, streamlining resource management and promoting efficient concurrency across write operations.