Fix Job arrays exceeding queueSize (#5920) #6345
Merged
+137
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Problem Job arrays bypassed executor.queueSize limits, causing resource overallocation. When `process.array = 30` and `executor.queueSize = 40`, up to 70 concurrent tasks could run (40 regular + 30 array tasks) instead of the intended 40. ## Solution - Modified TaskPollingMonitor.canSubmit() to account for array size in capacity calculations - Job arrays now consume slots equal to their array size (not just 1) - Added validation that throws IllegalArgumentException if array size exceeds queue capacity - Extracted complex logic into checkQueueCapacity() helper method to maintain clean ternary structure ## Changes - **TaskPollingMonitor.groovy**: Added getTaskSlots() and checkQueueCapacity() helper methods - **TaskPollingMonitorTest.groovy**: Added comprehensive parameterized tests using Spock @unroll - **ParallelPollingMonitorTest.groovy**: Added inheritance test to verify validation works in subclass ## Test Coverage - Exception validation for oversized arrays (3 test cases) - Queue capacity accounting with various scenarios (9 test cases) - Edge cases: unlimited capacity, process constraints, task readiness - Inheritance verification for ParallelPollingMonitor 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
pditommaso
added a commit
that referenced
this pull request
Aug 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes issue where job arrays bypassed
executor.queueSizelimits, causing resource overallocation. Previously, a job array would count as only 1 task against the queue limit regardless of its actual array size.Before:
process.array = 30+executor.queueSize = 40→ up to 70 concurrent tasksAfter:
process.array = 30+executor.queueSize = 40→ max 40 concurrent tasks (respects queue limit)Root Cause
The original
canSubmit()method only checkedrunningQueue.size() < capacity, treating job arrays as single tasks. This allowed arrays to exceed intended resource limits.Solution
Core Changes
IllegalArgumentExceptionif array size exceeds total queue capacitycheckQueueCapacity()helper while preserving original ternary structureImplementation Details
The
checkQueueCapacity()method:Test Coverage
Added comprehensive tests using Spock's parameterized testing with 12 new test cases covering:
Backward Compatibility
✅ Fully backward compatible - no breaking changes:
executor.queueSize = 0) works as beforeRelated