Skip to content

Solve Job arrays exceeding queuesize #6314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 28, 2025

Conversation

jorgee
Copy link
Contributor

@jorgee jorgee commented Jul 28, 2025

close #5920

Modifying canSubmit to avoid to avoid exceed executor.queueSize parameter.
Printing warning when array size exceeds the queueSize. This can make the task array 'unsubmitable' (Not sure if we should abort the execution in this situation)
Add unit tests
Tested with this pipeline

process test {
    array 10 
    input:
    val x

    """
    echo $x
    """
}

workflow {
   Channel.of(1..20) | test
}

With a config with awsbatch executor and different executor.queueSize values:

size 15: Only one job submitted at a time in AWS Batch.
size 5: Prints a warning but run continues without the task array jobs submitted.

Copy link

netlify bot commented Jul 28, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit a02aa25
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6887895fd6c20400082bd38d

@bentsherman bentsherman self-requested a review July 28, 2025 14:31
@bentsherman bentsherman merged commit fcf5b87 into master Jul 28, 2025
22 checks passed
@bentsherman bentsherman deleted the 5920-job-array-with-queuesize-exceeds-queuesize branch July 28, 2025 14:38
pditommaso added a commit that referenced this pull request Aug 14, 2025
@pditommaso
Copy link
Member

I went for a simpler solution for this problem #6345

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Job Array with queueSize exceeds queueSize
3 participants