Skip to content

Support Singularity, Output Nextflow Reports to params.outdir #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jeffquinn-msk wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Support Singularity, Output Nextflow Reports to params.outdir #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions conf/base.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,23 @@ process {
maxRetries = 2
}

}

dag {
enabled = true
file = "${params.outdir}/pipeline_info/pipeline_dag.html"
overwrite = true
}

trace {
enabled = true
file = "${params.outdir}/pipeline_info/pipeline_trace.txt"
fields = 'task_id,name,status,exit,realtime,%cpu,rss'
overwrite = true
}

report {
enabled = true
file = "${params.outdir}/pipeline_info/pipeline_report.html"
overwrite = true
}
32 changes: 26 additions & 6 deletions nextflow.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,13 +78,18 @@ params {

}

docker {
enabled = true
fixOwnership = true
def getContainerOptions (executor) {
if (executor == 'docker') {
return '-u root:root -e USERID=$UID -e XDG_CACHE_HOME=tmp/quarto_cache_home -e XDG_DATA_HOME=tmp/quarto_data_home -e QUARTO_PRINT_STACK=true'
} else if (executor == 'singularity') {
return '--env USERID=$UID --env XDG_CACHE_HOME=tmp/quarto_cache_home --env XDG_DATA_HOME=tmp/quarto_data_home --env QUARTO_PRINT_STACK=true'
Copy link
Author

@jeffquinn-msk jeffquinn-msk Jul 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For environment variables it would probably be better to set them in the task definition itself on second thought, passing them to the container options is confusing. Nextflow by default passes all env variables from the calling process down into the container anyway, and for the hardcoded ones, you can hard code them in the process block.

} else {
return ''
}
}

process {
containerOptions = '-u root:root -e USERID=$UID -e XDG_CACHE_HOME=tmp/quarto_cache_home -e XDG_DATA_HOME=tmp/quarto_data_home -e QUARTO_PRINT_STACK=true'
containerOptions = { getContainerOptions(workflow.containerEngine) }
stageInMode = 'copy'
}

Expand All @@ -94,6 +99,7 @@ profiles {
docker {
docker.enabled = true
docker.userEmulation = true
docker.fixOwnership = true
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
Expand All @@ -108,7 +114,20 @@ profiles {
shifter.enabled = false
charliecloud.enabled = false
}

mskcc_iris {
singularity {
enabled = true
autoMounts = true
}
process {
executor = 'slurm'
queue = 'componc_cpu,componc_gpu'
}
executor {
name = 'slurm'
queueSize = 25
}
}
test { includeConfig 'conf/test.config' }

}
Expand Down Expand Up @@ -152,4 +171,5 @@ def check_max(obj, type) {
return obj
}
}
}
}