Add custom config for seadragon cluster of MD Anderson Cancer Center #831
Conversation
|
@nf-core-bot fix linting |
| } | ||
|
|
||
| env { | ||
| SINGULARITY_CACHEDIR="/home/$USER/.singularity/cache" |
There was a problem hiding this comment.
Have you tested this works, e.g. using --custom_config_base with your pipeline of interest?
For some config scopes Nextflow will interpret that as a Nextflow variable rather than base. I think env is OK though, but likely worth to double check.
There was a problem hiding this comment.
Hi James, thank you for the comment. I have tested the config with the RNA seq and Sarek pipeline. Neither of them reported errors here. So it should be fine.
| envWhitelist='APPTAINERENV_NXF_TASK_WORKDIR,APPTAINERENV_NXF_DEBUG,APPTAINERENV_LD_LIBRARY_PATH,SINGULARITY_BINDPATH,LD_LIBRARY_PATH,TMPDIR,SINGULARITY_TMPDIR' | ||
| autoMounts = true | ||
| runOptions = '-B ${TMPDIR:-/tmp}' | ||
| cacheDir = "/home/$USER/.singularity/cache" |
conf/seadragon.config
Outdated
| max_memory = 950.GB // Maximum memory based on E80 nodes | ||
| max_cpus = 80 // Maximum CPUs based on E80 nodes | ||
| max_time = 240.h // Maximum runtime for long queues |
There was a problem hiding this comment.
Note this would mean that you will never be able to access the high-mem nodes.
Furthermor,e you need to replicate these values in the nextflow native resourceLimits directive - e.g.
Lines 12 to 21 in 03af77c
max_* have been deprecated in more recent nf-core pipelines, but you should still keep max_ for backards compatibility with older pipelines
There was a problem hiding this comment.
uploaded the max_memory to make highmem and vhighmem nodes available
|
|
||
| ## Notes | ||
|
|
||
| - **Data Storage**: All intermediate files will be stored in the `work/` directory within the job's launch directory. These files can consume significant space, so it is recommended to delete this directory after the pipeline completes successfully. |
There was a problem hiding this comment.
You could instead add cleanup = true to the config, so all files in this directory get deleted when a run completes successfully (if it fails, the intermediate files don't get deleted)
There was a problem hiding this comment.
e.g.
Line 8 in 03af77c
There was a problem hiding this comment.
Hi James, thanks for pointing that out. The reason I keep the work folder is that I may need to use the BAM files from the Sarek pipeline for other custom analyses. Is there a more elegant way to retain the BAM files? Perhaps by custom-defining them as final outputs?
Co-authored-by: James A. Fellows Yates <[email protected]>
name: New Custom Config For Seadragon
about: A new cluster config For MD Anderson Cancer Center Seadragon Cluster
Please follow these steps before submitting your PR:
[WIP]in its titlemasterbranchSteps for adding a new config profile:
conf/directorydocs/directorynfcore_custom.configfile in the top-level directoryREADME.mdfile in the top-level directoryprofile:scope in.github/workflows/main.yml.github/CODEOWNERS(**/<custom-profile>** @<github-username>)