Merge pull request #20 from CCBR/iss-18

Adds changelog, fixes bug with python/ray, fixes prepare output dir, adds bash script to submit to SLURM
CCBR · Feb 16, 2024 · 9231f7a · 9231f7a
2 parents 8aef060 + f29b6a7
commit 9231f7a
Show file tree

Hide file tree

Showing 42 changed files with 316 additions and 37 deletions.
diff --git a/.gitignore b/.gitignore
diff --git a/.pre-commit-config.yaml.yml b/.pre-commit-config.yaml.yml
@@ -0,0 +1,48 @@
+default_install_hook_types: [pre-commit, commit-msg]
+default_stages: [pre-commit]
+exclude: |
+  (?x)(
+      ^assets/|
+      ^docs/.*.html|
+      ^_extensions/
+   )
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v1.2.3
+    hooks:
+      - id: check-added-large-files
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+      - id: check-json
+  # spell check
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.2.4
+    hooks:
+      - id: codespell
+        # https://github.com/codespell-project/codespell/issues/1498
+  # Python formatting
+  - repo: https://github.com/psf/black
+    rev: 23.7.0
+    hooks:
+      - id: black
+  # R formatting
+  - repo: https://github.com/lorenzwalthert/precommit
+    rev: v0.1.2
+    hooks:
+      - id: style-files
+  # general linting
+  - repo: https://github.com/pre-commit/mirrors-prettier
+    rev: v2.7.1
+    hooks:
+      - id: prettier
+  # enforce commit format
+  - repo: https://github.com/compilerla/conventional-pre-commit
+    rev: v2.3.0
+    hooks:
+      - id: conventional-pre-commit
+        stages: [commit-msg]
+        args: []
+  - repo: https://github.com/citation-file-format/cffconvert
+    rev: 054bda51dbe278b3e86f27c890e3f3ac877d616c
+    hooks:
+      - id: validate-cff
diff --git a/.prettierignore b/.prettierignore
@@ -0,0 +1,8 @@
+# gitignore
+.nextflow*
+work/
+data/
+results/
+.DS_Store
+*.code-workspace
+assets/*.html
diff --git a/.prettierrc b/.prettierrc
@@ -0,0 +1,7 @@
+overrides:
+  - files:
+      - "*.md"
+      - "*.cff"
+      - ".prettierrc"
+    options:
+      tabWidth: 2
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,7 @@
+# dev
+
+# version v2.0
+- add CHANGELOG (iss#19, @slsevilla)
+- removes RAY from running (iss#17, @slsevilla)
+- adds metro_script, as temp workflow manager (iss#18, @slsevilla)
+- fixes prepare outdir bug (iss#15, @slsevilla)
diff --git a/CITATION.cff b/CITATION.cff
@@ -32,5 +32,5 @@ identifiers:
 - description: Archived snapshots of all versions
   type: doi
   value: 10.5281/zenodo.6643611
-version: 1.3
+version: 2.0
 date-released: 2022-06-14
diff --git a/LICENSE b/LICENSE
diff --git a/README.md b/README.md
diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-v1.2.1
+v2.0
diff --git a/data/BGL1_FpIhmvaf5_Variant.xlsx b/data/BGL1_FpIhmvaf5_Variant.xlsx
diff --git a/data/BGL1_Input_VRTs_forAASP.xlsx b/data/BGL1_Input_VRTs_forAASP.xlsx
diff --git a/data/METRO_logo.png b/data/METRO_logo.png
diff --git a/data/SB28_FpIhmvaf5_Variant.xlsx b/data/SB28_FpIhmvaf5_Variant.xlsx
diff --git a/data/SB28_Input_VRTs_forAASP.xlsx b/data/SB28_Input_VRTs_forAASP.xlsx
diff --git a/data/test_VAF20_Variant.csv b/data/test_VAF20_Variant.csv
diff --git a/data/test_VAF20_Variant.metro.tsv b/data/test_VAF20_Variant.metro.tsv
diff --git a/data/test_output_netmhc_final.tsv b/data/test_output_netmhc_final.tsv
diff --git a/data/test_prepare_input1.maf b/data/test_prepare_input1.maf
diff --git a/data/test_prepare_input2.maf b/data/test_prepare_input2.maf
diff --git a/data/test_prepare_input3.maf b/data/test_prepare_input3.maf
diff --git a/docs/METRO/build.md b/docs/METRO/build.md
@@ -1,4 +1,4 @@
-# 2. Build Synposis
+# 3. Build Synposis
 
 The `./metro` executable is composed of several inter-related sub commands. Please see `./metro -h` for all available options. The synopsis for the sub command `build` shows its parameters and their usage. Optional parameters are shown in square brackets.
 
@@ -10,7 +10,7 @@ $ ./metro build [-h] --ref-fa REF_FA \
 
 This part of the documentation describes options and concepts for the `./metro build` sub command in more detail. With minimal configuration, the build sub command enables you to build reference file. In its most basic form, `./metro build` only has _three required inputs_.
 
-## 2.1. Required Arguments
+## 3.1. Required Arguments
 
 Each of the following arguments are required. Failure to provide a required argument will result in a non-zero exit-code.
 
@@ -41,7 +41,7 @@ Each of the following arguments are required. Failure to provide a required argu
 > ***Example:*** 
 > `--output /scratch/$USER/refs/hg38_v36/`
 
-## 2.2 Optional Arguments
+## 3.3 Optional Arguments
 Each of the following arguments are optional and do not need to be provided. 
 
 `-h, --help`            
@@ -53,18 +53,18 @@ Each of the following arguments are optional and do not need to be provided.
 > ***Example:*** 
 > `--help`
 
-## 2.3 Example
+## 3.3 Example
 Build reference files for the run sub comamnd. Follow the setup in [Getting Started](https://ccbr.github.io/METRO/METRO/getting-started/) before this step.
 
 ```bash
 # login and load interactive session, as described in Getting Started
 
 # download reference files, as needed
-wget -P /output/dir/ http://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_mouse/release_M29/GRCm39.primary_assembly.genome.fa
+wget -P /output/dir/ http://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_mouse/release_M39/GRCm39.primary_assembly.genome.fa
 gunzip /scratch/$USER/METRO/refs/GRCm39.primary_assembly.genome.fa.gz
 
-wget -P /output/dir/ http://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_mouse/release_M29/gencode.vM26.annotation.gtf
-gunzip /scratch/$USER/METRO/refs/gencode.vM26.annotation.gtf.gz
+wget -P /output/dir/ http://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_mouse/release_M39/gencode.vM36.annotation.gtf
+gunzip /scratch/$USER/METRO/refs/gencode.vM36.annotation.gtf.gz
 
 # Build METRO reference files
 ## Github
@@ -77,6 +77,6 @@ singularity shell --bind /data/$USER docker://nciccbr/ccbr_metro_v1.4 nciccbr/cc
 ## Command
 ./metro build \
             --ref-fa /scratch/$USER/METRO/GRCm39.primary_assembly.genome.fa \
-            --ref-gtf /scratch/$USER/METRO/gencode.vM26.annotation.gtf \
+            --ref-gtf /scratch/$USER/METRO/gencode.vM36.annotation.gtf \
             --output /scratch/$USER/METRO
 ```
diff --git a/docs/METRO/citation.md b/docs/METRO/citation.md
diff --git a/docs/METRO/example.md b/docs/METRO/example.md
diff --git a/docs/METRO/find.md b/docs/METRO/find.md
@@ -1,4 +1,4 @@
-# 4. Find Synopsis
+# 5. Find Synopsis
 The `./metro` executable is composed of several inter-related sub commands. Please see `./metro -h` for all available options. The synopsis for the sub command `find` shows its parameters and their usage. Optional parameters are shown in square brackets.
 
 ```
@@ -10,7 +10,7 @@ $ ./metro find [-h] [--subset SUBSET] \
 
 This part of the documentation describes options and concepts for `./metro find` sub command in more detail. With minimal configuration, the `find` sub command enables you to start running metro pipeline. In its most basic form, `./metro find` only has _three required inputs_.
 
-## 4.1 Required Arguments
+## 5.1 Required Arguments
 Each of the following arguments are required. Failure to provide a required argument will result in a non-zero exit-code.
 
 `--input INPUT [INPUT ...]`  
@@ -40,7 +40,7 @@ Each of the following arguments are required. Failure to provide a required argu
 > ***Example:*** 
 > ` --transcripts transcripts.fa`
 
-## 4.2 Optional Arguments
+## 5.2 Optional Arguments
 Each of the following arguments are optional and do not need to be provided. 
 
 `-h, --help`            
@@ -61,15 +61,15 @@ Each of the following arguments are optional and do not need to be provided.
 > ***Example:*** 
 > `--subset 30`
 
-## 4.3 Example
+## 5.3 Example
 Find metro with the references files generated in the build example.
 
 ```bash 
 # login and load interactive session, as described in Getting Started
 
 ### Github
 module purge
-module load python/3.5
+module load python/3.8
 
 ## Docker
 singularity shell --bind /data/$USER docker://nciccbr/ccbr_metro_v1.4 nciccbr/ccbr_metro_v1.4

diff --git a/docs/METRO/getting-started.md b/docs/METRO/getting-started.md
diff --git a/docs/METRO/predict.md b/docs/METRO/predict.md
@@ -1,4 +1,4 @@
-# 5. Predict Synopsis
+# 6. Predict Synopsis
 The `./metro` executable is composed of several inter-related sub commands. Please see `./metro -h` for all available options. The synopsis for the sub command `predict` shows its parameters and their usage. Optional parameters are shown in square brackets.
 
 ```
@@ -14,7 +14,7 @@ $ ./ metro predict [-h] --mutationFile MUTATIONFILE \
 
 This part of the documentation describes options and concepts for `./metro input` sub command in more detail. With minimal configuration, the `predict` sub command enables you to generate prediction files for each mutated sequence identified in the metro `run` sub command.
 
-## 5.1 Required Arguments
+## 6.1 Required Arguments
 Each of the following arguments are required. Failure to provide a required argument will result in a non-zero exit-code.
 
 `--mutationFile MUTATIONFILE [MUTATIONFILE ...]`  
@@ -61,7 +61,7 @@ Each of the following arguments are required. Failure to provide a required argu
 > ***Example:*** 
 > `--alleleList H-2-Ld,H-2-Dd,H-2-Kb`
 
-## 5.2 Optional Arguments
+## 6.2 Optional Arguments
 Each of the following arguments are optional and do not need to be provided. Default values listed in each example will be used, if value not provided.
 
   `-h, --help`            
@@ -100,7 +100,7 @@ Each of the following arguments are optional and do not need to be provided. Def
 > ***Example:*** 
 > `--lowbind 2`
 
-## 5.3 Example
+## 6.3 Example
 Predict the binding of peptides to any MHC molecule of known sequence using artificial neural networks (ANNs) and perform filtering of output based on user-provided parameters.
 
 ```bash 

diff --git a/docs/METRO/prepare.md b/docs/METRO/prepare.md
@@ -1,4 +1,4 @@
-# 3. Prepare Synopsis
+# 4. Prepare Synopsis
 The `./metro` executable is composed of several inter-related sub commands. Please see `./metro -h` for all available options. The synopsis for the sub command `prepare` shows its parameters and their usage. Optional parameters are shown in square brackets.
 
 ```
@@ -12,7 +12,7 @@ $ ./metro prepare [-h] --mafFiles MAFFILES \
 
 This part of the documentation describes options and concepts for `./metro prepare` sub command in more detail. With minimal configuration, the `prepare` sub command enables you to create filtered MAF files for the metro `run` pipeline.
 
-## 3.1 Required Arguments
+## 4.1 Required Arguments
 Each of the following arguments are required. Failure to provide a required argument will result in a non-zero exit-code.
 `--mafFiles MAFFILES [MAFFILES ...]`  
 > **Input MAF-like file(s) to process.**  
@@ -41,7 +41,7 @@ Each of the following arguments are required. Failure to provide a required argu
 > ***Example:*** 
 > ` --outputprefix test`
 
-## 3.2 Optional Arguments
+## 4.2 Optional Arguments
 Each of the following arguments are optional and do not need to be provided. Default values listed in each example will be used, if value not provided.
 
   `-h, --help`            
@@ -80,15 +80,15 @@ Each of the following arguments are optional and do not need to be provided. Def
 > ***Example:*** 
 > `--impactFilter 2`
 
-## 3.3 Example
+## 4.3 Example
 Filter MAF files in preparation of metro run.
 
 ```bash 
 # login and load interactive session, as described in Getting Started
 
 ### Github
 module purge
-module load python/3.5
+module load python/4.5
 
 ## Docker
 singularity shell --bind /data/$USER,/usr/local/apps/cufflinks/2.2.1 docker://nciccbr/metro_v1.4:latest

diff --git a/docs/METRO/troubleshooting.md b/docs/METRO/troubleshooting.md
@@ -1 +1,2 @@
-# <center>** **COMING SOON** **</center>
+# Troubleshooting
+If you run into errors, please contact [email protected] for assistance!
diff --git a/docs/METRO/usage.md b/docs/METRO/usage.md
@@ -0,0 +1,34 @@
+# 2. Example Usage
+A workflow script is provided to allow users the ability to run locally or to submit to Biowulf SLURM.
+
+## 2.1 Runmodes
+There are several “flags” which have been created to run the commands. These are `build`, `prepare`, `find`, `predict`
+```
+bash metro_script.sh build echo
+```
+
+## 2.2 Handling Runmodes
+There are several options for handing the above runmodes. These include:
+
+1.	echo: this will print the SH file location
+2.	cat: this will print the contents of the SH file
+3.	sh: this will run the SH file locally
+4.	submit_batch: this will run the SH file on the SLURM cluster (use HPC dashboard to monitor)
+5.	submit_batch_large: this will run the SH file on the SLURM cluster with “larger” resources 
+
+## 2.3 Examples to run the workflow:
+Example printing out the workflow commands to the command line:
+```
+bash metro_script.sh build cat
+bash metro_script.sh prepare cat
+bash metro_script.sh find cat
+bash metro_script.sh predict cat
+```
+
+Example submitting to SLURM:
+```
+bash metro_script.sh build submit_batch
+bash metro_script.sh prepare submit_batch
+bash metro_script.sh find submit_batch
+bash metro_script.sh predict submit_batch_large
+```
diff --git a/docs/img/METRO_logo.svg b/docs/img/METRO_logo.svg
diff --git a/docs/img/fred_logo.png b/docs/img/fred_logo.png
diff --git a/docs/img/metro_logo.png b/docs/img/metro_logo.png
diff --git a/metro b/metro
@@ -522,7 +522,7 @@ def predict(sub_args):
     )
     print("Running: " + process)
     exitcode = bash(process)
-
+    
     # Read in output of netMHC    
     print("--Post-Processing")
     df_list=[]