Successful run of PDA download, updated status evaluation

mwyczalkowski · mwyczalkowski · commit c72d1d7eda4c · 2020-02-24T17:00:28.000-06:00
* Run is still ongoing.  Had a fatal error reported as described in README.project,
  but does not appear this was in fact fatal to the download
diff --git a/00_start_docker.compute1.sh b/00_start_docker.compute1.sh
@@ -0,0 +1,15 @@
+# Launch docker environment before running import, so that `parallel` is available
+
+# Map home directory (containing token) and storage directory
+#VOLUME_MAPPING="/storage1/fs1/home1/Active/home/m.wyczalkowski /storage1/fs1/m.wyczalkowski"
+VOLUME_MAPPING="/home/m.wyczalkowski /storage1/fs1/m.wyczalkowski"
+
+LSF_ARGS="-G compute-lding"
+IMAGE="mwyczalkowski/cromwell-runner"
+
+>&2 echo Launching $IMAGE on compute1
+CMD="bash src/start_docker.sh -I $IMAGE -M compute1 -g \"$LSF_ARGS\" $@ $VOLUME_MAPPING"
+echo Running: $CMD
+eval $CMD
+
+
diff --git a/00_start_docker.sh b/00_start_docker.sh
diff --git a/10_get_UUID.sh b/10_get_UUID.sh
@@ -1,8 +1,8 @@
 # We want all PDA genomic RNA-Seq samples available at GDC
+source gdc-import.config.sh
 
-CATALOG="/storage1/fs1/home1/Active/home/m.wyczalkowski/Projects/CPTAC3/CPTAC3.catalog/CPTAC3.Catalog.dat"
-OUT="dat/UUID_download.dat"
+OUT=$UUID
 mkdir -p dat
 
-grep PDA $CATALOG | grep genomic | cut -f 11 | sort > $OUT
+grep PDA $CATALOG_MASTER | grep genomic | cut -f 11 | sort > $OUT
 echo Written to $OUT
diff --git a/15_summarize_download.sh b/15_summarize_download.sh
@@ -65,9 +65,6 @@ echo "          Targeted Sequencing: $TARG_SIZE Tb in $TARG_COUNT files"
 echo "                        TOTAL: $TOT_SIZE Tb in $TOT_COUNT files"
 }
 
-mkdir -p dat
-UUID="dat/UUID-all.dat"
-
 >&2 echo Catalog: $CATALOG_MASTER
 
 head -n1 $CATALOG_MASTER > $CATALOG_H
diff --git a/20_start_download.sh b/20_start_download.sh
@@ -0,0 +1,9 @@
+DATAD="/storage1/fs1/m.wyczalkowski/Active/Primary/CPTAC3.share/CPTAC3-GDC"
+TOKEN="/home/m.wyczalkowski/Projects/CPTAC3/import/token/gdc-user-token.2020-01-31T20_48_13.912Z.txt"
+CATALOG="/home/m.wyczalkowski/Projects/CPTAC3/CPTAC3.catalog/CPTAC3.Catalog.dat"
+UUID="dat/UUID-download.dat"
+
+#TESTARGS=-1ddd
+TESTARGS=$@
+
+src/start_downloads.sh -S $CATALOG -O $DATAD -t $TOKEN -g "-G compute-lding" -M -q general $TESTARGS - < $UUID
diff --git a/30_evaluate_download_status.sh b/30_evaluate_download_status.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# author: Matthew Wyczalkowski m.wyczalkowski@wustl.edu
+
+# This is a wrapper around src/evaluate_status.sh with parsing of configuration for convenience
+# All arguments passed to here will be passed to evaluate_status.sh
+
+# TODO: src/evaluate_status.sh can accept specific UUIDs as arguments, wereas here we just
+# go through all UUIDs.  Might want to improve this to allow UUID lists to be passed
+
+source gdc-import.config.sh
+
+if [ $LSF == 1 ]; then
+    ARG="-M"
+fi
+
+CMD="bash src/evaluate_status.sh $ARG -S $CATALOG_MASTER -O $DATA_ROOT $@ - < $UUID"
+eval $CMD
+
diff --git a/3_import.sh b/3_import.sh
diff --git a/README.compute1-testing.md b/README.compute1-testing.md
@@ -1,7 +1,8 @@
 DATAD="/storage1/fs1/m.wyczalkowski/Active/Primary/CPTAC3.share/CPTAC3-GDC"
-TOKEN="/storage1/fs1/home1/Active/home/m.wyczalkowski/Projects/CPTAC3/import/token/gdc-user-token.2020-01-31T20_48_13.912Z.txt"
+#TOKEN="/storage1/fs1/home1/Active/home/m.wyczalkowski/Projects/CPTAC3/import/token/gdc-user-token.2020-01-31T20_48_13.912Z.txt"
+TOKEN="/home/m.wyczalkowski/Projects/CPTAC3/import/token/gdc-user-token.2020-01-31T20_48_13.912Z.txt"
 CATALOG="/home/m.wyczalkowski/Projects/CPTAC3/CPTAC3.catalog/CPTAC3.Catalog.dat"
-UUID="UUID.dat"
+UUID="UUIDs.dat"
 
 Starting docker for direct run:
 ```
diff --git a/README.project.md b/README.project.md
@@ -1,5 +1,27 @@
-Development work to:
-1. Update data transfer tool to v1.5
-    See https://gdc.cancer.gov/access-data/gdc-data-transfer-tool
-2. Confirm works on compute1
-3. Simplify model of downloading
+This project consists of development of Y3 version and download of PDA RNA-Seq data.
+See README.compute1-testing.md for details about development
+
+PDA download is defined by UUID list in dat/UUID-download.dat, and consists of the following:
+
+Total required disk space WGS: 0 Tb in 0 files
+                          WXS: 0 Tb in 0 files
+                      RNA-Seq: 1.69332 Tb in 205 files
+                    miRNA-Seq: 0 Tb in 0 files
+            Methylation Array: 0 Tb in 0 files
+          Targeted Sequencing: 0 Tb in 0 files
+                        TOTAL: 1.69332 Tb in 205 files
+
+# Downloading
+
+bash 20_start_download.sh -J5
+
+## Downloading errors
+
+```
+Processing 25 / 205 [ Mon Feb 24 21:55:52 UTC 2020 ]: 1da42751-ac08-44d3-be4a-079c3cad256d
+Running: parallel --semaphore -j5 --id 20200224210947 --joblog ./logs/parallel.1da42751-ac08-44d3-be4a-079c3cad256d.log --tmpdir ./logs "bash src/launch_download.sh -g \"-G compute-lding\" -M -q general -o /storage1/fs1/m.wyczalkowski/Active/Primary/CPTAC3.share/CPTAC3-GDC 1da42751-ac08-44d3-be4a-079c3cad256d /home/m.wyczalkowski/Projects/CPTAC3/import/token/gdc-user-token.2020-01-31T20_48_13.912Z.txt 2917532a-84e2-478f-8002-2cdc0933731a.rna_seq.genomic.gdc_realn.bam BAM"
+Fatal ERROR. Exiting.
+```
+
+However, the jobs still seem to be running and the download is going OK.  Perhaps the fatal error has something to do with tmux and exiting?
+
diff --git a/evaluate_batch_status.sh b/evaluate_batch_status.sh
diff --git a/gdc-import.config.sh b/gdc-import.config.sh
@@ -1,44 +1,45 @@
 # Download batch name.  
 
-BATCH="methylation.20191202"
+BATCH="Y3.dev.PDA"
 
 # Data download root directory.  Individual BAMS/FASTQs will be in,
 #   $DATA_ROOT/GDC_import/data/<UUID>/<FILENAME>
 
 # Download token from GDC, good for 30 days.  Generating a new one causes old ones to break
-GDC_TOKEN="../token/gdc-user-token.2019-12-02T23_49_17.255Z.txt"
+GDC_TOKEN="../token/gdc-user-token.2020-01-31T20_48_13.912Z.txt"
 
 # Master CATALOG file containing all samples.  We will download a subset of these
 # Master BamMap file which hold most current list of BamMaps on system.  This file will not be modified by any scripts 
 
 # katmai
-CATALOG="/home/mwyczalk_test/Projects/CPTAC3/CPTAC3.catalog"
-DATA_ROOT="/diskmnt/Projects/cptac_downloads_7"
-MGI=0
-SYSTEM="katmai"
-
-## denali
-#CATALOG="/home/mwyczalk_test/Projects/CPTAC3/CPTAC3.catalog"
-#DATA_ROOT="/diskmnt/Projects/cptac_downloads/data"
-#MGI=0
-#SYSTEM="denali"
+# CATALOG="/home/mwyczalk_test/Projects/CPTAC3/CPTAC3.catalog"
+# DATA_ROOT="/diskmnt/Projects/cptac_downloads_7"
+# SYSTEM="docker"
+# LSF=0
 
 # MGI
 #CATALOG="/gscuser/mwyczalk/projects/CPTAC3/CPTAC3.catalog"
 #DATA_ROOT="/gscmnt/gc2741/ding/CPTAC3-data"
-## Define this =1 if in MGI environment, =0 otherwise
-#MGI=1
 #SYSTEM="MGI"
+# LSF=1
+
+# compute1
+CATALOGD="/home/m.wyczalkowski/Projects/CPTAC3/CPTAC3.catalog"
+DATA_ROOT="/storage1/fs1/m.wyczalkowski/Active/Primary/CPTAC3.share/CPTAC3-GDC"
+SYSTEM="compute1"
+LSF=1
+
+UUID="dat/UUID-download.dat"
 
 # This is common to all systems
-CATALOG_MASTER="$CATALOG/CPTAC3.Catalog.dat"
-BAMMAP_MASTER="$CATALOG/BamMap/${SYSTEM}.BamMap.dat"
+CATALOG_MASTER="$CATALOGD/CPTAC3.Catalog.dat"
+BAMMAP_MASTER="$CATALOGD/BamMap/${SYSTEM}.BamMap.dat"
 CASES_MASTER="$CATALOG/CPTAC3.cases.dat"
 
-# This is the CATALOG file which will drive processing here: all samples in this file will be downloaded
 # This file is generated in step 2 as a subset of CATALOG_MASTER
-CATALOG_H="dat/$BATCH.Catalog.dat"
+# It is no longer used to drive the workflow but remains for convenience
+CATALOG_H="dat/${BATCH}.Catalog.dat"
 
 # BAMMAP is created as the final step of import process.
-BAMMAP="dat/$BATCH.BamMap.dat"
+BAMMAP="dat/${BATCH}.BamMap.dat"
 
diff --git a/src/evaluate_status.sh b/src/evaluate_status.sh

Original file line number	Diff line number	Diff line change
`@@ -65,9 +65,6 @@ echo " Targeted Sequencing: $TARG_SIZE Tb in $TARG_COUNT files"`
`65`	`65`	`echo " TOTAL: $TOT_SIZE Tb in $TOT_COUNT files"`
`66`	`66`	`}`
`67`	`67`
`68`		`-mkdir -p dat`
`69`		`-UUID="dat/UUID-all.dat"`
`70`		`-`
`71`	`68`	`>&2 echo Catalog: $CATALOG_MASTER`
`72`	`69`
`73`	`70`	`head -n1 $CATALOG_MASTER > $CATALOG_H`