Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Travis #112

Open
wants to merge 27 commits into
base: develop
Choose a base branch
from
Open

Fix Travis #112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
6478d5d
merge metadata changes and updated raw imports from develop to master…
Ldcabansay Mar 11, 2019
45bdf4b
Update README.md
garyluu Sep 17, 2019
93ae930
made CRAI files optional and added JSON with gs:// URIs
aofarrel Feb 18, 2020
af0bb6c
added input {}s and fixed some spacing
aofarrel Jun 9, 2020
393648e
(do not use) Float/Int fixes, better indentation
aofarrel Jun 16, 2020
6c487c2
Update u_of_michigan_aligner.wdl
aofarrel Jul 1, 2020
cb75cd6
Update u_of_michigan_aligner.wdl
aofarrel Jul 1, 2020
84a1be2
First fully-functional version
aofarrel Jul 2, 2020
f5792e3
removed comments, fixed spacing
aofarrel Jul 2, 2020
3611c8c
Merge pull request #104 from DataBiosphere/feature-wdl1.0
aofarrel Jul 2, 2020
dfbba01
hotfix for syntax error in comment
aofarrel Aug 14, 2020
a9fa887
Add GitHub Apps support
nolwarre Sep 24, 2020
ae9758e
Re-push
nolwarre Sep 24, 2020
24962b1
Added missing workflow
nolwarre Sep 24, 2020
8a262ac
Merge pull request #111 from DataBiosphere/feature/3694/migrate-to-gi…
nolwarre Sep 25, 2020
b9844cc
Removed the "string hack"
aofarrel Oct 27, 2020
dcce7ad
wdl 1.0-like checker changes
aofarrel Nov 10, 2020
bc6e868
wdl 1.0-like formatting for checker
aofarrel Nov 10, 2020
6cd4a78
remove sub() workaround, which doesn't work in wdl 1.0
aofarrel Nov 10, 2020
efd88fb
attempt to fix disk_size issue
aofarrel Nov 10, 2020
b0a1bed
test if issue is additional_disk
aofarrel Nov 10, 2020
8badfa0
cleanup
aofarrel Nov 10, 2020
817cbdb
Merge branch 'master' into feature/wdl1.0
aofarrel Nov 10, 2020
26efab4
Merge pull request #105 from DataBiosphere/feature/wdl1.0
aofarrel Nov 10, 2020
de683e0
change WDL imports from soon-to-be-deleted branch
aofarrel Nov 11, 2020
5bedf5a
[untested] change distribution to trusty
aofarrel Dec 4, 2020
b52a1ca
Update README.md
aofarrel Feb 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .dockstore.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
version: 1.2
workflows:
- subclass: WDL
primaryDescriptorPath: /variant-caller/variant-caller-wdl/topmed_freeze3_calling.wdl
testParameterFiles:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't you want to use the newer https://github.com/DataBiosphere/topmed-workflow-variant-calling?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good spot, thanks!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually since that's a different repo I'm a little hesistant. Does that one have its own page on Dockstore?

- /variant-caller/variant-caller-wdl/topmed_freeze3_calling.json
name: UM_variant_caller_wdl
- subclass: WDL
primaryDescriptorPath: /aligner/u_of_michigan_aligner/u_of_michigan_aligner.wdl
testParameterFiles:
- /aligner/u_of_michigan_aligner/u_of_michigan_aligner.json
name: UM_aligner_wdl
- subclass: WDL
primaryDescriptorPath: /aligner/functional-equivalence-wdl/FunctionalEquivalence.wdl
testParameterFiles:
- /aligner/functional-equivalence-wdl/FunctionalEquivalence.json
name: CCDG_aligner_functional_equivalent_wdl
- subclass: CWL
primaryDescriptorPath: /aligner/sbg-alignment-cwl/topmed-alignment.cwl
testParameterFiles:
- /aligner/sbg-alignment-cwl/topmed-alignment.sample.json
name: UM_aligner_cwl
- subclass: CWL
primaryDescriptorPath: /aligner/topmed-cwl/workflow/alignment_workflow.cwl
testParameterFiles:
- /test.json
name: CCDG_aligner_functional_equivalent_cwl
- subclass: CWL
primaryDescriptorPath: /variant-caller/sbg-variant-caller-cwl/topmed_freeze3_calling.json
testParameterFiles:
- /variant-caller/sbg-variant-caller-cwl/topmed_freeze3_calling.json
name: UM_variant_caller_cwl
- subclass: CWL
primaryDescriptorPath: /vcf-comparator/ConcordanceTestWorkflow.cwl
testParameterFiles:
- /test.json
name: gatk-vcf-comparator
5 changes: 5 additions & 0 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ language: java
jdk:
- oraclejdk8

# Cromwell requires oraclejdk8
# Xenial, the new default for travis, does not support open/Oraclejdk8
# https://travis-ci.community/t/expected-feature-release-number-in-range-of-9-to-12-but-got-8-installing-oraclejdk8/1345/16
# Thus our hands have been forced
dist: trusty

# found at https://github.com/GoogleCloudPlatform/Template/blob/master/.travis.yml
cache:
Expand Down
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
The original pipelines were assembled and written by Hyun Min Kang ([email protected]) and Adrian Tan ([email protected])
at the [Abecasis Lab at the University of Michigan](https://genome.sph.umich.edu/wiki/Abecasis_Lab)

See the [variant calling pipeline](https://github.com/statgen/topmed_freeze3_calling) and [alignment pipeline](https://github.com/statgen/docker-alignment) repositories
See also the [variant calling pipeline](https://github.com/statgen/topmed_freeze3_calling) and [alignment pipeline](https://github.com/statgen/docker-alignment) repositories

## Installing dependencies on your local system

Expand Down Expand Up @@ -43,7 +43,7 @@ To run workflows of data stored on `gcloud` you need to set an environment varia
`cromwell` is a Java executable and requires a Java Runtime Engine. Follow the instruction [here](http://cromwell.readthedocs.io/en/develop/tutorials/FiveMinuteIntro/) for a complete installation.

### 3. Dockstore
For Dockstore to run you need to install the [Java Runtime Engine](https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04). Find installation instructions for Dockstore [here](https://dockstore.org/onboarding) (you need to be logged in to Dockstore).
For Dockstore to run you need to install the [Java Runtime Engine](https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04). Find installation instructions for Dockstore [here](https://dockstore.org/quick-start).

## Running workflows

Expand Down
36 changes: 20 additions & 16 deletions aligner/u_of_michigan_aligner-checker/u_of_michigan_aligner_checker.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,30 @@
import "https://raw.githubusercontent.com/DataBiosphere/topmed-workflows/1.32.0/aligner/u_of_michigan_aligner/u_of_michigan_aligner.wdl" as TopMed_aligner
import "https://raw.githubusercontent.com/DataBiosphere/topmed-workflows/1.32.0/aligner/u_of_michigan_aligner-checker/u_of_michigan_aligner_checker_calculation.wdl" as checker
version 1.0

import "https://raw.githubusercontent.com/DataBiosphere/topmed-workflows/master/aligner/u_of_michigan_aligner/u_of_michigan_aligner.wdl" as TopMed_aligner
import "https://raw.githubusercontent.com/DataBiosphere/topmed-workflows/master/aligner/u_of_michigan_aligner-checker/u_of_michigan_aligner_checker_calculation.wdl" as checker
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does using 'master' mean TravisCI will only be pulling in the latest released WDLs? I guess maybe there isn't a better way to do this until relative imports are supported in Terra so they can be used here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember all the details, but if I recall correctly, it has to be master because 1.32.0 is out of date and doesn't work on Terra anymore. I'd be happy to work with your team on another release.


workflow checkerWorkflow {
String docker_image
input {
String docker_image

File input_crai_file
File input_cram_file
File? input_crai_file
File input_cram_file

File inputTruthCRAMFile
File inputTruthCRAMFile

File ref_alt
File ref_bwt
File ref_sa
File ref_amb
File ref_ann
File ref_pac
File ref_alt
File ref_bwt
File ref_sa
File ref_amb
File ref_ann
File ref_pac

File ref_fasta
File ref_fasta_index
File ref_fasta
File ref_fasta_index

File dbSNP_vcf
File dbSNP_vcf_index
File dbSNP_vcf
File dbSNP_vcf_index
}

call TopMed_aligner.TopMedAligner as aligner {
input:
Expand Down
39 changes: 25 additions & 14 deletions aligner/u_of_michigan_aligner-checker/u_of_michigan_aligner_checker_calculation.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,32 @@
version 1.0

task checkerTask {
File inputCRAMFile
File inputTruthCRAMFile
File referenceFile
input {
File inputCRAMFile
File inputTruthCRAMFile
File referenceFile

String docker_image

String docker_image
# Optional input to increase all disk sizes in case of outlier sample
# with strange size behavior
Int? increase_disk_size

# Optional input to increase all disk sizes in case of outlier sample with strange size behavior
Int? increase_disk_size
# Some tasks need wiggle room, and we also need to add a small amount of disk to prevent getting a
# Cromwell error from asking for 0 disk when the input is less than 1GB
Int additional_disk = select_first([increase_disk_size, 200])

# Some tasks need wiggle room, and we also need to add a small amount of disk to prevent getting a
# Cromwell error from asking for 0 disk when the input is less than 1GB
Int additional_disk = select_first([increase_disk_size, 200])
# The size function causes an error when a relative path is provided as input in the JSON
# input file. Somehow Cromwell confuses where the file is for the size function in this case.
# Float disk_size = size(inputTruthCRAMFile, "GB") + size(inputCRAMFile, "GB") + size(referenceFile, "GB") + additional_disk

# Additionally, the older version below does not work in WDL 1.0 for reasons I cannot fathom
# Float disk_size = additional_disk

Float disk_size = additional_disk
# The size function causes an error when a relative path is provided as input in the JSON
# input file. Somehow Cromwell confuses where the file is for the size function in this case.
# Float disk_size = size(inputTruthCRAMFile, "GB") + size(inputCRAMFile, "GB") + size(referenceFile, "GB") + additional_disk
# For these reasons additional_disk is now used for the disks runtime attribute rather than disk_size
# Since the input and the truth file are both small this is probably an acceptable compromise, but
# if the inputs ever get changed to something larger this may require revision.
}

command {
# The md5sums for the SAM files without headers created from the CRAM files should match
Expand All @@ -29,6 +40,6 @@ task checkerTask {

runtime {
docker: docker_image
disks: "local-disk " + sub(disk_size, "\\..*", "") + " HDD"
disks: "local-disk " + ceil(additional_disk) + " HDD"
}
}
20 changes: 20 additions & 0 deletions aligner/u_of_michigan_aligner-checker/u_of_michigan_aligner_checker_gs_urls.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"checkerWorkflow.input_cram_file": "gs://topmed_workflow_testing/topmed_aligner/input_files/NWD176325.25reads.cram",
"checkerWorkflow.input_crai_file": "gs://topmed_workflow_testing/topmed_aligner/input_files/NWD176325.25reads.cram.crai",

"checkerWorkflow.inputTruthCRAMFile": "gs://topmed_workflow_testing/topmed_aligner_checker/truth_NWD176325.25reads.cram",

"checkerWorkflow.ref_alt": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa.alt",
"checkerWorkflow.ref_bwt": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa.bwt",
"checkerWorkflow.ref_pac": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa.pac",
"checkerWorkflow.ref_ann": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa.ann",
"checkerWorkflow.ref_amb": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa.amb",
"checkerWorkflow.ref_sa": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa.sa",
"checkerWorkflow.ref_fasta": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa",
"checkerWorkflow.ref_fasta_index": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/hs38DH.fa.fai",

"checkerWorkflow.dbSNP_vcf": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/Homo_sapiens_assembly38.dbsnp138.vcf.gz",
"checkerWorkflow.dbSNP_vcf_index": "gs://topmed_workflow_testing/topmed_aligner/reference_files/hg38/Homo_sapiens_assembly38.dbsnp138.vcf.gz.tbi",

"checkerWorkflow.docker_image": "statgen/alignment:1.0.0"
}
Loading