human variation .mk files

[njbowen]

I have not learned makefile scripting customization very well yet. i have the snpcall.mk running on an ubuntu system with the default caller.
and I was able to get singularity installed. I tried to run the snpcall.mk workflow inserting

# Variant caller module.
SNP_MODULE ?= src/run/deepvariant.mk 

but am getting this error following this command:
make -f src/run/deepvariant.mk REF=/home/myusername/refs/alias/hg38/fasta/default/hg38.fa BAM=tmp/SRR6808334.chr1.bam VCF=tmp/SRR6808334.chr1.vcf.gz run
make[1]: *** No rule to make target 'run'. Stop.

i found that the R1 ?=reads/read1.fq for the bwa.mk needs to be something like this instead, I think:

# First in pair.
R1 ?= reads/${SRR}_1.fastq

and this command in the workflow stops as well:

`$ freebayes --pvar 0.5 -f refs/AF086833.fa bam/SRR1553425.bam | bcftools norm -f refs/AF086833.fa -d all -O z  > vcf/SRR1553425.fb.vcf.gz`
Failed to read from standard input: unknown file type

is this error coming from piping the output of freebayes into bcftools ?

also is it possible to run singularity from MacOS? thanks.

[ialbert]

I will be cleaning up these examples shortly to make them fully consistent with other examples. I have not been using these lately and looks like these drifted out of consistency with the others. I'll be adding a test that you and everyone can run that would run all modules and workflows.

[ialbert]

A new makefile called Test.mk has been added. A section in the documentation describes the testing framework (make sure to refresh the page to get the updated content):

https://www.biostarhandbook.com/books/workflows/help/install/#4-how-to-test-all-modules

Here is how the usage would go. Get the updated code

bio code

then first run the

bash src/setup/init-stats.sh

setup script that will create two additional environments, one called stats and one called salmon_env that runs salmon. The tests assume that these environments are present. Then run:

make -f src/Test.mk

to see the various targets it can run. For example the command below tests all modules:

make -f src/Test.mk test

or test all recipes:

make -f src/Test.mk recipes

to test the deepvariant module (UNIX only)

make -f src/Test.mk deepvariant

[njbowen]

above make -f src/Test test
should be -f src/Test.mk test correct? same for recipes line as well..

[ialbert]

yes, correction made

[ialbert]

There is a more general change that affects all modules as well.

It used to be that the modules had "smart" defaults, basically were prefilled with names generated from accession numbers and SRR numbers.

This "intelligent" behavior seemed useful initially as the examples all worked like "magic" and looked elegant.

In practice, the tacit behavior leads to many subtle bugs and can result in inadvertently using the wrong read/reference/annotation, etc. It tripped me up many times as well. When used in real-life scenarios, we always override all inputs to control the file naming.

Going forward, all inputs will be labeled with non-existing files like R1=read1.fq for reads or REF=genome.fa when the reference needs to be a genome and REF=transcripts.fa when references need to be a transcripts etc.

I hope that the new kind of labeling will better communicate what needs to go where.

The module's end user must enter the correct read name, such as R1=read/SRR1553425_1.fastq or REF=refs/AF086833.fa.

In the end, bioinformatics is the science of plugging the right file to the right place so we have to approach it that way and not overautomate that process.

In addition, every file will be checked, and the makefile will try to produce a more meaningful error messages when a file is missing rather than tacitly using an existing file that the user did not intend to include.

I will be working through the examples to clean those up as well.