-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathCHANGES.txt
406 lines (345 loc) · 32.5 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
07aug24 * No major changes; updated install_R_deps.R, run_kdetrees.R, run_parallel_cmmds.pl, run_parallel_molecClock_test_with_paup.sh,
estimate_pangenome_phylogenies.sh, run_get_phylomarkers_pipeline.sh so that 'make test' completes in Ubuntu 20.04, silencing
parallel citation messages, but note that this software won't be possible without it:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
20apr24 * run_get_phylomarkers_pipeline.sh v2.8.4.0_2024-04-20
- refactored all cleanup code into functions called by trap 'cleanup_trap "DIR" END' ABRT EXIT HUP INT QUIT TERM
- removed obsolete commented lines to tidy code layout
- SC revised compliance
* lib/get_phylomarkers_fun_lib v2024-04-20
- refactored all cleanp code from run_get_phylomarkers_pipeline 2.8.4.0_2024-04-20
into multiple, block-specific cleanup functions for better modularization and easier maintenance:
function cleanup_DNA_top_markers_dir
function cleanup_DNA_recomb_dirs
function cleanup_DNA_top_dir
function cleanup_PopGen_run
function cleanup_PROT_top_markers_dir
function cleanup_PROT_recomb_dirs
function cleanup_PROT_top_dir
- The cleanup functions are called by strategically-located trap calls to function cleanup_trap,
which also call print_citation before exiting the main script
19apr24 * run_get_phylomarkers_pipeline.sh 2.8.3.0_2024-04-19
- added likelihood mapping test of tree-likeness and phylogentic signal for -A I (IQ-TREE) and -t DNA|PROT
- top markers for -A I -t DNA|PROT need to pass both the lmapping and min median support values
- added trap print_citation calls at strategic points in the main script where exit is called when insufficient markers are left
after filtering to continue, i.e., at exit points that are not related to errors
* lib/get_phylomarkers_fun_lib v2024-04-19
- added print_citation, called by trap calls at strategic points in the main script, i.e., at exit points that are not related to errors
18Apr24 * created GitHub release v2.2.1_2024-04-18 https://github.com/vinuesa/get_phylomarkers/releases/tag/2.2.1
* created DockerHub image v20240418
16-17Apr24 * run_get_phylomarkers_pipeline.sh v2.8.1.4_2024-04-17
-Improved reporting at the end of run (STDOUT and file written to disk)
of key ouput files and figures generated by the pipeline, calling new
functions rm_top_dir_prefix_from_wkdir and read_svg_figs_into_hash
* get_phylomarkers_fun_lib v2024-04-17
- created new functions rm_top_dir_prefix_from_wkdir and read_svg_figs_into_hash
- prepended missing "$bindir"/ to several iqtree calls in run_ASTRAL and compute_ASTRALspTree_branch_lenghts
- improved print_start_time progress reporting from run_ASTRAL and compute_ASTRALspTree_branch_lenghts
15Apr24 * compiled a static versions of snp-sites for Linux (bin/linux/snp-sites-static),
which is directly called from run_get_phylomarkers_pipeline.sh v2.8.1._2024-04-15
Thanks to Alfredo Hernandez from CCG-UNAM & LCG-UNAM for compiling this binary
* run_get_phylomarkers_pipeline.sh v2.8.1._2024-04-15 now calls $bindir/snp-sites-static
* Updated Dockerfile (v2024-04-15) to avoid installing non-required R packages and snp-sites
* Updated INSTALL.md and README.md
* updated install_R_deps.R and apt-install_R_dependencies.sh v2024-04-15: replaced devtools for the much lighter remotes; removed uncessary R packages
* updated .travis.yml v2024-04-15; does not call apt install snp-sites, as it is now provided as static binary in bin/linux/snp-sites-static
14Apr24 * compiled new static versions of astral and wastral for Linux
* removed dynamically linked snp-sites for Linux, which is now called from run_get_phylomarkers_pipeline.sh v2.8.0.2_2024-04-14 from the system binary in PATH
- NOTES: snp-sites should be statically compiled or it will fail when called as '$bindir/snp-sites'
The Docker image and TravisCI install snp-sites via apt install
* run_get_phylomarkers_pipeline.sh 2.8.0.2_2024-04-14
- removed '$bindir/' from snp-sites call, because it is dynamically linked, and won't run on TravisCI. It is currently called from PATH
* run_get_phylomarkers_pipeline.sh v2.8.0.1_2024-04-14
- changed default IQT_threads=AUTO to IQT_threads=2 # used only with concatenated supermatrix 2|INTEGER; AUTO is too slow
- prepended '$bindir/' to iqtree and snp-sites calls that still missed it. NOTE: snp-sites needs to be statically compiled
- SC2044: replaced two for loops over find output for a more robust while read loop reading from <(cmd)
13Apr24 * install_R_deps.R v2024-04-13; removed packages cluster, dendextend, and factoextra
* run_get_phylomarkers_pipeline.sh v2.8.0.0_2024-04-13
- significant extension of runmode 2 (-R 2) for population genetics:
- the FASTA files of neutral loci are saved in their own directory and concatenated
- SNP sites are extracted from the concatenated alignment with snp-sites
and save as FASTA and VCF formats
- ML trees are estimated from the SNP supermatrix using either IQ-TREE or FastTree
- uses read -a ARR <<< "$(get_critical_TajD_values ..." to split command output into ARR
instead of using pipes to awk or ARR=$( (cmd) ); this fixes an obscture bug of unset variables
that allowed popGen mode (-R 2) to run only if using debugging options -D 1 || -D 2
* New Docker image on Dockerhub
* run_test_suite.sh v2024-04-13
- runs under Bash's unnofficial strict mode: set -euo pipefail
- adds a second test for -R 2 (popGen) to estimate the popgen tree using FastTree
- removed hcluster test, as it belongs to get_homologues
* test_get_phylomarkers.t v2024-04-13
- Now runs 24 tests
- added test for -R 2 (popGen), estimating popgen tree using FastTree
- both tests for -R 2 (FT & IQT) now grep the line reporting the generation of the SNP matrix
- All -R 1 IQT runs are under -I 2 -N 2
* .travis.yml v2024-04-13 now installs snp-sites in the 'apt install' call
12Apr24 * run_get_phylomarkers_pipeline.sh v2.7.6.7_2024-04-11
- Collects results of filtering steps in the filtering_results_h hash,
and prints an overview of the pipeline's filtering process before exiting
10Apr24 * run_get_phylomarkers_pipeline.sh v2.7.6.6_2024-04-10
- prints script invocation arguments
- fixed test for symlink call in top markers dir
9Apr24 * get_phylomarkers_fun_lib v2024-04-09
- full shellcheck compliance
- added complex protein mixture models for concatenated protein alignments
* run_get_phylomarkers_pipeline.sh v2.7.6.4_2024-04-09
- full shellcheck compliance
- redirect filtered lines to a while read loop as process substitutions <(commands)
instead of for loops
7-8Apr24 * run_get_phylomarkers_pipeline.sh v2.7.6.3_2024-04-08:
- improved file reading with while read and Bash's parameter expansions
- added symlink test for top markers
- improved help dislay and other minor edits
06apr24 * get_phylomarkers_fun_lib v2024-04-06
- run_ASTRAL computes best-fitting model from prot or dna concat alignments using IQT
with wASTRAL species tree as constraint
- run_ASTRAL calls compute_ASTRALspTree_branch_lenghts from prot or dna concat alignments
using IQT with wASTRAL species tree as constraint
- run_ASTRAL now calls astral4 AND wASTRAL from the ASTER package,
using the wASTRAL species tree for the downstream analyses listed above
05apr24 * wastral v1.18.3.5 for Linux; first commit 2024-04-05; run_ASTRAL now calls also wastral
02apr24 * test_get_phylomarkers.t: fixed regexes t17,t19,t20; PASSES TRAVIS CI BUILD using all 23 tests
* run_get_phylomarkers_pipeline.sh v2.7.4_2024-04-02: run_ASTRAL is called before FT or IQT on concatenated alignment
* run_get_phylomarkers_pipeline.sh v2.7.3_2024-04-02 & get_phylomarkers_fun_lib v2024-04-02 renamed mispelled SRHtests
* .travis.yml v2024-04-02; removed env: global: secure: ***; set up new ATP;
* NEW RELEASE v2.1.1: GET_PHYLOMARKERS_v2.1.1_2024-04-01; https://github.com/vinuesa/get_phylomarkers/releases/tag/v2.1.1
01apr24 * run_kdetrees.R v0.4.0_2024-04-01 checks for number of outlier trees, and quit if non are found;
* run_get_phylomarkers_pipeline.sh v2.7.2_2024-04-01; improved checking and handling of run_kdetrees output
* updated LICENSE.txt, removed ASTRAL-III directory and java code; test_get_phylomarkers.t v2024-04-01; test 21 (popGen) runs with -D 1
* Dockerfile v2024-04-01 and Docker image on DockerHub
* run_kdetrees.R v0.3.1_20mar24; explicitly exits with status = 0
30mar24 * downgraded back again to iqtree2 v2.2.0 (latest release, Mar 25, 2022), in bin/linux/
30mar24 * added astral4 v1.18.4.5, written in C, from the ASTER package to bin/linux/
30mar24 * get_phylomarkers_fun_lib v2024-03-30 passes all test_get_phylomarkers.t under strict mode; run_ASTRAL calls astral4 C-binary, instead of ASTRAL-III (java jar).
30mar24 * run_get_phylomarkers_pipeline.sh 2.7.0_2024-03-30 passes all test_get_phylomarkers.t under strict mode; calls astral4 C-binary, instead of ASTRAL-III (java jar).
28mar24 * run_get_phylomarkers_pipeline.sh 2.6.0_2024-03-28 implements the maximal matched-pairs tests to asses SRH assumptions. Calls new function parse_SRM_tests
28mar24 * run_get_phylomarkers_pipeline.sh 2.5.2_2024-03-28 passes all tests under strict mode; used arrays to count and report fastas producing trees with < 4 external branches.
27mar24 * run_get_phylomarkers_pipeline.sh 2.5.1_2024-03-27 and get_phylomarkers_fun_lib now run under full Bash's unofficial strict mode: set -euo pipefail; optimal shellcheck compliance
26mar24 * updated iqtree2 binary for Linux to v2.3.1 (released Mar 25, 2024)
14jan23 * run_get_phylomarkers_pipeline.sh 2.5.0_2023-01-14:
- calls compute_ASTRALspTree_branch_lenghts;
- restored compute_suppValStas_and_RF-dist.R call for prot alignments;
- fixed bug in iqtree call: -B 1000 "$IQT_threads" => -B 1000 -T "$IQT_threads"
15jan23 * get_phylomarkers_fun_lib v2023-01-16: run_ASTRAL computes the gene concordance factors (gCF); new function compute_ASTRALspTree_branch_lenghts uses iqtree to compute an ASTRAL species tree with branch lenghts
14dec22 * run_get_phylomarkers updated check_scripts_in_path: Pass USER as 1st argument to get_script_PID, and progname as 2cnd arg from run_get_phylomarkers_pipeline.sh and estimate_pangenome_phylogenies.sh to get_script_PID calls; Localized user and prog in get_phylomarkers_fun_lib::get_script_PID to avoid unbound variable USER/user in function call
14dec22 * get_phylomarkers_fun_lib v2022-12-14: Pass USER as 1st argument to get_script_PID, and progname as 2cnd arg from run_get_phylomarkers_pipeline.sh and estimate_pangenome_phylogenies.sh; Localized user and prog in get_phylomarkers_fun_lib::get_script_PID
19nov22 * run_get_phylomarkers v2.4.5_2022-11-14; fixed --version call and improved code layout and shellcheck compliance; updated Dockerfile .gitignore .dockerignore for new Docker image
19nov22 * run_get_phylomarkers v2.4.4_2022-11-14; optimization of IQT calls results in ~20x faster execution time of the test suite; added IQT_threads; removed -T AUTO
18nov22 * get_phylomarkers_fun_lib v2022-11-17: 1) mostly shellcheck compliant:
- proper variable localization as list, before assignment;
- uses mapfile -t array < <(mycommand) to read command output lines into array.
- calls functioncaller if (( DEBUG > 0 )) && caller 0
14nov22 * run_get_phylomarkers v2.4.3_2022-11-14; updated iqtree calls to iqtree2 syntax; print_start_time now uses printf builtin instead of calling date
13nov22 * get_phylomarkers_funlib v2022-11-13; minor modifications for greater shellcheck compliance; fixed PERL5LIB: unbound variable error in export PERL5LIB call
13nov22 * v2.4.2_2022-11-13 run_get_phylomarkers implements strict mode; simplified if checks for command output status; consistent bash arithmetic syntax; shellcheck and other corrections
23oct22 * added Publication DOI, GPLv3 and DockerHub badges
24jul22 * iqtree-2.2.0-MacOSX
2jul22 * added encrypted GitHub token using travis CLI; encrypted env var GITHUB_PAT, for remotes::install_github()
2jul22 * added library remotes for remotes::install_github(); kdetrees; calls Rscript install_kdetrees_from_github.R using library(remotes
16Jun22 * updated to iqtree v.2.2.0; new docker image uploaded to DockerHub https://hub.docker.com/r/vinuesa/get_phylomarkers
13Jun22 * manual version 2022-06-13; with ASTRAL ref and new flowchart
12Jun22 * updated documentation and R's ape and kdetrees packages
10Jun22 * added binaries and code to estimate species trees from filtered gene trees using ASTRAL-III
04Oct21 * run_test_suite.sh v2021-10-04; corrected typo in test #9; -o instead of -O
23Sep21 * INSTALL.md v2021-09-23; setup R_LIBS_SITE and libnw with impromved text & markup formatting of code snippets
21Sep21 * install_R_deps.R v2021-09-21; calls plyr before dplyr; added stringi before stringr; additional notes
20Sep21 * .travis.yml v2021-09-20; adds curl build-essential bc make g++ to sudo apt-get install call
19Sep21 * test_get_phylomarkers.t runs 22 tests; fixed a rm kde*out case; adjusted install_R_deps.R; new Dockerfile that runs make test during final build stage; removed hcluster_pangenome_matrix
18Sep21 * install_R_deps.R installs ape and kdetrees from source; fixed estimate_pangenome_phylogenies.sh & test_get_phylomarkers.t; GitHub builds with Travis CI
17Sep21 * run_get_phylomarkers_pipeline.sh 2.3.0 and all aux. scripts pass cleanly all 22 tests (run_make_tests.sh)
16Sep21 * updated README.md; added check_libnw() to estimate_pangenome_phylogenies.sh
16Sep21 * extended number of tests in test_get_phylomarkers.t (22) and run_test_suite.sh (9)
16Sep21 * run_get_phylomarkers_pipeline.sh v2.3.0, passes cleanly all tests; no STERR messages after extensive checking in the final cleanup section (run_make_tests.sh)
16Sep21 * updated documentation md and index.html files; issues a warning message if the kdetrees test could not be run, instead of dying for -t PROT (5.3)
15Sep21 * updated/fixed Dockerfile to include python2.7 & libpython2.7.so.1.0 required by PAUP* for popGen;added run_test_suite.sh vinuesa/get_phylomarkers:latest pushed to Docker hub
14Sep21 * updated FastTree source and linux binary to version 2.1.11; updated iqtree binary for linux to v2.1.3; added
13Sep21 * Dockerfile v.2021-09-13 and corresponding images vinuesa/get_phylomarkers 2021-09-13-190724 and vinuesa/get_phylomarkers:latest pushed to Docker hub
13Sep21 * run_get_phylomarkers_pipeline.sh v2.2.9.3_13sep2021: removed erroneously pasted line
13Sep21 * run_get_phylomarkers_pipeline.sh v2.2.9.2_13sep2021: sets PRINT_KDE_ERR_MESSAGE=0; use find . -type l delete to find & remove symlinks; added Docker hub links;
09Sep21 * added .dockerignore and Dockerfile; updated README.md and INSTALL.md; reduced list of R packages to install; build new Docker image
30Jan21 * run_get_phylomarkers_pipeline.sh v.2.2.9.1_30Jan2021: issues a warning message if the kdetrees test could not be run, instead of dying
30Oct20 * uses perl 5.30 and focal-cran40 repo
02Apr20 * updated install_R_deps.R to use ubuntu bionic-cran35/ repo
02Apr20 * activated the GitHub integration to start testing and deploying on Travis CI
31Mar20 * added .travis.yml + test_get_phylomarkers.t + Makefile + cpanfile
10Jan20 * estimate_pangenome_phylogenies.sh v1.1_10Jan20; added option -S <abayes|UFBoot|both>
24Nov19 * updated IQ-TREE v2.0-rc1 binaries (64-bit) for ubuntu Linux and MacOSX
03Sep19 * updated PAUP* versions: paup4a166_ubuntu64 and paup4a166_osx
20Aug19 * updated FastTree v2.1.11 Double precision (No SSE3) for Linux
20Aug19 * updated IQ-TREE v1.6.12 binaries (64-bit) for ubuntu Linux and MacOSX
16Jul19 * tidied the ${perl_scripts[@]} in check_scripts_in_path() and changed hard-coded -e min. = 4 for $min_no_ext_branches
15Jul19 * updated manual: minor fixes and more info on running docker
17Jun19 * updated iqtree binary to v.1.6.11 for ubuntu Linux (64-bit) and MacOSX
09Apr19 * updated PAUP* build 165 binaries for ubuntu Linux (64-bit) and MacOSX
27Feb19 * updated iqtree binary to v.1.6.10 for ubuntu Linux (64-bit) and MacOSX
02Jan19 * updated IQ-TREE v1.6.9 binaries for ubuntu Linux (64-bit) and MacOSX
18Nov18 * added function print_software_versions()
14Nov18 * updated PAUP* build 164 binaries for ubuntu Linux (64-bit) and MacOSX
14Nov18 * updated IQ-TREE v1.6.8 binaries for ubuntu Linux (64-bit) and MacOSX
24Oct18 * updated paup (build 163) and IQ-TREE v1.6.7.1, both for ubuntu Linux (64-bit) and MacOSX
3Oct18 * more input file name cleanup code to remove characters like ',/;()[]: from input files that may interfere with the shell; 2.2.6_3Oct2018
11Jul18 * prepared new release tagged 1.2.7
11Jul18 * updated iqtree for linux and macosx-intel to v1.6.6; updated paup to release 162
27May18 * updated iqtree for linux and macosx-intel from v1.6.3 to v1.6.5
9May18 * adapted regex for -f EST in run_get_phylomarkers.sh to properly parse the new FASTA header format of clusters computed by GET_PHYLOMARKERS_EST
8May18 * added option -f <STD|EST> to parse GET_HOMOLOGUES_EST FASTA header; updated GET_PHYLOMARKERS reference
31Mar18 * run_get_phylomarkers_pipeline.sh v2.2.2; added [ $no_tree_counter ] && WARNING msg about discarding trees with < 1 internal branches
31Mar18 * updated install instructions
* prepared new release tagged 1.2.5
* uploaded new docker image
30Mar18 * updated linux and macosx binaries for iqtree (v1.6.3) and paup (v4a161)
29Mar18 # Code:
* get_phylomarkers_fun_lib 29032018;
- compute_MJRC_tree() fix to contend with possible Bug in IQ-TREE v1.6.2 and v1.6.3
IQT v1.6.2 and v1.6.3, but not v1.6.1!, cannot compute the mjrc tree from the debug set,
claiming that the number of taxa are different across trees when running with FT ???
There is not problem with the IQ-TREE run on the same sequences...
>>> Therefore check_output was changed for simple warning messages to avoid premature exit <<<
- fixed quoting of variables passed to compute_FT_gene_tree_stats() as arguments
* run_get_phylomarkers_pipeline.sh 2.2.1_29Mar18
- fixed call of compute_MJRC_tree, properly adding the quotes around variables
29Mar18 # Code run_get_phylomarkers_pipeline.sh v2.2.0_29Mar18 and estimate_pangenome_phylogenies.sh v1.0.5_28Mar18
* check_scripts_in_path() checks wether USER is regular or root
to add symlinks to $HOME/bin or /usr/local/bin, respectivel
10Mar18 * updated linux and macOSX iqtree binaries to version 1.6.2
18Feb18 # Code run_get_phylomarkers_pipeline.sh v2.1.9_18Feb18 and get_phylomarkers_fun_lib v18022018
* fixed redundant output messages written to logfile
* made more shellcheck compliant
# Manual
* More detailed and structured manual in html and md formats
17Feb18 * added $distrodir to a nw_reroot call that was missing it
* made lib/get_phylomarkers_fun_lib more shellcheck compliant
15Feb18 * run_get_phylomarkers_pipeline.sh 'v2.1.7_15Feb18'; added more DEBUG INFO to initial FASTA sanity checks
* fixed typos in manual
12Feb18 * added html-based manual and tutorial, with table of contents, under https://vinuesa.github.io/get_phylomarkers
11Feb18: * run_get_phylomarkers_pipeline.sh 'v2.1.5_11Feb18' is more shellcheck-compliant
* get_script_PID in get_phylomarkers_fun_lib now uses pgrep; the library is more shellcheck-compliant
8Feb18: # 1. Changes to estimate_pangenome_phylogenies.sh v1.0.1_8Feb18
* Added: -v flag; check_scripts_in_path(); check_dependencies with verbosity; activated set_pipeline_environment; should find dependencies: Thanks to Felipe Lira
* v1.0.3_8Feb18; fix in set_pipeline_environment: changed to readlink -n when $OSTYPE -> darwin
2. Changes in run_get_phylomarkers_pipeline.sh 'v2.1.4_8Feb18' v2.1.4_8Feb18
* fixed set_pipeline_environment (removed DEBUG messages, which broke the function return when the script was run with -D; fixed for darwin: scriptdir=$(readlink -n "${BASH_SOURCE[0]}")
7Feb18: * Moved print_codon_tables() back from get_phylomarkers_fun_lib to run_get_phylomarkers_pipeline.sh v2.1.3_7Feb18 so that -C can print them; Thanks to Stefano Romano!
* relocated the assignment of logdir to top of main script; edited INSTALL.md, README.md and manual.md
5Feb18: * Extended the tutorial on exploring the container's environment and file system
* Updated the README.md file to include links to the tutorial and install instructions of the Docker environment and container usage
4Feb18: * -v flag also prints list of required system binaries, like GET_HOMOLOGUES; minor fix in cleanup code.
3Feb18: * Extended the INSTALL.md file to include instructions on how to setup a locak Docker environment and run image instances
* Extended the tutorial material, to include instructions on How to access the sequence data on the host (your machine) from a container instance
2Feb18: # changes in tagged release v1.2.1
* minor fix in cleanup/tidying of directories in run_get_phylomarkers_pipeline.sh v2.1.1_2Feb18
* this release has the first fully working docker image for all runmodes and -A I|F -K ... Passed all tests
2Feb18: # changes in tagged release v1.2.0 and v1.2.1
* added statically compiled bc for linux (Thanks to Alfredo J. Herandez @CCG_UNAM) to avoid problmes in docker containers that don't find the
interpreter /lib64/ld-linux-x86-64.so.2 in dynamically compiled binaries
* added new function compute_MJRC_tree() based on iqtree -con, which computes the MJRC trees for all non-recomb input trees and top markers, for -F A|I and -t DNA|PROT
* updated run_get_phylomarkers_pipeline.sh to 'v2.1.0_2Feb18', which calls compute_MJRC_tree() and performs proper cleanup/tidying of directories when -R 2
* further checked cleanup functions
1Feb18: * run_get_phylomarkers_pipeline.sh v2.0.4_1Feb18 performs a more thorough and consistent final cleanup/tidying of directories
* added bc Mach-O 64-bit x86_64 executable to bin/macosx-intel
* changed calls to rename for $distrodir/rename.pl in get_phylomarkers_fun_lib so that docker container instances can find it
31JAN18: * updated the install instructions to reflect changes associated to release 1.1.0,
which include links to the Docker image, manual and tutorials + additional dependiencies.
* added -v flag to run_get_phylomarkers_pipeline.sh to print version; added find to the list of binaires to check
29JAN18: * Added the pars and seqboot binaries to bin/macosx-intel (compiled on Darwin14)
* Added the docs/GET_PHYLOMARKERS_manual.md and docs/pics dirs/files, the latter with the pipeline flowchart
* Moved the manual-specific parts of the README.md file to docs/GET_PHYLOMARKERS_manual.md, providing links to it
23JAN18: * Added the new script estimate_pangenome_phylogenies.sh v1.0_23Jan18, along with the binaries it depends on:
- from phylip v3.69: consense pars seqboot; and from newick_utilities: nw_reroot nw_support
* This makes the GET_PHYLOMARKERS v2.0.1 suite complete, as described in the manuscript submitted to
Front. Microbiol. on Jan 15th, 2018
v2.0.1_22Jan18: * Fixed a parsing problem introduced due to changing file globings like *.fasta for ./*fasta
* extended the lists of models checked by check_IQT_DNA_models and check_IQT_PROT_models to match those currently
implemented by IQ-TREE and displayed in the -h & -H help_messages
* Made syntax more compliant with shellcheck standards
* Added more [ DEBUG -eq 1 ] && debugging messages, now also to signal entry and exit from each function
* set (reduced) min_supp_val=0.7 and nrep_IQT_searches=5
* Further improved documentation displayed with -H
v2.0_22Jan18: # >>> MAJOR UPGRADE: uses iqtree v1.6.1 as default search algorithm throughout the pipeline <<<
* now uses IQ-TREE (v1.6.1; Dec. 28th, 2017 release) as the default algorithm, both for gene- and species-tree searches!
* Based on the benchmark results presented in the GET_PHYLOMARKERS paper, model-selection is enabled by default
during gene tree searches, making use of the -fast flag. The number of models evaluated (hard-coded) depends on -T
* moved most functions to file lib/get_phylomarkers_fun_lib
* added new key functions: estimate_FT_gene_trees, compute_FT_gene_tree_stats estimate_IQT_gene_trees compute_IQT_gene_tree_stats
process_IQT_species_trees_for_molClock.
* added $search_thoroughness and $search_algorithm particles to dir_suffix
* thoroughly updated the extended Help text printed to STDOUT when -H is invoked
* updated the README.md file accordingly
* updated the help messages printed to STDOUT with -h
* added DISCLAIMER and LICENSE notices to the headers of the run_get_phylomarkers_pipeline.sh and get_phylomarkers_fun_lib files
v1.9.13_20Jan18: * updated temporal citation; minor cleanup of text printed to STDOUT;
* moved changes.txt to CHANGES.txt and formated the file consistently for better readability
v1.9.12_19Jan18: * changed -K 1 to -K (flag) to run mol-clock; replaced echo for msg within functions;
* changed all file globings like *.fasta for ./*fasta ... so names with dashes won't become options
v1.9.11_17Jan18: * added -slow to FastTree with -T high (supermatrix); added msg function to cleanup code; updated citation message at end;
* moved changes messages from top of script to this file
v1.9.10.1_1Jan18: # >>> MAJOR UPGRADE: this is the version using iqtree v1.6.1 with the -fast flag
on which some of the benchmark analyses reported in the paper were based on <<<
* changed iqtree-omp v1.5.6 to iqtreev.1.6.1 (first released on Dec 26th, 2017; fixed bug thanks to our report on Dec. 28th, 2017);
* added -fast flag to the model selection calls of IQ-TREE!!!
* added functions check_IQT_DNA_models() check_IQT_PROT_models() to check -S strings passed by user to IQ-TREE
* grep '^Gamma20LogLk' instead of ML_Lengths2 also for protein data
* Changed spr_length=8 to spr_length=10 to match the default value of FastTree 2.1.10
v 1.9.9.2_31Dic17: * grep '^Gamma20LogLk' instead of ML_Lengths2 to print the FastTree lnL score to STDOUT
v1.9.9.1_23Dic17: renamed the iqtree binary to iqtree-omp to be explicit about the multicore version
v1.9.9.0_22Dic17: aded IQ-tree searching option for the concatenated alignment, controlled with new options -A, -N and -S
v1.9.8.4_17Nov17: improved/expanded -h help message; thorough and consistent tidying of directories; cleanup of code comments
v1.9.8.3_17Nov17: added get_homologues manual url to ERROR message to better assist users
v1.9.8.2_17Nov17: another sanity check: make sure there are equal number of fna and faa files to start working on
v1.9.8.1_17Nov17: fixed name of the add_labels2tree.pl in one of the calls with -R 1 -t PROT and code cleanup
to avoid problems with old versions of locally insalled binaries or scripts, we set $bindir and $distrodir in front of PATH
v1.9.7.5_15Nov17: added check for minimal versions of clustalo FastTree parallel and paup
v1.9.7.5_15Nov17: fixed check for $HOME/bin in $PATH
v1.9.7.4_15Nov17: matched paths of results dir and log
v1.9.7.3_14Nov17: fixed bug/typo in get_script_PID()
v1.9.7.2_14Nov17: popGen dir cleanup + top_X_markers dir for -R 1 -t PROT
v1.9.7.1_14Nov17: added strain composition check on f?aed files to make sure each one contains a single instance for the same number of strains
This is a critical check to avoid problems with inparalogues in some fastas if get_homologues.pl was run with -t X and without -e.
Trees could be mislabeled in that case, and some alignments will most likely contain a different strain composition,
generating a chimaeric concatenated file; A useful ERROR message is printed (run compare_clusters.pl with -t NUM_GENOMES.
Also calls run_parallel_cmmds.pl with parallel --gnu; some code cleanup
v1.9.7_14Nov17: now using parallel instead of pexec binary, which failed in CentOS
v1.9.6.4_12Nov17: prints total number of trees with < 1 internal branches;
corrected regex so that all trees and alns with < 1 int br are removed (do not pass to downstream analyses)
v1.9.6.3_11Nov17:added -R${runmode} to dir_suffix
v1.9.6.2_11Nov17: all DEBUG|VERBOSE messages written to logfile
v1.9.6_11nov17: fixed code that writes Phi-test results to Phi_results_11nov17.tsv;
prints the Warning: will remove file* because it has < 5 branches! to log file (no only STDOUT)
v1.9.5_8Nov17: the number of sequences in input FASTA files is checked upfront
v1.9.5_7Nov17: prints a logfile for FastTree runs and captures the lnL value of the corresponding tree;
more detailed info reported on alignments with too few informative sites for the Phi test to be work on
v1.9.4_6Nov17: added error checking for Phi run when sequences lack enough polymorphism for the test to work
v1.9.3_10Oct17: improved error message when kdetrees fail
v1.9.2_1Jun17: code cleanup > removed comments and print_development_notes(); append the $distrodir/lib/perl to PERL5LIB and export
v1.9.1_1Jun17: more sensible directory cleanup ...
v1.9_31May17: improved progress messages and directory cleanup ...
v1.8.4_30May17: prepended $ditrodir/ to perl scripts that use FindBin; so that it can find the required libs in $ditrodir/lib/perl
Added -n $n_cores flag, which is passed to run_parallel_cmmds.pl '' $n_cores, so that it runs on MacOSX!!! <<< Thanks Alfredo!
automatically set n_cores=no_proc if [ -z $n_cores ]
v1.8.1_24May17: fixed problmes with @INC searching of rename.pl by prepending $distrodir/rename.pl
v1.8_23May17: Added R code in count_tree_branches() to use local_lib if ape is not installed systemwide;
searches and prints the number of available cores on HOSTNAME
exports R_LIBS="$R_LIBS:$distrodir/lib/R" to fix issues with library paths in R scripts
v1.7_17May17: Changed exit for Warning when pal2nal does not return a codon alignment! so that the pipeline can proceed
v1.6_15May17: added extensive debugging messages throughout the code for easier debugging; activated the -V flag
v1.5: fixed set_bindirs and check_homebinpath(), to export to PATH;
fully tested on a new linux account without $HOME/bin dir using freshly cloned distro; Note that R and Perl libs were already in ENV
v1.3: further refinement in set_bindirs() and check_homebinpath(), validated on yaxche; minor code cleanup
v1.2_13May17: refined the logic of set_bindirs(); added get_start_time(); improved error checking code, including get_script_PID()
fixed a bug in -t PROT
v1.1_13May17: Major update to facilitate installation by users: added set_pipeline_environment(), check_homebinpath(), set_bindirs(),
which ckeck and setup the ENVIRONMENT for the pipeline. Added new bin/ and pop_gen_tables/ dirs for consistency.
The bin/linux dir contains the 2cnd party binaries compiled for 64bit linux machines. Need to compile for darwin
v1.0_11May17: git commited; first version on GitHub: https://github.com/vinuesa/get_phylomarkers
v0.9_10May17: important speed improvement due to running pal2nal.pl and run_parallel_molecClock_test_with_paup.sh
in parallel with run_parallel_cmmds.pl
v0.8_9May17: added -R 1|2, for Phylo|popGen, based on popGen_summStats.pl and pre-computed Tajima's D and Fu-Li crit value tables
with missing values predicted by lm() in R, based on the original critical values in Tajima's 89 and Fu and Li 1993 papers
The critical CI values are gathered by get_critical_TajD_values() and get_critical_FuLi_values()
v0.7_2May17: added -e min_no_ext_branches -c $codontable and -C print_codontable() for pal2nal.pl, which is now run from within a loop
runs compute_suppValStas_and_RF-dist.R in the top_X_markers dir
v0.6_1May17: added get_script_PID() and count_tree_branches(); added -t BOTH
v0.5_29April17: Added -t PROT
v0.4_27April17: Added print_usage_notes()