[WIP] add code & workflow to update metagenome catalog

[ctb]

Bring in some of the catalog stuff from https://github.com/bluegenes/2022-magsearch-tr/ per ctb/magsearch#13 (comment).

[ctb]

currently breaks on downloading from NCBI -

...
Reusing existing connection to www.ncbi.nlm.nih.gov:443.
HTTP request sent, awaiting response... 400 Bad Request. Both list of IDs and query_key are empty
2022-12-18 06:05:19 ERROR 400: Bad Request. Both list of IDs and query_key are empty.

[luizirber]

currently breaks on downloading from NCBI -

...
Reusing existing connection to www.ncbi.nlm.nih.gov:443.
HTTP request sent, awaiting response... 400 Bad Request. Both list of IDs and query_key are empty
2022-12-18 06:05:19 ERROR 400: Bad Request. Both list of IDs and query_key are empty.

yes, the SRA discontinued that API (I don't think it was ever public...)

Official method is to use entrez to download it, something like
esearch -db sra -query '"METAGENOMIC"[Source] NOT amplicon[All Fields]' | efetch -format runinfo -mode text > catalog.csv

Main issue is that downloading it all... kind of breaks efetch. I can do the daily/small date ranges download, but for all the matches it always breaks after some time.

[luizirber]

Might need to use bigquery, but not sure how to automate that outside GCP: https://edwards.flinders.edu.au/identifying-metagenomes-from-the-sra-in-the-cloud/