Performance comparison of old vs new loki code

[van-truong]

Need to quantify how the code changes are improving user experience of loki-build

• network speed (mbps) to download data from each source
• per file download time
• add that^ up to get per source download time (some sources have multiple files)
• per file processing time
• add that^ up to get per source processing
• total time to process each source locally
• start to finish time to compile into SQLite database

[van-truong]

Times from July 28th loki-build run by Brandon (parallel download but no changes to processing yet)

Data Source	Download Start Time	Download End Time	Download Duration (hh:mm:ss:ms)	Processing Start Time	Processing End Time	Processing Duration (hh:mm:ss:ms)
Total loki-build time	07/28/2024 5:24:07	07/28/2024 16:33:33	11:09:00:26	07/28/2024 16:34:33	07/29/2024 3:48:14	11:14:00:19
biogrid total (112.76 MB)	07/28/2024 5:24:07	07/28/2024 5:24:15	00:00:00:08	07/28/2024 16:34:33	07/28/2024 16:36:06	00:02:00:27
chainfiles total (27.15 MB)	07/28/2024 5:24:07	07/28/2024 5:24:39	00:00:00:32	07/28/2024 16:36:06	07/28/2024 16:36:58	00:00:00:52
dbsnp total (13.13 GB)	07/28/2024 5:24:07	07/28/2024 5:36:22	00:12:00:15	07/28/2024 16:36:58	07/28/2024 18:18:24	01:42:00:34
entrez total (2.76 GB)	07/28/2024 5:24:07	07/28/2024 5:26:51	00:02:00:44	07/28/2024 18:18:24	07/28/2024 18:39:10	00:21:00:14
go total (45.83 MB)	07/28/2024 5:24:07	07/28/2024 5:24:08	00:00:00:01	07/28/2024 18:39:10	07/28/2024 18:39:56	00:00:00:46
gwas total (277.22 MB)	07/28/2024 5:24:07	07/28/2024 5:27:25	00:03:00:18	07/28/2024 18:39:56	07/28/2024 18:41:37	00:02:00:19
mint total (85.42 MB)	07/28/2024 5:24:07	07/28/2024 5:27:10	00:03:00:03	07/28/2024 18:41:37	07/28/2024 18:42:22	00:01:00:15
oreganno total (52.69 MB)	07/28/2024 5:24:07	07/28/2024 5:24:35	00:00:00:28	07/28/2024 18:42:22	07/28/2024 18:44:16	00:02:00:06
pfam total (22.83 GB)	07/28/2024 5:24:07	07/28/2024 16:33:33	11:09:00:26	07/28/2024 18:44:16	07/28/2024 19:08:07	00:24:00:09
pharmgkb total (2.99 MB)	07/28/2024 5:24:07	07/28/2024 5:24:10	00:00:00:03	07/28/2024 19:08:07	07/28/2024 19:08:35	00:00:00:28
reactome total (216.63 MB)	07/28/2024 5:24:07	07/28/2024 5:24:21	00:00:00:14	07/28/2024 19:08:35	07/28/2024 19:09:04	00:01:00:31
ucsc_ecr total (8.89 GB)	07/28/2024 5:24:07	07/28/2024 5:36:22	00:12:00:15	07/28/2024 19:09:04	07/29/2024 3:48:14	08:39:00:10