Readonly Pipeline

[haohangyan]

This PR contains all the code we need for the readonly pipeline.

When we run the command bash indra_db/readonly_dumping/test_bash.sh, the pipeline will start running.
An overview of the pipeline:

• Set up environment variables and database passwords.
• Get file paths for required initial dump files and verify them.
• Dump raw statements, reading text content meta, and text refs principal from the principal database.
• Get all files that will be loaded into the readonly database.
• Recreate the local database and import data using the readonly_dumping script.
• (Not tested) create and remove a local database dump file after uploading it to S3.
• (Not tested) upload an end-date file to S3 and restore the dump to a readonly instance using pg_restore.

The pipeline contains two main parts: export_assembly and readonly_dumping.

Some main stages in the export assembly include:

1. Running the knowledgebase pipeline
2. Running statement distillation
3. Running preprocessing
4. Merging processed knowledgebase statements with processed raw statements
5. Running grounding and deduplication
6. Calculating refinements
7. Calculating the belief score

In the readonly_dumping part, we dump the tables into the local Postgres database in the following order: “belief”, “raw_stmt_src”, “reading_ref_link”, “evidence_counts”, “pa_agent_counts”, “mesh_concept_ref_counts”, “mesh_term_ref_counts”, “name_meta”, “text_meta”, “other_meta”, “source_meta”, “agent_interactions”, “fast_raw_pa_link”, “raw_stmt_mesh_concepts”, “raw_stmt_mesh_terms”, “mesh_concept_meta”, “mesh_term_meta”.

Some tasks that need to be completed:

• When the final readonly database is generated, the bash file needs a script to upload the database to an S3 bucket. We need to determine the storage destination for where the database should be placed.
• Everything is currently running using the test_bash.sh. This will be fixed so we can use readonly_dumping_bash.sh soon.

Depends on sorgerlab/indra#1460.

[kkaris]

There are updates needed by both of us @haohangyan, in general I think it looks good