feat: Add from-genesis subcommand (#342)

* feat: Add access_key/account adapters for storing genesis

* feat: Add `from-genesis` command without implementation

* feat: Download genesis file

* feat: Store genesis accounts/access_keys

* refactor: Initialize lake framework after processing genesis

* docs: Add info on `from-genesis` to `README.md`

* feat: Add `betanet` s3 genesis url

* refactor: Replace manual chain id conversion with `impl ToString`

* feat: Show `genesis.json` download progress in logs

* feat: Use `genesis.json` on disk if exists

* chore: Remove unused `actix-rt`

* chore: Remove unused imports

* refactor: Stream genesis record processing

* doc: Add note in `README.md` about RAM requirements for testnet genesis

* fix: Set `target` for genesis logs

* refactor: Remove genesis chain ID file prefix

* refactor: Store `genesis.json` next to executable

* refactor: Log genesis download every ~100kb

* doc: Update `CHANGELOG.md`

README.md

@@ -86,10 +86,11 @@ Command to run NEAR Indexer for Explorer have to include the chain-id and start
 You can choose NEAR Indexer for Explorer start options:
  - `from-latest` - start indexing blocks from the latest finalized block
  - `from-interruption` - start indexing blocks from the block NEAR Indexer was interrupted last time but earlier for `<number_of_blocks>` if provided
+ - `from-genesis` - download and store accounts/access keys in genesis file and start indexing from the genesis block
  - `from-block --height <block_height>` - start indexing blocks from the specific block height
 
 #### Storing genesis file
-Unlike the original NEAR Indexer for Explorer you **can't** tell Indexer to store data from genesis (Accounts and Access Keys) by adding key `--store-genesis` to the `run` command. So please, ensure you took care about the genesis data in your database in order this indexer to work properly. This capability will be implemented eventually, it's progress can be tracked here: #327.
+When starting Indexer for Explorer with `from-genesis`, the entire genesis file will be loaded in to memory before iterating the stored accounts/access keys. As of writing this, `mainnet` and `betanet` both have relatively small genesis files (<1GB), but the `testnet` file size is around 5GB. Therefore, if you intend to store the `testnet` genesis records, make sure that your system has sufficient RAM to hande the memory load.
 
 #### Strict mode
 NEAR Indexer for Explorer works in strict mode by default. In strict mode, the Indexer will ensure parent data exists before storing children, infinitely retrying until this condition is met. This is necessary as a parent (i.e. `block`) may still be processing while a child (i.e. `receipt`) is ready to be stored. This scenario will likely occur if you have not stored the genesis file or do not have all data prior to the block you start indexing from. In this case, you can disable strict mode to store data prior to the block you are concerned about, and then re-enable it once you have passed this block.

database/Cargo.toml

@@ -6,6 +6,8 @@ edition = "2021"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
+actix = "0.13.0"
+actix-rt = "2.2.0"
 anyhow = "1.0.51"
 base64 = "0.11"
 bigdecimal = "=0.1.0"
@@ -25,6 +27,7 @@ actix-diesel = { git = "https://github.com/frol/actix-diesel", rev = "3a001986c8
 near-indexer-primitives = "0.16.0"
 near-primitives = "0.16.0"
 near-crypto = "0.16.0"
+near-chain-configs = "0.16.0"
 
 [features]
 default = []

database/src/adapters/access_keys.rs

@@ -4,6 +4,7 @@ use actix_diesel::dsl::AsyncRunQueryDsl;
 
 use diesel::{ExpressionMethods, PgConnection, QueryDsl};
 use futures::try_join;
+use tracing::info;
 
 use crate::models;
 use crate::schema;
@@ -145,3 +146,26 @@ pub async fn handle_access_keys(
 
     Ok(())
 }
+
+pub(crate) async fn store_access_keys_from_genesis(
+    pool: actix_diesel::Database<PgConnection>,
+    access_keys_models: Vec<models::access_keys::AccessKey>,
+) -> anyhow::Result<()> {
+    info!(
+        target: crate::EXPLORER_DATABASE,
+        "Adding/updating {} access keys from genesis...",
+        access_keys_models.len(),
+    );
+
+    crate::await_retry_or_panic!(
+        diesel::insert_into(schema::access_keys::table)
+            .values(access_keys_models.clone())
+            .on_conflict_do_nothing()
+            .execute_async(&pool),
+        10,
+        "Failed to store AccessKeys from genesis".to_string(),
+        &access_keys_models
+    );
+
+    Ok(())
+}

database/src/adapters/accounts.rs

@@ -6,6 +6,7 @@ use bigdecimal::BigDecimal;
 
 use diesel::{BoolExpressionMethods, ExpressionMethods, PgConnection, QueryDsl};
 use futures::try_join;
+use tracing::info;
 
 use crate::models;
 use crate::schema;
@@ -249,3 +250,26 @@ pub async fn get_lockup_account_ids_at_block_height(
                 .collect()
         })
 }
+
+pub(crate) async fn store_accounts_from_genesis(
+    pool: actix_diesel::Database<PgConnection>,
+    accounts_models: Vec<models::accounts::Account>,
+) -> anyhow::Result<()> {
+    info!(
+        target: crate::EXPLORER_DATABASE,
+        "Adding/updating {} accounts from genesis...",
+        accounts_models.len(),
+    );
+
+    crate::await_retry_or_panic!(
+        diesel::insert_into(schema::accounts::table)
+            .values(accounts_models.clone())
+            .on_conflict_do_nothing()
+            .execute_async(&pool),
+        10,
+        "Failed to store Accounts from genesis".to_string(),
+        &accounts_models
+    );
+
+    Ok(())
+}