Skip to content

Commit

Permalink
Update data_platform_with_deep_indexed_data_and_staking_reports.md
Browse files Browse the repository at this point in the history
  • Loading branch information
bigslimdogg authored Jul 28, 2023
1 parent 560798a commit 08052ab
Showing 1 changed file with 11 additions and 23 deletions.
34 changes: 11 additions & 23 deletions data_platform_with_deep_indexed_data_and_staking_reports.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,36 +209,20 @@ We suggest proposal with 1 milestone to:
### Milestone 1 — Forked Superset with indexed data

- **Estimated duration:** 1.5 month
- **FTE:** 200 FTE hours [100 USD/hr]
- **FTE:** 3.5
- **Costs:** 20,000 USD

| Number | Deliverable | Specification |
| --- | --- | --- |
| 0a. | License | Apache 2.0 |
| 0b. | Documentation | We will provide a documentation page about how to
- self-host indexer
- how to setup airflow
- how to connect it with DWH
- how to setup DWH
- how to setup UI (BI tool) |
| 0b. | Documentation | We will provide a documentation page about how to launch docker’s files for ETL (indexer + orchestration), DWH connection, launching UI (BI tool). |
| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. |
| 0d. | Docker | We will provide a Dockerfile(s) that can be used to run indexer locally, set up data base (DWH), and run BI. |
| 1. | ETL component development | We already develop Mbelt, so this step is for:
- deploy indexer for public data platform
- develop the airflow DAG to run data pipeline
- start to write data to DWH |
| 2. | Public data in DWH | We has public project in bigQ already, so we need:
- make Polkadot/Kusama/Moonbeam/Moonriver data public
- provide test for pipeline
- publish SLI/SLO dashboard of refreshing the data
- Staking data marts development and publishing |
| 3. | Forking Superset | - Diving into superset’s arhcitecture of creating new functions etc.
- Modifying Sign In page, creating additional actions in controllers
- Main page modifications
- Dashboard tab modification
- Charts tab modification
- SQL lab tab modification
- User’s settings modification |
| 1. | Develop of ETL component | The result of developing ETL component contains the docker with Mbelt (indexer) and Airflow with DAGs to write indexer’s output into DWH (data base). |
| 2. | Publish DWH data | Our goal for publishing data is: providing raw (indexer’s data) data into public bigQ project with the ability to query and use this data for public users. The chains are Polkadot, Kusama, Moonbeam, Moonriver.
We would also provide reports about staking on top of raw data in chains above.
The additional result is SLI/SLO dashboard of ETL process to provide more transparency for users. |
| 3. | Forking Superset | Forking Superset will allow us to modify sign-in page to provide new users to join. We will make some changes on UI side to improve user’s experience as well. |

## **Future plans**

Expand All @@ -259,3 +243,7 @@ Not applicable****
**How did you hear about the Grants Program?**

Web3 Foundation Website and previous grants: [Multiblockchain ETL](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/Multiblockchain%20ETL.md)

**How did you hear about the Grants Program?**

Web3 Foundation Website and previous grants: [Multiblockchain ETL](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/Multiblockchain%20ETL.md)

0 comments on commit 08052ab

Please sign in to comment.