From 09859a71da489f6ed49ff782169fc030428e995d Mon Sep 17 00:00:00 2001 From: Marigold Date: Fri, 6 Sep 2024 13:36:45 +0200 Subject: [PATCH 1/4] :scroll: Add docs for multi-dim indicators From d1f462a7106c1336bcb18eb1e6547501801069e7 Mon Sep 17 00:00:00 2001 From: Marigold Date: Fri, 6 Sep 2024 13:36:51 +0200 Subject: [PATCH 2/4] wip --- docs/architecture/workflow/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/architecture/workflow/index.md b/docs/architecture/workflow/index.md index 00fa9d7b2c9..bb231e72fb8 100644 --- a/docs/architecture/workflow/index.md +++ b/docs/architecture/workflow/index.md @@ -290,6 +290,8 @@ ds_explorer.save() TODO + + ### Exporting data to GitHub One common use case for the `export` step is to commit a dataset to a GitHub repository. This is useful when we want to make a dataset available to the public. The pattern for this looks like this: From 89615ca7a62efa642a824f2f18cd1b2a1c9e6c8f Mon Sep 17 00:00:00 2001 From: Marigold Date: Fri, 6 Sep 2024 13:49:00 +0200 Subject: [PATCH 3/4] wip --- docs/architecture/workflow/index.md | 74 ++++++++++++++++++++++++++++- docs/guides/sharing-external.md | 4 +- 2 files changed, 75 insertions(+), 3 deletions(-) diff --git a/docs/architecture/workflow/index.md b/docs/architecture/workflow/index.md index bb231e72fb8..089f77fc976 100644 --- a/docs/architecture/workflow/index.md +++ b/docs/architecture/workflow/index.md @@ -288,8 +288,80 @@ ds_explorer.save() ### Creating multi-dimensional indicators -TODO +Multi-dimensional indicators are powered by a configuration that is typically created from a YAML file. The structure of the YAML file looks like this: + +```yaml +definitions: + table: grapher/energy/2024-05-08/primary_energy_consumption/primary_energy_consumption + +title: + title: Energy use + titleVariant: by energy source +defaultSelection: + - World + - Europe + - Mexico +topicTags: + - Energy +dimensions: + - slug: source + name: Energy source + choices: + - slug: all + name: All sources + group: Aggregates + description: Total energy use + - slug: fossil + name: Fossil fuels + group: Aggregates + description: The sum of coal, oil and gas + ... + - slug: metric + name: Metric + choices: + - slug: total + name: Total consumption + description: The amount of energy consumed nationally per year + - slug: per_capita + name: Consumption per capita + description: The average amount of energy each person consumes per year + ... + +views: + - dimensions: + source: all + metric: total + indicators: + y: "{definitions.table}#primary_energy_consumption__twh" + - dimensions: + source: all + metric: per_capita + indicators: + # This could be a list of indicators + y: "{definitions.table}#primary_energy_consumption_per_capita__kwh" +``` + +`dimensions` field specifies selectors and `views` field specifies views for the selection. Since there are tons of possible configurations, `views` are usually generated programatically. You can also use a combination of manually defined views and generated views. See module `etl.multidim` for helper functions or examples from `etl/steps/export/multidim/`. + +The export step loads the YAML file, adds `views` to the config and then calls function +```python +def run(dest_dir: str) -> None: + engine = get_engine() + + # Load YAML file + with open(CURRENT_DIR / "energy.yml") as istream: + config = yaml.safe_load(istream) + config['views'] = ... + + multidim.upsert_multidim_data_page("mdd-energy", config, engine) +``` + +To see the multi-dimensional indicator in Admin, run +```bash +etlr export://explorers/multidim/latest/name --export +``` +and see the preview at http://staging-site-my-branch/admin/grapher/mdd-name. ### Exporting data to GitHub diff --git a/docs/guides/sharing-external.md b/docs/guides/sharing-external.md index 9acd4e77420..e836ab90988 100644 --- a/docs/guides/sharing-external.md +++ b/docs/guides/sharing-external.md @@ -11,6 +11,6 @@ Sometimes it's useful to share our work with external people to get feedback bef To share explorers with the public, follow these steps: 1. Create a branch wiht `-public` suffix (thus creating staging server). -2. Publish your explorer. -3. Commit something to trigger manual deploy (this is only needed once after publishing). +2. Set `isPublished` to `true` in your explorer configuration. +3. Trigger manual deploy from Admin (this is only needed to do once) or commit to trigger it automatically. 4. Share your explorer with public on e.g. https://staging-site-my-branch.tail6e23.ts.net/explorers/my-explorer. From aa7ef38ee67d87a1ee7c53672e410345f3e8d646 Mon Sep 17 00:00:00 2001 From: Marigold Date: Fri, 6 Sep 2024 13:52:55 +0200 Subject: [PATCH 4/4] wip --- docs/architecture/workflow/index.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/architecture/workflow/index.md b/docs/architecture/workflow/index.md index 089f77fc976..3a7b693682f 100644 --- a/docs/architecture/workflow/index.md +++ b/docs/architecture/workflow/index.md @@ -341,9 +341,12 @@ views: y: "{definitions.table}#primary_energy_consumption_per_capita__kwh" ``` -`dimensions` field specifies selectors and `views` field specifies views for the selection. Since there are tons of possible configurations, `views` are usually generated programatically. You can also use a combination of manually defined views and generated views. See module `etl.multidim` for helper functions or examples from `etl/steps/export/multidim/`. +The `dimensions` field specifies selectors, and the `views` field defines views for the selection. Since there are numerous possible configurations, `views` are usually generated programmatically. However, it's a good idea to create a few of them manually to start. + +You can also combine manually defined views with generated ones. See the `etl.multidim` module for available helper functions or refer to examples from `etl/steps/export/multidim/`. Feel free to add or modify the helper functions as needed. + +The export step loads the YAML file, adds `views` to the config, and then calls the function. -The export step loads the YAML file, adds `views` to the config and then calls function ```python def run(dest_dir: str) -> None: engine = get_engine() @@ -352,16 +355,19 @@ def run(dest_dir: str) -> None: with open(CURRENT_DIR / "energy.yml") as istream: config = yaml.safe_load(istream) + # Programatically generate views config['views'] = ... multidim.upsert_multidim_data_page("mdd-energy", config, engine) ``` To see the multi-dimensional indicator in Admin, run + ```bash -etlr export://explorers/multidim/latest/name --export +etlr export://multidim/energy/latest/energy --export ``` -and see the preview at http://staging-site-my-branch/admin/grapher/mdd-name. + +and check out the preview at http://staging-site-my-branch/admin/grapher/mdd-name. ### Exporting data to GitHub