From ec774c1a33ca18f076fc7e6d616c37e37a887b14 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Edgar=20Ram=C3=ADrez=20Mondrag=C3=B3n?= Date: Tue, 3 Dec 2024 17:18:38 -0600 Subject: [PATCH] docs: Document how to configure nested stream maps values with environment variables in Meltano --- docs/stream_maps.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/stream_maps.md b/docs/stream_maps.md index a47539158..7c65b88c8 100644 --- a/docs/stream_maps.md +++ b/docs/stream_maps.md @@ -778,3 +778,31 @@ the `key_properties` in an extract-load pipeline. For instance, it is common to "append-only" loading behavior in certain targets, as may be required for historical reporting. This does not change the underlying nature of the `primary_key` configuration in the upstream source data, only how it will be landed or deduped in the downstream source. + + +### Q: How do I use Meltano environment variables to configure stream maps? + +**Answer:** Environment variables in Meltano can be used to configure stream maps, but you first need to add the corresponding settings +to your plugins `settings` option. For example: + +```yaml +plugins: + extractors: + - name: tap-csv + variant: meltanolabs + pip_url: git+https://github.com/MeltanoLabs/tap-csv.git + settings: + - name: stream_maps.customers.email + - name: stream_maps.customers.email_domain + - name: stream_maps.customers.email_hash + - name: stream_maps.customers.__else__ + - name: stream_maps.stream_map_config +``` + +Then, you can set the following environment variables: + +```shell +TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_DOMAIN='email.split("@")[-1]' +TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_HASH='md5(config["hash_seed"] + email)' +TAP_CSV_STREAM_MAP_CONFIG_HASH_SEED='01AWZh7A6DzGm6iJZZ2T' +```