Skip to content

Commit 959fe08

Browse files
docs: Document how to configure nested stream maps values with environment variables in Meltano (#2796)
1 parent 87da56d commit 959fe08

File tree

1 file changed

+28
-0
lines changed

1 file changed

+28
-0
lines changed

docs/stream_maps.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -778,3 +778,31 @@ the `key_properties` in an extract-load pipeline. For instance, it is common to
778778
"append-only" loading behavior in certain targets, as may be required for historical reporting. This does not change the
779779
underlying nature of the `primary_key` configuration in the upstream source data, only how it will be landed or deduped
780780
in the downstream source.
781+
782+
783+
### Q: How do I use Meltano environment variables to configure stream maps?
784+
785+
**Answer:** Environment variables in Meltano can be used to configure stream maps, but you first need to add the corresponding settings
786+
to your plugins `settings` option. For example:
787+
788+
```yaml
789+
plugins:
790+
extractors:
791+
- name: tap-csv
792+
variant: meltanolabs
793+
pip_url: git+https://github.com/MeltanoLabs/tap-csv.git
794+
settings:
795+
- name: stream_maps.customers.email
796+
- name: stream_maps.customers.email_domain
797+
- name: stream_maps.customers.email_hash
798+
- name: stream_maps.customers.__else__
799+
- name: stream_maps.stream_map_config
800+
```
801+
802+
Then, you can set the following environment variables:
803+
804+
```shell
805+
TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_DOMAIN='email.split("@")[-1]'
806+
TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_HASH='md5(config["hash_seed"] + email)'
807+
TAP_CSV_STREAM_MAP_CONFIG_HASH_SEED='01AWZh7A6DzGm6iJZZ2T'
808+
```

0 commit comments

Comments
 (0)