[Docs]: Document how to configure nested stream maps values with environment variables in Meltano

[edgarrmondragon]

Documentation type

How-to guides

Description

Example adapted from the meltano-map-transform README:

plugins:
  extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-csv.git
    settings:
    - name: stream_maps.customers.email
    - name: stream_maps.customers.email_domain
    - name: stream_maps.customers.email_hash
    - name: stream_maps.customers.__else__
    - name: stream_maps.stream_map_config
    # Note that these don't work: https://github.com/meltano/meltano/issues/6382
    config:
      stream_maps.customers.email: null
      stream_maps.customers.__else__: null

TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_DOMAIN='email.split("@")[-1]'
TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_HASH='md5(config["hash_seed"] + email)'
TAP_CSV_STREAM_MAP_CONFIG_HASH_SEED='01AWZh7A6DzGm6iJZZ2T'

[aaronsteers]

Cross linking this as a workaround/mitigation of:

• Nested config values in setting arrays and objects don't have environment variables expanded meltano#3171 and
• feature: Support env_var expansion in stream_maps config meltano#7151

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant

[edgarrmondragon]

User asking about something like this in Slack:

Are there any good examples of how to use the MELTANO_MAP_TRANSFORMER_STREAM_MAPS environment variable. Meltano Hub mentions it but it’s not immediately apparent to me what it overwrites because there is a separate stream_maps key for each mapping in the meltano.yml file which begs the question: when I use this variable which mapping is it going to control?

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant