Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Tap fails if stream maps config contains a key for an unselected stream #1942

Open
1 task
edgarrmondragon opened this issue Sep 7, 2023 · 1 comment
Open
1 task
Labels

Comments

@edgarrmondragon
Copy link
Collaborator

edgarrmondragon commented Sep 7, 2023

Singer SDK Version

0.31.1

Is this a regression?

  • Yes

Python Version

NA

Bug scope

Mapping (stream maps, flattening, etc.)

Operating System

NA

Description

Consider the countries sample in this repo.

  1. Generate the Singer catalog for the tap:

    $ poetry run python samples/sample_tap_countries --discover > countries.catalog.json
  2. Create a stream maps config for the tap. This one creates slugs for both the continent and country name:

    $ jq < countries.stream_maps.json
    {
        "stream_maps": {
            "continents": {
                "slug": "str(name.lower().replace(' ', '-'))"
            },
            "countries": {
                "slug": "str(name.lower().replace(' ', '-'))"
            }
        }
    }
  3. Run the tap with the generated catalog and stream maps config:

    $ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl

    This should work as expected.

  4. Now, open countries.catalog.json and edit it to deselect the countries stream

    $ jq '.streams[1].metadata[-1]' < countries.catalog.json
    {
      "breadcrumb": [],
      "metadata": {
        "inclusion": "available",
        "selected": false,   <----- this right here needs to be 'false'
        "selected-by-default": true,
        "table-key-properties": [
          "code"
        ]
      }
    }
  5. Run the tap again:

    $ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl

    You'll end up with

    singer_sdk.exceptions.StreamMapConfigError: Invalid key properties for 'countries': [code]. Property 'code' was not detected in schema.
    

Code

No response

Copy link

stale bot commented Sep 6, 2024

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@stale stale bot added the stale label Sep 6, 2024
@stale stale bot closed this as completed Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant