Letting tap developers unselect certain streams in the default catalog

[edgarrmondragon]

The SDK tries to be helpful to people new to singer and not familiar with catalogs, by selecting all available streams by default in taps with known streams, which is the case of taps for most API sources.

However there are situations in which a Tap developer may which to change this behavior for certain streams and mark them as not selected in the default catalog. Users would still be able to select those streams normally by editing the catalog file or using Meltano's select feature.

A few scenarios where this would be useful come to mind:

1. A stream is only available to premium or paying users of the API, so it would fail under "normal" circumstances.
2. A stream that outputs a large volume of data was added after the Tap was initially released. Developers may want to make this stream opt-in for existing users, so their syncs don't suddenly start running for much longer times.

Out in the wild

• feat: Add traffic streams MeltanoLabs/tap-github#193

  The traffic streams require additional permissions on the GitHub token.

• initial new stream with integration logs MeltanoLabs/tap-slack#14

  Similarly, the new stream requires admin privileges.

• feat: Add tags endpoint MeltanoLabs/tap-stackexchange#96

  The tags endpoint is massive in some sites. Would be nice to make it opt-in.

• https://github.com/MeltanoLabs/tap-gitlab

  A bunch of streams are opt-in, based on config.

Proposal

This would be as easy as adding a class attributes to the base stream class:

class Stream:
   selected_by_default: bool = True

Then the generated catalog metadata would check this attribute to decide whether the stream is selected when there's no input catalog (i.e. --catalog is not passed or the Tap is run in discovery mode).

I'm curious what folks think.

cc @kgpayne @visch

[visch]

@edgarrmondragon wow that's a nice simple way of doing it. This would be a nice feature I don't have it come up too often but it does pop up now and again

Another scenario that comes up with all SQL taps is the infromation-schema esk tables. re MeltanoLabs/tap-postgres#54 , in my opinion we should be able to select those tables if we want them, but it doesn't make a ton of sense to have them selected by default as most of the time you don't actually want the information schema information.

[edgarrmondragon]

Ah that's a good idea! If there's a way in each SQL db to tell if a table is coming from the internal catalog we could use that to set the selection to False by default 😁

[aaronsteers]

Just to chime in here with a couple thoughts. Yes, I definitely would agree this would be super helpful - and the use case you mentioned about very large streams being added and not wanting these new streams to slow down users who are running stable productions in production using the default selection rules.

The StreamMetadata class in _singerlib.catalog does have selected_by_default but no easy way to set it as of now. The suggested approach of setting an explicit but optional Stream.selected_by_default = False declaration sounds like a very nice and usable approach, and is nicely parallel to the existing Stream.selected property.

Related:

• stream-level issue: [Feature]: Support de-selecting a stream by default #1203
• property-level: Improve support for selected-by-default option on Property - [opened] #579 (had a GitLab PR here from @laurentS but unfortunately wasn't completed+merged prior to the GitHub migration).

Thanks for raising, @edgarrmondragon !

[edgarrmondragon]

I knew I had logged an issue at some point 😅. Thanks for linking #1203!

[kgpayne]

@edgarrmondragon this is a great idea 🙌 I had a similar thought when working on the test framework, with a slightly different approach. Given that the Meltano select capabilities (as well as schema overrides and metadata injection) were implemented for a world before/without the SDK, I think it would be reasonable to add them as SDK native functionality. We'd still need that functionality in Meltano to cater for non-SDK taps, but eventually it might make sense to remove it or provide a legacy-tap wrapper.

Adding these capabilities to the standard tap config would allow users of SDK-based taps to apply selection in config, including the SAMPLE_CONFIG used for testing, and allow developers to select just the streams they want to work on. WDYT?

Edit: Idea more coherently would be to allow developers to supply default selection values in config as part of porting selection in general to the SDK. This might solve the testing use case and the newly added stream problem together (by supplying a default selection and enabling/promoting selection overrides other than *.*) 😅

[edgarrmondragon]

@edgarrmondragon this is a great idea 🙌 I had a similar thought when working on the test framework, with a slightly different approach. Given that the Meltano select capabilities (as well as schema overrides and metadata injection) were implemented for a world before/without the SDK, I think it would be reasonable to add them as SDK native functionality.

@kgpayne That's a much bigger lift, though and I'm happy for that functionality to remain in Meltano. If we want to publish something that would make the benefits available to the wider Singer community we could think about something like https://github.com/chrisgoddard/singer-discover.

Adding these capabilities to the standard tap config would allow users of SDK-based taps to apply selection in config, including the SAMPLE_CONFIG used for testing, and allow developers to select just the streams they want to work on. WDYT?

I'm not sold on that option. For one, changing config would affect the result of --discover in cases where it really shouldn't (i.e. the streams are known beforehand). It also introduces complexity in how catalog and configuration inputs interact to decide if a stream is selected.

[kgpayne]

@edgarrmondragon yeh agree it's a much broader change, and also that the two changes aren't mutually exclusive either. The reason I mention is that this change address some stream selection challenges, but not the ones I've run into 😅

Apart from the "new streams added in a later release of the tap" use case, applying selection for testing is still a big pain point, especially when streams are not known in advanced as with SQL.

Re: discovery, there are already cases (such as the tap-snowflake variants) where a form of selection is already implemented in config to be applied during discovery specifically to limit the cost of listing all remote streams. So from that perspective, supporting selection and other catalog manipulations as a native config feature standardises and enhances behaviours that are already being implemented in the community ad-hoc.

[kgpayne]

Thinking about it, another alternative might be to add singer-discover (as a library) or a native implementation to the test framework only, as that's where my primary pain point is at the moment. That doesn't solve the discovery point though 😅

[edgarrmondragon]

Re: discovery, there are already cases (such as the tap-snowflake variants) where a form of selection is already implemented in config to be applied during discovery specifically to limit the cost of listing all remote streams. So from that perspective, supporting selection and other catalog manipulations as a native config feature standardises and enhances behaviours that are already being implemented in the community ad-hoc.

Oh yeah, totally. There's definitely cases where cases where you want to limit discovery to certain areas of the source, and databases are the perfect example. But for REST APIs, all the streams are usually known beforehand and thus discovering them incurs no extra cost, but syncing them may certainly do.

One thing that we can use the Guild for is to come up with clear terminology. In the above there's two types of sources:

1. Sources with a static catalog, such as APIs and other sources with with slowly-changing streams.
2. Sources with a dynamic catalog, such as databases. The catalog can't be discovered without connecting to the source system.

@kgpayne Wdyt about using static catalog and dynamic catalog to refer to these? I'm of course open to better terms 😅

With that in mind, static catalog taps should try to avoid putting filtering logic in the configuration and rely solely on the catalog, while dynamic catalog taps should use filtering logic only to affect discovery but rely on the catalog of selection of discovered entities.

[pnadolny13]

This came up again today in meltano/hub#1295 related to voxmedia/tap-facebook-pages#8 (comment) where a subset of streams has some custom hardcoded logic that only the tap developer's org can use.

[edgarrmondragon]

@pnadolny13 I got a draft PR for this: #1672

[edgarrmondragon]

This is supported since v0.27.0.

[tayloramurphy]

#1652 (comment)