How should we list tap-universal-file and all of the combinations?

[tayloramurphy]

https://github.com/MeltanoLabs/tap-universal-file

Likely all of the combos

• tap-s3-csv
• tap-s3-avro
• tap-s3-jsonl
• tap-avro

etc...

[pnadolny13]

@tayloramurphy this is tricky. Here are some options:

1. list it as a standalone plugin. Similar to how tap-spreadsheets-anywhere is listed as its own plugin and not also listed as tap-csv ,etc.
2. Option 1 but also adding a note to other relevant plugin pages like tap-s3-avro to suggest also considering tap-universal-file with a link. Something like tap-s3-avro features are also supported by tap-universal-analytics which is now recommended as a better alternative
3. Option 1 but also changing all the existing variations like tap-s3 and tap-s3-csv and tap-s3-avro etc. to hidden 😨 and add a ton of keywords to tap-universal-file. They'd still be accessible but not discoverable. When someone searches "s3" they really only see universal file instead of all the other variations.
4. Add a new variant to each of these plugins with tap-universal-file as its source, similar to the airbyte-wrapper style. With a list of curated config options to reflect only the plugin specific options i.e. the avro plugin page wouldnt show the config options for jsonl or csv. The plugin would be listed as tap-s3-avro but with an executable of tap-universal-file. A downside here is that it will be harder to maintain configs automatically if we pick and chose which ones we want to show i.e. the automation to refresh metadata wont work for these.
5. The same as option 2 but all configs are shown.

At first thought my opinion would be for option 1 potentially with variation 3 too. It would really clean up our listed plugins and funnel everyone to use the same well maintained tap instead of duplicating effort and causing confusion. Could be an aggressive move though 😅 .

What do you think? cc @visch

[tayloramurphy]

@pnadolny13 I think I like the option of having a variant for each combination but not affecting the metadata update. For each combination we can add a settings_preamble around what's needed for that combo.

Then we can flag a feature on the SDK to basically output required settings for a given set of filetypes and destinations. Not sure what that would look like honestly, but seems doable.

I'm mainly optimizing for having lots of options in the search and results list b/c I don't think "universal file" is descriptive enough.