-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compatibility with upcoming deprecation of plugin setting kind: hidden
and kind: password
#1418
Comments
@ReubenFrankel thanks for the write up! How would we do this without breaking existing user's projects? We dont have yaml schema versioning of plugins on the hub so I'm not sure how we'd be able to transition to this new schema. It seems like we should be able to add the new keys to the definitions but we couldnt change the kind otherwise any non-latest meltano versions would stop treating that setting as sensitive. When would we be safe to remove them so the deprecation warnings go away? |
One option might be to not show the deprecation messages or put them behind a feature flag for now (like how The other way would be to backport the changes to older Meltano versions as patch releases. I don't think this is something Meltano has done before - probably for good reason since it feels like a violation of the principles of semantic versioning IMO. |
@meltano/engineering would appreciate some thoughts on this. Seems like the conversation in #1205 (comment) is very relevant. |
This is what I think would happen based on what steps we take on the hub: Update the hub metadata to reflect this new change:
Don't update the hub metadata to reflect this new change:
@tayloramurphy I think the challenge here is slightly different than #1205 (comment) because that is trying to make sure the plugin python package and plugin metadata content pulled from the hub are in sync. This is versioning between the hub plugin metadata schema and the meltano package version that consumes it. Versioning in this case would mean updating our hub API route
I think this is probably the path forward but it requires us to have multiple hub API versions. We could keep v1 up basically indefinitely because all meltano versions >=2.0.0<2.20.0 use it and we could also serve a v2 version with these new breaking changes that meltano versions >=2.20.0 would use. Eventually we'd want to deprecate v1 but we'd probably want to set a date far in the future and give lots of warnings to not disrupt users who update versions slowly. I wonder how much work it is to update the hub to serve multiple API versions. |
@pnadolny13 I tested with
(somewhat ironically leaked sensitive credentials from |
We decided after Office Hours that @edgarrmondragon was going to take a spike to see what's required to update the Hub API version. @edgarrmondragon can you create an issue for that so we can track? |
When meltano/meltano#7733 and meltano/meltano#7874 are released in a future Meltano version,
kind: hidden
andkind: password
will become deprecated and show a warning message if existing setting definitions use them.Problem
The deprecation message will show until the plugin definition is updated (i.e. plugin
.lock
file no longer contains deprecatedkind
s) on the Hub andmeltano lock --update
is run. Since all versions of Meltano reference the same Hub instance, removing the deprecatedkind
s for all plugin definitions would potentially break all but the latest versions of Meltano if a user was to ever update their plugin.lock
files (they don't know anything about the newhidden: true
andsensitive: true
setting properties, unless the features are back-ported).Changes (solution permitting)
kind: hidden
changes to
kind: password
changes to
kind
as actual data typeConsider providing actual expected data types as
kind
now that this is possible, if applicable for a plugin. I imagine that the majority of plugins will not require this, but consider the following setting:should probably change to
The text was updated successfully, but these errors were encountered: