This repository has been archived by the owner on Sep 23, 2024. It is now read-only.
Keep non-discoverable metadata from the nodes #88
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
#59 introduced changes to update streams' schemas and metadata before sync.
refresh_streams_schema
in stream_utils.py preserves non-discoverable metadata for a stream, but discards field's metadata. So even if a field is excluded with"selected": false
in a provided catalog the tap still includes it.Proposed changes
For all fields in the newly discovered schema keep non-discoverable metadata from the original catalog.
For instance this preserves
selected
key. Later this key is used herepipelinewise-tap-postgres/tap_postgres/sync_strategies/common.py
Line 12 in 8c4732e
In a nutshell: it restores the fields metadata behaviour to the version before #59 .
The risk: this can be a breaking change in cases where a provided catalog has wrong/outdated fields metadata. Examples: 1) fields metadata has wrong keys which are being overwritten with newly discovered schema 2) metadata excludes some fields, but they are still being synced. After the change the excluded columns will not be synced.
Types of changes
What types of changes does your code introduce to PipelineWise?
Put an
x
in the boxes that applyThis can be breaking changes in cases if fields metadata is used in a not expected way.
Checklist
setup.py
is an individual PR and not mixed with feature or bugfix PRs[AP-NNNN]
(if applicable. AP-NNNN = JIRA ID)AP-NNN
(if applicable. AP-NNN = JIRA ID)