Feat: increase default type precision in redshift/mssql view workaround #2791
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
treysp
changed the title
treysp
force-pushed
the
trey/schema-diff-precision
branch
2 times, most recently
from
8138a6a
to
71d76af
Compare
treysp
force-pushed
the
trey/varchar-workaround-max
branch
from
9fcc19f
to
074c1a9
Compare
treysp
changed the title
treysp
force-pushed
the
trey/varchar-workaround-max
branch
from
45a20f2
to
658ed8b
Compare
izeigerman
reviewed
Jun 21, 2024
| ) -> t.Dict[str, exp.DataType]: | ||
| # get default lengths for types that support "max" length | ||
| types_with_max_default_param = { | ||
| k: [self.SCHEMA_DIFFER.parameterized_type_defaults[k][0][0]] |
There was a problem hiding this comment.
I'm not sure we care about all types. I think we only care about Varchar don't we?
There was a problem hiding this comment.
Wouldn't this simplify the implementation?
There was a problem hiding this comment.
Or do we think the same issue applies to types VARBINARY?
There was a problem hiding this comment.
I figured it could happen for any of these types due to transpilation, but VARCHAR is much more widely used than NVARCHAR/CHAR/VARBINARY so in practice it's probably the only one that matters. I'll update.
izeigerman
reviewed
Jun 21, 2024
treysp
force-pushed
the
trey/varchar-workaround-max
branch
from
658ed8b
to
b719b06
Compare
| parameter = self.SCHEMA_DIFFER.get_type_parameters(col_type) | ||
| type_default = types_with_max_default_param[col_type.this] | ||
| if parameter == type_default: | ||
| col_type.set("expressions", [exp.DataTypeParam(this=exp.Var(this="max"))]) |
There was a problem hiding this comment.
Suggested change
| col_type.set("expressions", [exp.DataTypeParam(this=exp.Var(this="max"))]) | |
| col_type.set("expressions", [exp.DataTypeParam(this=exp.var("max"))]) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Redshift and MSSQL use the
VarcharSizeWorkaroundMixinto infer the size of a varchar column when it cannot be resolved from model code.Data may be truncated if it is inferred to the default length (256 in Redshift, 1 in MSSQL) and that was not intended by the user, so this PR increases the length to "max" if a column with the default length is detected.
The most likely cause of unintentional default length is transpilation from an engine where the bare type's default is unlimited. For example, in postgres
VARCHARwithout length parameter has unlimited length, so transpilation to RedshiftVARCHARwith default length 256 results in unintentionally truncating the field.