Add Athena Sql dialect, PCTs & SDTs #4120
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add Athena PCT, Sql dialect & SDT support (finos#4054) * Add Athena PCT support (finos#4022) * Athena PCT setup * Update copyright * Add athena jdbc driver and pct support * Minor fixes * Update athena driver jar path * Remove AthenaUserNamePasswordAuthenticationStrategy * Add Vault in AthenaTestConnectionIntegration * Update athenaExtension.pure * Update athena jdbc jar source * Change athena driver scope to provided * Add Athena PCT setup - change S3 bucket name (finos#4025) * Athena PCT setup * Update copyright * Add athena jdbc driver and pct support * Minor fixes * Update athena driver jar path * Remove AthenaUserNamePasswordAuthenticationStrategy * Add Vault in AthenaTestConnectionIntegration * Update athenaExtension.pure * Update athena jdbc jar source * Change athena driver scope to provided * Update S3 bucket name * Update build.yml * Athena SDT changes * Athena jdbc jar direct installation to .m2 * Add sql dialect functions for maths, aggregation, strings * Add more sql dialect functions * Switch to sql dialect flow for Athena * Change S3 bucket name * Athena Sql dialect changes * Athena Sql dialect changes revert commented part in max.pure * Athena Sql dialect changes remove duplicate function * Add Athena Driver setup to github workflow * Switch to Sql dialect flow for athena * Update poms --------- Co-authored-by: Sai Sriharsha Annepu <[email protected]> Co-authored-by: vijaylather <[email protected]> * Update build.yml * Update build.yml * Update workgroup default to primary (finos#4056) * Add athena jdbc dependency to PCT (finos#4059) * Change aws region to us-east-1 (finos#4060) * Fix Athena SDTs + Change AWS region (finos#4063) * Fix athena SDTs * Change aws region to us-east-2 * Change athena schema to default (finos#4066) * Fix Test_Relational_Athena_UnclassifiedFunctions_PCT & Test_Relational_Athena_VariantFunctions_PCT * Fix PCT table column starting with '_' * Athena Essential Functions PCT changes * Fix Athena PCTs * Change S3 bucket name * Fix Athena sql dialect flow * Fix PCT error messages * Athena Essential Functions PCT changes and SDT change * Fix Athena Grammar & Relational PCTs * Fix Athena Relational PCTs * pom version updates for athena modules * Remove forkCount in athena PCT --------- Co-authored-by: Sai Sriharsha Annepu <[email protected]> Co-authored-by: vijaylather <[email protected]>
gs-ssh16
reviewed
Sep 25, 2025
| <ignoredUnusedDeclaredDependency>org.slf4j:slf4j-api</ignoredUnusedDeclaredDependency> | ||
| </ignoredUnusedDeclaredDependencies> | ||
| <ignoredUsedUndeclaredDependencies> | ||
| <ignoredUsedUndeclaredDependency>com.amazon.athena:AthenaJDBC42</ignoredUsedUndeclaredDependency> |
| <dependency> | ||
| <groupId>ch.qos.logback</groupId> | ||
| <artifactId>logback-classic</artifactId> | ||
| <scope>runtime</scope> |
| let expr = { | ||
| | #TDS | ||
| colOneNum, Col_two_letter, _col_3Mix | ||
| colOneNum, Col_two_letter, col_3Mix |
Contributor
There was a problem hiding this comment.
why should column name be updated?
Contributor
Author
There was a problem hiding this comment.
Athena does not support column names starting with '_'. So, changed the column name
Contributor
Author
There was a problem hiding this comment.
removed this change
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| <plugin> |
Contributor
There was a problem hiding this comment.
Same comments as engine server pom file
...org/finos/legend/engine/language/pure/grammar/test/TestAthenaConnectionGrammarRoundtrip.java
Outdated
Show resolved
Hide resolved
...e/src/main/resources/core_relational_athena/relational/sqlQueryToString/athenaExtension.pure
Outdated
Show resolved
Hide resolved
| $types->at($x)->match([ | ||
| d:meta::relational::metamodel::datatype::Date[1] | 'Date ' + $literalStr, | ||
| t:meta::relational::metamodel::datatype::Timestamp[1] | 'Timestamp ' + $literalStr, | ||
| t:meta::relational::metamodel::datatype::Timestamp[1] | if ($literalStr->matches('\'\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(?:\\.\\d+)?\''), |
Contributor
There was a problem hiding this comment.
can we use standard sql converter
| ), | ||
| sqlFunctionTest( | ||
| simpleFunctionCall('dateadd', [literal(%2014-12-04T15:22:23), literal(123), literal('MILLISECOND')]), | ||
| simpleFunctionCall('dateadd', [literal(%2014-12-04T15:22:23.000), literal(123), literal('MILLISECOND')]), |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
What does this PR do / why is it needed ?
Add Athena Sql dialect, PCTs & SDTs