Skip to content

Add Athena Sql dialect, PCTs & SDTs #4120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Paarth002 wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Add Athena Sql dialect, PCTs & SDTs #4120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
215745d
Athena PCT +SDT changes (#4092)
Paarth002 Sep 16, 2025
d9650d0
Run athena tests only
gs-ssh16 Sep 16, 2025
278168b
Update build.yml
gs-ssh16 Sep 16, 2025
31d9113
Update build.yml (#4094)
Paarth002 Sep 16, 2025
8be5d8f
Fix athena PCTs + maxPoolSize changes (#4096)
Paarth002 Sep 17, 2025
4d3fca4
Fix substring in athenaSqlDialect
Paarth002 Sep 24, 2025
1ec0cb8
Fix github build.yml
Paarth002 Sep 24, 2025
90c1b4c
Remove athena PCT/SDT testing on github
Paarth002 Sep 24, 2025
6e59ab0
Fix pom versions
Paarth002 Sep 24, 2025
12a187b
Fix mod tests
Paarth002 Sep 25, 2025
04386b2
Fix mod PCT
Paarth002 Sep 25, 2025
9fa60db
Remove Athena jar download step from build
Paarth002 Sep 25, 2025
b497e65
Athena fixes
Paarth002 Sep 25, 2025
3bd24e8
Change profile to pct-cloud-test-athena
Paarth002 Sep 25, 2025
3eb25fb
date add sdt millisecond fix
vijaylather Oct 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ jobs:
env:
MAVEN_OPTS: "-Xmx1g"
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
mvn -B -e surefire:test -pl ${{ steps.modulesToInclude.outputs.INCLUSION_MODULES }} ${{ needs.build.outputs.profiles }} -T2 -DargLine="-Xmx6g" -Dsurefire.reports.directory=${GITHUB_WORKSPACE}/surefire-reports-aggregate

Expand Down Expand Up @@ -270,4 +270,4 @@ jobs:
working-directory: legend-docs
run: |
git commit -m "maint: update legend-engine docs from ref: ${{ github.sha }}"
git push
git push
2 changes: 1 addition & 1 deletion .run/Maven clean install all Relational PCT Modules.run.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
<option value="install" />
<option value="--fail-at-end" />
<option value="-pl" />
<option value=":legend-engine-xt-relationalStore-h2-PCT,:legend-engine-xt-relationalStore-duckdb-PCT,:legend-engine-xt-relationalStore-memsql-PCT,:legend-engine-xt-relationalStore-spanner-PCT,:legend-engine-xt-relationalStore-postgres-PCT,:legend-engine-xt-relationalStore-snowflake-PCT,:legend-engine-xt-relationalStore-databricks-PCT" />
<option value=":legend-engine-xt-relationalStore-athena-PCT,:legend-engine-xt-relationalStore-h2-PCT,:legend-engine-xt-relationalStore-duckdb-PCT,:legend-engine-xt-relationalStore-memsql-PCT,:legend-engine-xt-relationalStore-spanner-PCT,:legend-engine-xt-relationalStore-postgres-PCT,:legend-engine-xt-relationalStore-snowflake-PCT,:legend-engine-xt-relationalStore-databricks-PCT" />
</list>
</option>
<option name="multimoduleDir" />
Expand Down
1 change: 1 addition & 0 deletions ...rc/test/java/org/finos/legend/engine/extensions/collection/generation/TestExtensions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -651,6 +651,7 @@ protected Iterable<String> getExpectedCodeRepositories()
.with("core_persistence_relational")
.with("core_dataquality")
.with("core_relational")
.with("core_relational_athena")
.with("core_relational_bigquery")
.with("core_relational_spanner")
.with("core_relational_trino")
Expand Down
5 changes: 5 additions & 0 deletions legend-engine-config/legend-engine-repl/legend-engine-repl-relational/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,11 @@
<artifactId>legend-engine-xt-relationalStore-snowflake-PCT</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.finos.legend.engine</groupId>
<artifactId>legend-engine-xt-relationalStore-athena-PCT</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.finos.legend.engine</groupId>
<artifactId>legend-engine-xt-relationalStore-memsql-PCT</artifactId>
Expand Down
22 changes: 22 additions & 0 deletions legend-engine-config/legend-engine-server/legend-engine-server-http-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,23 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>dependency-analyze</id>
<configuration>
<ignoredUnusedDeclaredDependencies>
<ignoredUnusedDeclaredDependency>org.slf4j:slf4j-api</ignoredUnusedDeclaredDependency>
</ignoredUnusedDeclaredDependencies>
<ignoredUsedUndeclaredDependencies>
<ignoredUsedUndeclaredDependency>com.amazon.athena:AthenaJDBC42</ignoredUsedUndeclaredDependency>
Copy link
Contributor

@gs-ssh16 gs-ssh16 Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this?

Copy link
Contributor Author

@Paarth002 Paarth002 Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

</ignoredUsedUndeclaredDependencies>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Expand Down Expand Up @@ -1183,6 +1200,11 @@
<artifactId>legend-engine-xt-relationalStore-snowflake-PCT</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.finos.legend.engine</groupId>
<artifactId>legend-engine-xt-relationalStore-athena-PCT</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.finos.legend.engine</groupId>
<artifactId>legend-engine-xt-relationalStore-databricks-PCT</artifactId>
Expand Down
6 changes: 6 additions & 0 deletions ...-engine-core-pure/legend-engine-pure-ide/legend-engine-pure-ide-light-http-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,12 @@
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>org.finos.legend.engine</groupId>
<artifactId>legend-engine-xt-relationalStore-athena-PCT</artifactId>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>org.finos.legend.engine</groupId>
<artifactId>legend-engine-xt-relationalStore-memsql-PCT</artifactId>
Expand Down
2 changes: 2 additions & 0 deletions ...ne-pure-ide-light-http-server/src/main/java/org/finos/legend/engine/ide/PureIDELight.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,9 @@ protected MutableList<RepositoryCodeStorage> buildRepositories(SourceLocationCon
.with(this.buildCore("legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-PCT", "relational-postgres-pct"))
.with(this.buildCore("legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-duckdb/legend-engine-xt-relationalStore-duckdb-PCT", "relational-duckdb-pct"))
.with(this.buildCore("legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-h2/legend-engine-xt-relationalStore-h2-PCT", "relational-h2-pct"))
.with(this.buildCore("legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-athena/legend-engine-xt-relationalStore-athena-PCT", "relational-athena-pct"))
.with(this.buildCore("legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-sqlDialectTranslation-pure", "external-store-relational-sql-dialect-translation-memsql"))
.with(this.buildCore("legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-athena/legend-engine-xt-relationalStore-athena-sqlDialectTranslation-pure", "external-store-relational-sql-dialect-translation-athena"))
;

}
Expand Down
2 changes: 1 addition & 1 deletion ...tionalStore-PCT-pure/src/main/resources/core_external_test_connection/pct_relational.pure
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -863,7 +863,7 @@ function meta::relational::tests::pct::process::buildJoins(class:Class<Any>[1],
let targetType = $p->functionReturnType().rawType->toOne();
let target = $classTableMappings->filter(ctm|$ctm.class == $targetType)->toOne();
let srcColumn = $src.table.columns->cast(@Column)->filter(c|$c.name == '_pureId')->toOne();
let fkColumn = ^Column(name='_'+$class.name->toOne()+'_'+$p.name->toOne(), type=^meta::relational::metamodel::datatype::Integer());
let fkColumn = ^Column(name=$class.name->toOne()+'_'+$p.name->toOne() + '_', type=^meta::relational::metamodel::datatype::Integer());
let targetTable = $target.table;
$targetTable->mutateAdd('columns', $fkColumn);
let srcAlias = ^TableAlias(database=$db, name='src', relationalElement=$src.table);
Expand Down
Loading