Skip to content

Iceberg Catalog show create catalog fails when glue database has extra parameters #25691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

raj-manvar
Copy link
Member

@raj-manvar raj-manvar commented Apr 29, 2025

Description

When a Glue database has extra parameters, Iceberg connectors fails with io.trino.spi.TrinoException: No PropertyMetadata for property: error

Additional context and related issues

It's possible for a Glue schema to have extra property metadata, for example below database has extra "CreatedBy" property: ( relevant AWS glue client doc )

raj-manvar:~/Downloads$ aws glue get-database --name rajmanvartest
{
    "Database": {
        "Name": "rajmanvartest",
        "Description": "raj manvar test",
        "LocationUri": "s3a://rajmanvartest/",
        "Parameters": {
            "CreatedBy": "raj-manvar"
        },
        "CreateTime": "2025-04-18T16:36:35-05:00",
        "CreateTableDefaultPermissions": [
            {
                "Principal": {
                    "PrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"
                },
                "Permissions": [
                    "ALL"
                ]
            }
        ],
        "CatalogId": "1111111111111"
    }

For such cases, when running a show create schema rajmanvartest query fails with the below stacktrace

io.trino.spi.TrinoException: No PropertyMetadata for property: CreatedBy
	at io.trino.metadata.PropertyUtil.lambda$toSqlProperties$0(PropertyUtil.java:197)
	at com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:300)
	at io.trino.metadata.PropertyUtil.toSqlProperties(PropertyUtil.java:190)
	at io.trino.sql.rewrite.ShowQueriesRewrite$Visitor.showCreateSchema(ShowQueriesRewrite.java:674)
	at io.trino.sql.rewrite.ShowQueriesRewrite$Visitor.visitShowCreate(ShowQueriesRewrite.java:520)
	at io.trino.sql.rewrite.ShowQueriesRewrite$Visitor.visitShowCreate(ShowQueriesRewrite.java:219)
	at io.trino.sql.tree.ShowCreate.accept(ShowCreate.java:61)
	at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
	at io.trino.sql.rewrite.ShowQueriesRewrite.rewrite(ShowQueriesRewrite.java:216)
	at io.trino.sql.rewrite.StatementRewrite.rewrite(StatementRewrite.java:54)
	at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:93)
	at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:87)
	at io.trino.execution.SqlQueryExecution.analyze(SqlQueryExecution.java:289)
	at io.trino.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:222)
	at io.trino.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:892)
	at io.trino.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:153)
	at io.trino.$gen.Trino_457_20250428_213635_2.call(Unknown Source)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1570)

This is because the getSchemaProperties for Iceberg Connector's implementation for Glue catalog type implemented here also adds the Glue database parameters.
The above implementation is called from the Iceberg Connector's Connector Metadata implementation here

However, IcebergSchemaProperties only supports the location of the table as the available property, because of this any additional Glue database parameters fails the show create schema query.

The Hive Connector's implementation for Glue type only queries the table location and doesn't consider Glue database parameters as implemented here.

This MR updates the Iceberg connector to not consider Glue database parameters so that Iceberg Catalog's show create schema for these type of Glue schema succeeds.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Apr 29, 2025
@github-actions github-actions bot added the iceberg Iceberg connector label Apr 29, 2025
@raj-manvar raj-manvar changed the title Iceberg Catalog show create catalog fails when glue schema has extra parameters Iceberg Catalog show create catalog fails when glue database has extra parameters Apr 29, 2025
Copy link
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test to BaseIcebergConnectorSmokeTest.

@raj-manvar
Copy link
Member Author

Thanks @ebyhr for review. I had tried adding some tests, but the create schema Trino query doesn't support creating an equivalent glue database with extra parameters from what I see.
Can you help show any tests that have the schema used for testing created without running create schema ? Or any guidance on how to create a glue database which can be used for a test cases would be very helpful. Thanks.

@raj-manvar
Copy link
Member Author

https://github.com/trinodb/trino/blob/master/plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/catalog/glue/TestTrinoGlueCatalog.java This test creates a Glue database, but doesn't run any Trino queries directly. Can't find any test file which created a custom Glue database and also runs Trino query

@raj-manvar
Copy link
Member Author

oh looks like this is a known issue #24744

@raj-manvar
Copy link
Member Author

@ebyhr https://github.com/trinodb/trino/blob/master/plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/catalog/glue/TestIcebergGlueCatalogConnectorSmokeTest.java seems better place to add this test. This test class has glue client for being able to create a glue database with properties and also has queryRunner available to run show create schema queries. Is it okay to add test to this class instead ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed iceberg Iceberg connector
Development

Successfully merging this pull request may close these issues.

2 participants