HIVE-29233: Iceberg: Validate HiveRESTCatalogClient with external RESTCatalogs like Gravitino #6108

difin · 2025-09-29T22:45:10Z

What changes were proposed in this pull request?

Implemented a new qtest-driverTestIcebergRESTCatalogGravitinoLlapLocalCliDriver for testing HiveRESTCatalogClient with Gravitino Iceberg Rest Server and a q-test for it.

Why are the changes needed?

To validate support for external RestCatalogs like Gravitino.

Does this PR introduce any user-facing change?

No

How was this patch tested?

New q-test.

zhangbutao · 2025-09-30T09:52:44Z

Hi @difin, Can we use the standalone HMS instead Gravitino as the Iceberg REST server for E2E testing?

deniskuzZ · 2025-09-30T11:52:51Z

Hi @difin, Can we use the standalone HMS instead Gravitino as the Iceberg REST server for E2E testing?

@zhangbutao, we already have a test for HMS RestCatalog (iceberg_rest_catalog_hms.q).
This PR aims to validate support for external RestCatalogs like Gravitino (iceberg_rest_catalog_gravitino.q).

.../test/java/org/apache/hadoop/hive/cli/TestIcebergRESTCatalogGravitinoLlapLocalCliDriver.java

okumin · 2025-10-01T07:57:08Z

I checked out this branch and ran mvn test -Pitests -pl itests/qtest-iceberg -Dtest=TestIcebergRESTCatalogGravitinoLlapLocalCliDriver.java -Dtest.output.overwrite=false -Dqfile_regex=iceberg_rest_catalog_gravitino. I received 500 probably from Gravitino. I'm waiting for CI to conclude if it is my local issue or not

See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/ for specific test cases logs.
 MetaException(message:Server error: null: {
"servlet":"org.glassfish.jersey.servlet.ServletContainer-324c64cd",
"message":"org.glassfish.jersey.server.ContainerException: java.lang.NoSuchMethodError: &apos;void org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(java.lang.String)&apos;",
"url":"/iceberg/v1/config",
"status":"500"
})```

deniskuzZ · 2025-10-03T13:35:41Z

itests/qtest-iceberg/pom.xml

+                <artifactItem>
+                  <groupId>com.h2database</groupId>
+                  <artifactId>h2</artifactId>
+                  <version>2.2.224</version>


${h2database.version}, property is defined in a root pom

deniskuzZ · 2025-10-03T13:39:12Z

iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q

+--! qt:replace:/(\S\"iceberg-version\\\":\\\")(\w+\s\w+\s\d+\.\d+\.\d+\s\(\w+\s\w+\))(\\\")/$1#Masked#$3/
+
+set hive.stats.autogather=false;
+set metastore.client.impl=org.apache.iceberg.hive.client.HiveRESTCatalogClient;


should we set these configs in the Driver, similar to iceberg.catalog.ice01.uri?

ice01 is hardcoded there anyways: https://github.com/apache/hive/pull/6108/files#diff-398be9698c49244815af06140fe75eaaeef75ec6e497a74b8953cbda334bf122R122

metastore.client.impl

metastore.catalog.default

iceberg.catalog.ice01.type

These configs are set in the driver too.
I added them here in q-test just for clarity to make them visible in the q-test itself.
Do you think it is better to keep them only in the driver?

deniskuzZ · 2025-10-03T13:47:10Z

.../test/java/org/apache/hadoop/hive/cli/TestIcebergRESTCatalogGravitinoLlapLocalCliDriver.java

+
+    if (warehouseDir != null && Files.exists(warehouseDir)) {
+      try (var paths = Files.walk(warehouseDir)) {
+        paths.sorted(Comparator.reverseOrder())


can we use FileUtils.cleanDirectory(warehouseDir) ?

Done using .deleteDirectory().

cleanDirectory(dir) → empties the folder but leaves the folder in place.
deleteDirectory(dir) → deletes the folder and everything under it.

deniskuzZ · 2025-10-03T13:59:31Z

itests/qtest-iceberg/src/test/resources/gravitino-h2-test-template.conf

@@ -0,0 +1,14 @@
+gravitino.iceberg-rest.httpPort = 9001


can we read this config in Driver?

Yes, made it configurable like the warehouseDir.

deniskuzZ · 2025-10-03T14:02:07Z

.../test/java/org/apache/hadoop/hive/cli/TestIcebergRESTCatalogGravitinoLlapLocalCliDriver.java

+   * to avoid overwriting container data.</p>
+   */
+  private void startWarehouseDirSync() {
+    fileSyncExecutor.scheduleAtFixedRate(() -> {


can we refactor

fileSyncExecutor.scheduleAtFixedRate(this::syncWarehouseDir, 0, 5, TimeUnit.SECONDS); private void syncWarehouseDir() { if (!gravitinoContainer.isRunning()) { return; } // The try-with-resources block handles the core logic try (/*... setup TarArchiveInputStream ...*/) { // ... (rest of the sync logic) } catch (Exception e) { LOG.error("Warehouse folder sync failed: {}", e.getMessage()); } }

okumin · 2025-10-03T03:13:53Z

.../test/java/org/apache/hadoop/hive/cli/TestIcebergRESTCatalogGravitinoLlapLocalCliDriver.java

+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0


You have to copy checkstyle/asf.header or change the template

https://sonarcloud.io/project/issues?sinceLeakPeriod=true&issueStatuses=OPEN%2CCONFIRMED&pullRequest=6108&id=apache_hive&open=AZmX8jO00mHwehPdTX0p

I'm trying to modify existing files. So far, we need to add exact headers to new files.
https://issues.apache.org/jira/browse/HIVE-29245

okumin

+1. Please modify the files where SonarQube reported minor issues.
https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6108&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true

sonarqubecloud · 2025-10-06T15:02:45Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
3.9% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added tests pending tests unstable and removed tests pending labels Sep 29, 2025

deniskuzZ reviewed Sep 30, 2025

View reviewed changes

.../test/java/org/apache/hadoop/hive/cli/TestIcebergRESTCatalogGravitinoLlapLocalCliDriver.java Show resolved Hide resolved

difin force-pushed the rest_client_gravitino branch from c949a61 to f3abd30 Compare September 30, 2025 15:11

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels Sep 30, 2025

difin force-pushed the rest_client_gravitino branch from f3abd30 to 0497faa Compare September 30, 2025 20:14

asf-ci-hive added tests pending and removed tests failed labels Sep 30, 2025

difin force-pushed the rest_client_gravitino branch from 0497faa to b604394 Compare September 30, 2025 20:51

asf-ci-hive added tests failed and removed tests pending labels Sep 30, 2025

difin changed the title ~~HIVE-29233: Iceberg: HiveRESTCatalogClient test with Gravitino Iceberg Rest Server~~ HIVE-29233: Iceberg: Validate HiveRESTCatalogClient test external RESTCatalogs like Gravitino Sep 30, 2025

difin changed the title ~~HIVE-29233: Iceberg: Validate HiveRESTCatalogClient test external RESTCatalogs like Gravitino~~ HIVE-29233: Iceberg: Validate HiveRESTCatalogClient with external RESTCatalogs like Gravitino Sep 30, 2025

difin force-pushed the rest_client_gravitino branch from b604394 to 8656b64 Compare September 30, 2025 20:58

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels Oct 1, 2025

difin force-pushed the rest_client_gravitino branch from 8656b64 to 54bb487 Compare October 1, 2025 03:56

asf-ci-hive added tests pending and removed tests unstable labels Oct 1, 2025

asf-ci-hive added tests unstable and removed tests pending labels Oct 1, 2025

difin force-pushed the rest_client_gravitino branch from 4390c5b to 8865c8c Compare October 2, 2025 21:43

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Oct 2, 2025

code review comments Oct 2.

c58ec82

difin force-pushed the rest_client_gravitino branch from 8865c8c to c58ec82 Compare October 3, 2025 02:42

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Oct 3, 2025

deniskuzZ reviewed Oct 3, 2025

View reviewed changes

code review comments Oct 3.

3eec1ac

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Oct 3, 2025

okumin reviewed Oct 5, 2025

View reviewed changes

okumin approved these changes Oct 6, 2025

View reviewed changes

code review comments Oct 6.

04807a8

asf-ci-hive added tests pending and removed tests passed labels Oct 6, 2025

asf-ci-hive added tests passed and removed tests pending labels Oct 6, 2025

difin merged commit d46d900 into apache:master Oct 6, 2025
2 checks passed

HIVE-29233: Iceberg: Validate HiveRESTCatalogClient with external RESTCatalogs like Gravitino #6108

HIVE-29233: Iceberg: Validate HiveRESTCatalogClient with external RESTCatalogs like Gravitino #6108

Uh oh!

Conversation

difin commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

zhangbutao commented Sep 30, 2025

Uh oh!

deniskuzZ commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

okumin commented Oct 1, 2025

Uh oh!

deniskuzZ Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

okumin left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Oct 6, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

difin commented Sep 29, 2025 •

edited

Loading

deniskuzZ commented Sep 30, 2025 •

edited

Loading

deniskuzZ Oct 3, 2025 •

edited

Loading

deniskuzZ Oct 3, 2025 •

edited

Loading

deniskuzZ Oct 3, 2025 •

edited

Loading