HIVE-29305 (addendum): Exclude AWS SDK v2 from hive-exec by okumin · Pull Request #6398 · apache/hive

okumin · 2026-03-31T15:25:31Z

What changes were proposed in this pull request?

We exclude AWS SDK from compile dependencies.

Why are the changes needed?

The current configuration causes a dependency hell and .github/workflows/docker-GA-images.yml does not succeed due to the issue.

2026-03-16T07:33:59,366 ERROR [qtp1563324258-54] handler.AbstractRequestHandler: CreateDatabaseHandler [bb8d9170-e10e-4005-80e6-060184a4b566-79] -  Create database newdb: Failed
java.util.concurrent.ExecutionException: java.lang.NoSuchFieldError: Class software.amazon.awssdk.core.client.config.SdkClientOption does not have member field 'software.amazon.awssdk.core.client.config.SdkClientOption CLIENT_ENDPOINT_PROVIDER'
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:503) ~[guava-22.0.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:462) ~[guava-22.0.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79) ~[guava-22.0.jar:?]
...
Caused by: java.lang.NoSuchFieldError: Class software.amazon.awssdk.core.client.config.SdkClientOption does not have member field 'software.amazon.awssdk.core.client.config.SdkClientOption CLIENT_ENDPOINT_PROVIDER'
	at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeAwsConfiguration(AwsDefaultClientBuilder.java:189) ~[aws-core-2.41.3.jar:?]
	at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeChildConfiguration(AwsDefaultClientBuilder.java:172) ~[aws-core-2.41.3.jar:?]
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:197) ~[bundle-2.26.19.jar:?]
	at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:37) ~[bundle-2.26.19.jar:?]
	at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:26) ~[bundle-2.26.19.jar:?]
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:164) ~[bundle-2.26.19.jar:?]
	at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:137) ~[hadoop-aws-3.4.2.jar:?]

I asked a contributor to include AWS SDK which was a transitive dependency in #6167. The decision was most likely wrong. They should not be a compile dependency and users should inject a bundle AWS SDK when they really use it so that S3A, Hive, or any other services use the consistent SDK.

Does this PR introduce any user-facing change?

No. Hive 4.2.0 also did not bundle SDK in hive-exec.jar directly.

How was this patch tested?

I confirmed the smoke test passed with this configuration.
https://github.com/okumin/hive/actions/runs/23786219926

diff

Master branch with aws-secretsmanager-caching = 2.1.1

[INFO] +- com.amazonaws.secretsmanager:aws-secretsmanager-caching-java:jar:2.1.1:compile
[INFO] |  +- software.amazon.awssdk:secretsmanager:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:aws-json-protocol:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:third-party-jackson-core:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:protocol-core:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:http-auth-aws:jar:2.41.3:compile
[INFO] |  |  |  +- software.amazon.awssdk:checksums-spi:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:checksums:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:sdk-core:jar:2.41.3:compile
[INFO] |  |  |  +- software.amazon.awssdk:profiles:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:retries:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:auth:jar:2.41.3:compile
[INFO] |  |  |  +- software.amazon.awssdk:http-auth-aws-eventstream:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.eventstream:eventstream:jar:1.0.1:compile
[INFO] |  |  +- software.amazon.awssdk:http-auth-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:http-auth:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:identity-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:http-client-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:regions:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:annotations:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:utils:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:aws-core:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:utils-lite:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:metrics-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:json-utils:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:endpoints-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:retries-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:apache-client:jar:2.41.3:runtime
[INFO] |  |  \- software.amazon.awssdk:netty-nio-client:jar:2.41.3:runtime
[INFO] |  \- com.github.spotbugs:spotbugs-annotations:jar:4.9.8:compile

After upgrading aws-secretsmanager-caching to 2.2.0

[INFO] +- com.amazonaws.secretsmanager:aws-secretsmanager-caching-java:jar:2.2.0:compile
[INFO] |  +- software.amazon.awssdk:secretsmanager:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:aws-json-protocol:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:third-party-jackson-core:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:protocol-core:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:http-auth-aws:jar:2.41.3:compile
[INFO] |  |  |  +- software.amazon.awssdk:checksums-spi:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:checksums:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:sdk-core:jar:2.41.3:compile
[INFO] |  |  |  +- software.amazon.awssdk:profiles:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:retries:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:auth:jar:2.41.3:compile
[INFO] |  |  |  +- software.amazon.awssdk:http-auth-aws-eventstream:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.eventstream:eventstream:jar:1.0.1:compile
[INFO] |  |  +- software.amazon.awssdk:http-auth-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:http-auth:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:identity-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:http-client-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:regions:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:annotations:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:utils:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:aws-core:jar:2.41.3:compile
[INFO] |  |  |  \- software.amazon.awssdk:utils-lite:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:metrics-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:json-utils:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:endpoints-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:retries-spi:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk:apache-client:jar:2.41.3:runtime
[INFO] |  |  \- software.amazon.awssdk:netty-nio-client:jar:2.41.3:runtime
[INFO] |  +- software.amazon.awssdk:aws-crt-client:jar:2.41.3:compile
[INFO] |  |  +- software.amazon.awssdk.crt:aws-crt:jar:0.40.3:compile
[INFO] |  |  \- software.amazon.awssdk:crt-core:jar:2.41.3:compile
[INFO] |  \- com.github.spotbugs:spotbugs-annotations:jar:4.9.8:compile

The final dependency (this patch)

[INFO] +- software.amazon.awssdk:bundle:jar:2.26.19:test
[INFO] +- com.amazonaws.secretsmanager:aws-secretsmanager-caching-java:jar:2.2.0:compile
[INFO] |  \- com.github.spotbugs:spotbugs-annotations:jar:4.9.8:compile

okumin · 2026-04-01T01:10:19Z

@ryukobayashi I found we would not be able to build a Docker image, so created an additional patch

ryukobayashi · 2026-04-01T03:07:06Z

@okumin Thanks for the information. I will backport it once it's merged!

pom.xml

ql/pom.xml

pom.xml

deniskuzZ

+1

okumin · 2026-04-02T11:23:13Z

I tried to update the versions and am testing S3A via https://github.com/okumin/hive/actions/runs/23897648469

pom.xml

sonarqubecloud · 2026-04-02T17:04:05Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

okumin · 2026-04-03T11:27:24Z

@deniskuzZ JFYI: I am going to merge this PR tomorrow and unblock #6387 if there are no additional comments.

deniskuzZ · 2026-04-03T11:29:59Z

@deniskuzZ JFYI: I am going to merge this PR tomorrow and unblock #6387 if there are no additional comments.

@okumin, please go ahead, I've already added +1

okumin added 2 commits March 30, 2026 18:28

HIVE-29305 (addendum): Exclude AWS SDK v2 from hive-exec

8b99970

Add a test dependency

5c4e1ab

asf-ci-hive added tests pending tests passed and removed tests pending labels Mar 31, 2026

okumin mentioned this pull request Apr 1, 2026

HIVE-29463: Publish nightly Docker images #6387

Open

deniskuzZ reviewed Apr 1, 2026

View reviewed changes

pom.xml Show resolved Hide resolved

deniskuzZ reviewed Apr 1, 2026

View reviewed changes

ql/pom.xml Show resolved Hide resolved

Move comment

e4f86c9

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Apr 1, 2026

deniskuzZ reviewed Apr 2, 2026

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

Upgrade AWS SDK v2

72ff3f7

asf-ci-hive added tests pending and removed tests passed labels Apr 2, 2026

deniskuzZ approved these changes Apr 2, 2026

View reviewed changes

deniskuzZ reviewed Apr 2, 2026

View reviewed changes

pom.xml Show resolved Hide resolved

Mark provided

1054a19

asf-ci-hive added tests passed tests pending and removed tests pending tests passed labels Apr 2, 2026

asf-ci-hive added tests passed and removed tests pending labels Apr 2, 2026

okumin merged commit 25d4a85 into apache:master Apr 4, 2026
2 checks passed

okumin deleted the HIVE-29305-addendum branch April 4, 2026 04:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-29305 (addendum): Exclude AWS SDK v2 from hive-exec#6398

HIVE-29305 (addendum): Exclude AWS SDK v2 from hive-exec#6398
okumin merged 5 commits intoapache:masterfrom
okumin:HIVE-29305-addendum

okumin commented Mar 31, 2026 •

edited

Loading

Uh oh!

okumin commented Apr 1, 2026

Uh oh!

ryukobayashi commented Apr 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deniskuzZ left a comment

Uh oh!

okumin commented Apr 2, 2026

Uh oh!

Uh oh!

sonarqubecloud bot commented Apr 2, 2026

Uh oh!

okumin commented Apr 3, 2026

Uh oh!

deniskuzZ commented Apr 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

okumin commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

diff

Master branch with aws-secretsmanager-caching = 2.1.1

After upgrading aws-secretsmanager-caching to 2.2.0

The final dependency (this patch)

Uh oh!

okumin commented Apr 1, 2026

Uh oh!

ryukobayashi commented Apr 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deniskuzZ left a comment

Choose a reason for hiding this comment

Uh oh!

okumin commented Apr 2, 2026

Uh oh!

Uh oh!

sonarqubecloud bot commented Apr 2, 2026

Quality Gate passed

Uh oh!

okumin commented Apr 3, 2026

Uh oh!

deniskuzZ commented Apr 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

okumin commented Mar 31, 2026 •

edited

Loading