[FLINK-37683][Connectors/AWS] Enable SSO credentials #197

soisyourface · 2025-04-09T15:56:14Z

There is a conversation in slack that has some additional information.

Purpose of the change

SSO Credentials do not work without including the sso and ssooidc dependencies in the shaded jar.

Verifying this change

I've tested this locally by doing mvn clean install and then using 5.1-SNAPSHOT in my flink job. I can add further tests if this PR is wanted.

Significant changes

(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)

Dependencies have been added or upgraded
Public API has been changed (Public API is any class annotated with @Public(Evolving))
Serializers have been changed
New feature has been introduced
- If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

add sso and ssooidc dependencies to the base pom, similar to sts

leekeiabstraction

Thank you Ryan for raising this PR!

Can we add test case? An example would be using ProfileTokenProviderBuilder to load a test implementation of profile file returns a token provider of class SsoProfileCredentialsProvider?

… config

…t the SSO stuff... but maybe this is valuable?

soisyourface · 2025-04-17T17:32:38Z

added this test
i'm not confident that it is doing much, so let me know if this is way off from what you were thinking.

i imagine the only "real" way to test the shadedness is to use an E2E test, yeah?

...connector-aws-base/src/test/java/org/apache/flink/connector/aws/util/AWSGeneralUtilTest.java

leekeiabstraction · 2025-04-23T13:26:28Z

...connector-aws-base/src/test/java/org/apache/flink/connector/aws/util/AWSGeneralUtilTest.java

+    assertThatThrownBy(() -> factory.create(context))
+      .isInstanceOf(java.io.UncheckedIOException.class)
+      .hasRootCauseInstanceOf(java.nio.file.NoSuchFileException.class)
+      .hasMessageContaining(".aws/sso/cache/");
+  }


Can you elaborate why we're asserting that an exception is thrown here? What would the full exception message look like?

here is the full exception message:

[ERROR] org.apache.flink.connector.aws.util.AWSGeneralUtilTest.testGetCredentialsProviderReturnsSsoProviderForSsoProfile Time elapsed: 0.925 s <<< ERROR! java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/me/.aws/sso/cache/b6c9aa45a3f785e14b3cdb559fe03c92b9a2fb00.json at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.tokenFromFile(SsoAccessTokenProvider.java:58) at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.resolveToken(SsoAccessTokenProvider.java:51) at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory$SsoProfileCredentialsProvider.<init>(SsoProfileCredentialsProviderFactory.java:107) at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory$SsoProfileCredentialsProvider.<init>(SsoProfileCredentialsProviderFactory.java:88) at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory.create(SsoProfileCredentialsProviderFactory.java:68) at org.apache.flink.connector.aws.util.AWSGeneralUtilTest.testGetCredentialsProviderReturnsSsoProviderForSsoProfile(AWSGeneralUtilTest.java:991) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84) at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.lambda$execute$1(JUnitPlatformProvider.java:199) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:193) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) Caused by: java.nio.file.NoSuchFileException: /home/me/.aws/sso/cache/b6c9aa45a3f785e14b3cdb559fe03c92b9a2fb00.json at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) at java.base/java.nio.file.Files.newByteChannel(Files.java:380) at java.base/java.nio.file.Files.newByteChannel(Files.java:432) at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) at java.base/java.nio.file.Files.newInputStream(Files.java:160) at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.tokenFromFile(SsoAccessTokenProvider.java:55) ... 75 more

in this case the exception is being thrown because there is no cache with the credentials in it. we can always just remove the check but this essentially means that if it makes it this far then the sso/ssooidc class loader stuff worked because the dependencies were found.

at least that was my initial thinking. but i really don't know if these tests are valuable, as mentioned in a previous comment.

leekeiabstraction · 2025-04-23T13:28:10Z

...connector-aws-base/src/test/java/org/apache/flink/connector/aws/util/AWSGeneralUtilTest.java

@@ -952,4 +958,41 @@ void testNewInstanceOfDefaultCredentialsProviderCreatedForEachClient() {

        return builder;
    }
+
+  @Test
+  void testGetCredentialsProviderReturnsSsoProviderForSsoProfile() throws IOException {


The test case name indicates that we're expecting SsoProvider to be returned but actual assertion is on UncheckedIOException being thrown. Can you elaborate?

likely a leftover name from a previous iteration of writing the tests.
renamed.

the bigger question is: does this test even do anything?

soisyourface and others added 2 commits April 7, 2025 17:11

add sso and ssooidc dependencies to the base pom, similar to sts

2aad3fb

soisyourface changed the title ~~[FLINK-XXXX][Connectors/AWS] Enable SSO credentials~~ [FLINK-37683][Connectors/AWS] Enable SSO credentials Apr 15, 2025

leekeiabstraction suggested changes Apr 16, 2025

View reviewed changes

soisyourface added 2 commits April 16, 2025 17:57

fix: Remove unsupported 'allowUndeclaredRTE' property from Checkstyle…

65ac913

… config

add a test... but i think an e2e test is really the only thing to tes…

Loading
Loading status checks…

60d4bfc

…t the SSO stuff... but maybe this is valuable?

boring-cyborg bot added the component=BuildSystem label Apr 17, 2025

soisyourface and others added 4 commits April 17, 2025 10:15

Merge branch 'apache:main' into main

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

f19f84a

whoops, removed this to try and fix an error

e516173

Merge branch 'main' of github.com:soisyourface/flink-connector-aws

Loading
Loading status checks…

6d1ab91

grr, IDEA added this plugin...

Loading
Loading status checks…

c987345

leekeiabstraction reviewed Apr 23, 2025

View reviewed changes

rename the test

Loading
Loading status checks…

f9c8ed4

soisyourface requested a review from leekeiabstraction April 25, 2025 18:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[FLINK-37683][Connectors/AWS] Enable SSO credentials #197

[FLINK-37683][Connectors/AWS] Enable SSO credentials #197

soisyourface commented Apr 9, 2025 •

edited

Loading

Uh oh!

leekeiabstraction left a comment

Uh oh!

soisyourface commented Apr 17, 2025

Uh oh!

Uh oh!

leekeiabstraction Apr 23, 2025

Uh oh!

soisyourface Apr 25, 2025

Uh oh!

leekeiabstraction Apr 23, 2025

Uh oh!

soisyourface Apr 25, 2025 •

edited

Loading

Uh oh!

[FLINK-37683][Connectors/AWS] Enable SSO credentials #197

Are you sure you want to change the base?

[FLINK-37683][Connectors/AWS] Enable SSO credentials #197

Conversation

soisyourface commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Verifying this change

Significant changes

Uh oh!

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soisyourface commented Apr 17, 2025

Uh oh!

Uh oh!

leekeiabstraction Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

soisyourface Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

leekeiabstraction Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

soisyourface Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

soisyourface commented Apr 9, 2025 •

edited

Loading

soisyourface Apr 25, 2025 •

edited

Loading