CDK CLI commands fail with "UnauthorizedException: Session token not found or invalid" after upgrading CDK to 2.173.0

[bbmarkus]

Please add your +1 👍 to let us know you have encountered this

Status: RESOLVED

Overview:

This failure happens under the following conditions:

• CLI version 2.173.0
• Credential source is SSO, configured in the /aws/config file.
• A region is specified in the profile section of the config.
• This region is different than the one in the sso_session, which is the one where the IAM Identity center was configured.

Complete Error Message:

Unable to determine the default AWS account (CredentialsProviderError): UnauthorizedException: Session token not found or invalid

Workaround:

Revert to version 2.172.0.

Solution:

In the call to the credential provider, pass the region in the parentClientConfig property instead of in the clientConfig property. The clientConfig is for configuring the inner auth client directly,
and has the highest priority, whereas parentClientConfig is the upper data client
and has lower priority than the sso_region but still higher priority than STS global region.

Related Issues:

---

Original issue

Describe the bug

Thiis morning after upgrading CDK installation to 2.173.0, all CDK command fail with the following error when using --verbose

Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
Unable to determine the default AWS account (CredentialsProviderError): UnauthorizedException: Session token not found or invalid

Rolling back to previous version 2.172.0 seems to fix this for us.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

2.172.0

Expected Behavior

Command executes succefully

Current Behavior

All commands fail due to UnauthorizedException when trying to determine the default account.

Reproduction Steps

0. Upgraded CDK and AWS cli (cli installed using the official snap package, upgrade was done as troubleshooting step)
1. Using SSO for AWS cli login (default profile, if that can make a difference, i.e. not providing --profile as part of the commands)
2. Run CDK command, e.g. cdk diff -e StackName --fail --verbose

Possible Solution

No response

Additional Information/Context

Command aws sts get-caller-identity returns the correct information, so this doesn't seem to be an authentication or SSO config issue.

CDK CLI Version

2.173.0

Framework Version

No response

Node.js Version

22

OS

Ubuntu (WSL)

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

@bbmarkus Good morning. Somehow using CDK version 2.173.0 (build b5c2189), I'm unable to reproduce the issue.

• Setup SSO account in IAM Identity Center, assigning Permission Set to SSO user.
• Setup SSO account using aws sso configure providing profile name as default. This configured SSO account in ~/.aws/config file as below:

  [default]
  sso_session = SSOAdmin
  sso_account_id = <<ACCOUNT-ID>>
  sso_role_name = AdministratorAccess
  region = us-west-2
  output = json
  
  [sso-session SSOAdmin]
  sso_start_url = https://d-<<SOME-ID>>.awsapps.com/start
  sso_region = us-west-2
  sso_registration_scopes = sso:account:access
  

• Login to SSO using AWS CLI command aws sso login not providing profile name so that it uses default profile.
• Using below CDK stack:

  import * as cdk from 'aws-cdk-lib';
  import { Bucket } from 'aws-cdk-lib/aws-s3';
  
  export class CdktestStackNew extends cdk.Stack {
    constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
      super(scope, id, props);
  
      new Bucket(this, 'testbucket');
    }
  }
  
  const app = new cdk.App();
  
  new CdktestStackNew(app, 'CdktestStackNew', {
    env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
  });

  Executed cdk diff -e CdktestStackNew --fail --verbose. It gives the below output:

  CDK toolkit version: 2.173.0 (build b5c2189)
  ...
  ...
  Reading cached notices from /Users/ashdhin/.cdk/cache/notices.json
  Toolkit stack: CDKToolkit
  Setting "CDK_DEFAULT_REGION" environment variable to us-west-2
  Looking up default account ID from STS
  Default account ID: <<ACCOUNT-ID>>
  Setting "CDK_DEFAULT_ACCOUNT" environment variable to <<ACCOUNT-ID>>
  ...  
  ...
  Retrieved account ID <<ACCOUNT-ID>> from disk cache
  Assuming role 'arn:aws:iam::<<ACCOUNT-ID>>:role/cdk-hnb659fds-lookup-role-<<ACCOUNT-ID>>-us-west-2'.
  Retrieved account ID <<ACCOUNT-ID>> from disk cache
  the stack 'CdktestStackNew' has not been deployed to CloudFormation or describeStacks call failed, skipping changeset 
  creation.
  Stack CdktestStackNew
  Parameters
  [+] Parameter BootstrapVersion BootstrapVersion: {"Type":"AWS::SSM::Parameter::Value<String>","Default":"/cdk-bootstrap/hnb659fds/version","Description":"Version of the CDK Bootstrap resources in this environment, automatically  retrieved from SSM Parameter Store. [cdk:skip]"}
  
  Conditions
  [+] Condition CDKMetadata/Condition CDKMetadataAvailable: {"Fn::Or":[{"Fn::Or":[{"Fn::Equals":[{"Ref":"AWS::Region"},"af-south-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-east-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-northeast-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-northeast-2"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-northeast-3"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-south-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-south-2"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-southeast-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-southeast-2"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-southeast-3"]}]},{"Fn::Or":[{"Fn::Equals":[{"Ref":"AWS::Region"},"ap-southeast-4"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ca-central-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"ca-west-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"cn-north-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"cn-northwest-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-central-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-central-2"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-north-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-south-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-south-2"]}]},{"Fn::Or":[{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-west-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-west-2"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"eu-west-3"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"il-central-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"me-central-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"me-south-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"sa-east-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"us-east-1"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"us-east-2"]},{"Fn::Equals":[{"Ref":"AWS::Region"},"us-west-1"]}]},{"Fn::Equals":[{"Ref":"AWS::Region"},"us-west-2"]}]}
  
  Resources
  [+] AWS::S3::Bucket testbucket testbucketCEBFCF91
  
  
  ✨  Number of stacks with differences: 1
  Reading cached notices from /Users/ashdhin/.cdk/cache/notices.json
  

Executing aws sts get-caller-identity without profile name gives below output:

{
    "UserId": "<<SOME-ID>>:<<USERNAME>>",
    "Account": "<<ACCOUNT-ID>>",
    "Arn": "arn:aws:sts::<<ACCOUNT-ID>>:assumed-role/AWSReservedSSO_AdministratorAccess_<<SOME-ID>>/<<USERNAME>>"
}

Could you try executing aws sso login again? Unsure if your SSO session has expired.

Thanks,
Ashish

[wetherc]

Using a non-default profile, I'm able to reproduce the described error as follows:

• AWS CLIV2 version: aws-cli/2.18.15 Python/3.12.6 Darwin/24.2.0 exe/x86_64
• AWS CDK version: 2.173.0 (build b5c2189)

SSO configuraton:

[profile test-profile]
sso_session = mysession
sso_account_id = <Account ID>
sso_role_name = AdministratorAccess
region = us-east-2
output = json

[sso-session mysession]
sso_start_url = https://d-<some ID>.awsapps.com/start/#/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

(Maybe notably, this is using a named profile; no default profile is configured)

• Refresh SSO session with aws sso login --profile test-profile
• Create a new test CDK stack:

#!/usr/bin/env python3

import os
import aws_cdk as cdk
from constructs import Construct


class TestStack(cdk.Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(self, construct_id, **kwargs)
        cdk.aws_s3.Bucket(scope, "testBucket")


app = cdk.App()
TestStack(
    app,
    "TestStack",
    env=cdk.Environment(
        account='<account ID>',
        region='us-east-2'
    ),
)

app.synth()

• Attempt to diff stack with cdk diff --profile test-profile --verbose
• Diff fails with

CDK toolkit version: 2.173.0 (build b5c2189)
Command line arguments: {
  _: [ 'diff' ],
  verbose: 1,
  v: 1,
  profile: 'test-profile',
  lookups: true,
...
...
Reading cached notices from /Users/<username>/.cdk/cache/notices.json
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to us-east-2
Unable to determine the default AWS account (CredentialsProviderError): UnauthorizedException: Session token not found or invalid
...
...
env: {
  CDK_DEFAULT_REGION: 'us-east-2',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '38.0.1',
  CDK_CLI_VERSION: '2.173.0'
}
Need to perform AWS calls for account <account ID>, but no credentials have been configured
Reading cached notices from /Users/<username>/.cdk/cache/notices.json
Need to perform AWS calls for account <account ID>, but no credentials have been configured

Diffing the same stack with the same SSO configuration succeeds by forcibly downgrading to 2.172.0 (build 0f666c5)

aws sts get-caller-identity --profile test-profile returns

sts get-caller-identity
{
    "UserId": "<some ID>:<username>",
    "Account": "<account ID>",
    "Arn": "arn:aws:sts::<account ID>:assumed-role/AWSReservedSSO_AdministratorAccess_<some ID>/<username>"
}

[ashishdhingra]

Using a non-default profile, I'm able to reproduce the described error as follows:

• AWS CLIV2 version: aws-cli/2.18.15 Python/3.12.6 Darwin/24.2.0 exe/x86_64
• AWS CDK version: 2.173.0 (build b5c2189)

SSO configuraton:

[profile test-profile]
sso_session = mysession
sso_account_id = <Account ID>
sso_role_name = AdministratorAccess
region = us-east-2
output = json

[sso-session mysession]
sso_start_url = https://d-<some ID>.awsapps.com/start/#/
sso_region = us-east-1
sso_registration_scopes = sso:account:access

sts get-caller-identity
{
"UserId": ":",
"Account": "",
"Arn": "arn:aws:sts:::assumed-role/AWSReservedSSO_AdministratorAccess_/"
}

@wetherc Thanks for the inputs. I'm able to reproduce it by using different regions in sso-session and SSO profile:

...
Reading cached notices from /Users/ashdhin/.cdk/cache/notices.json
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to us-east-2
Unable to determine the default AWS account (CredentialsProviderError): UnauthorizedException: Session token not found or invalid
context: {
...
....
Unable to resolve AWS account to use. It must be either configured when you define your CDK Stack, or through the environment
Reading cached notices from /Users/ashdhin/.cdk/cache/notices.json
Unable to resolve AWS account to use. It must be either configured when you define your CDK Stack, or through the environment

Below is my SSOAdmin profile configured in ~/.aws/config:

[sso-session SSOAdmin]
sso_start_url = https://d-<<SOME-ID>>.awsapps.com/start
sso_region = us-west-2
sso_registration_scopes = sso:account:access

[profile SSOAdmin]
sso_session = SSOAdmin
sso_account_id = <<ACCOUNT-ID>>
sso_role_name = AdministratorAccess
region = us-east-2
output = json

Ensured to execute aws sso logout --profile SSOAdmin and aws sso login --profile SSOAdmin to be sure.

Just checked, this works in CDK CLI (and package) version 2.172.0 (build 0f666c5).

[otaviomacedo]

This regression was definitely caused by #32456.

Now that we are forcing the credential provider to use the region configured in the profile, it will do so. If the user is configured in a different region, the authentication will fail:

[profile test]
sso_session = my-sso
sso_account_id = 111111111111
sso_role_name = AdministratorAccess
region = us-east-2 # wrong region
output = json


[sso-session my-sso]
sso_start_url = https://d-abcde.awsapps.com/start/#
sso_region = us-east-1 # right region 
sso_registration_scopes = sso:account:access

It's the same thing that happens if we don't set the region in the profile and use the wrong region in sso-session:

[profile test]
sso_session = my-sso
sso_account_id = 111111111111
sso_role_name = AdministratorAccess
# no region
output = json


[sso-session my-sso]
sso_start_url = https://d-abcde.awsapps.com/start/#
sso_region = us-east-2 # wrong region 
sso_registration_scopes = sso:account:access

[otaviomacedo]

@wetherc help me understand your use case. Why do you have different regions in the profile and in the SSO session?

[wetherc]

Our organization's Identity Center config / access portal / etc are all deployed into us-east-1 from a generic network services account in our OU. The separate account our team deploys into restricts resources to us-east-2 for geographic proximity to other outside-of-AWS resources we exchange traffic with. We set the default region of profiles interacting with that account to us-east-2 to mirror its deployment restrictions, and explicitly override the region in any CDK stacks where we actively want to deploy to a non-east-2 region

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.