Get-S3Bucket : Invalid URI: The format of the URI could not be determined.

[KristofKuli]

Describe the bug

We are using Azure Devops pipeline to issue the above mentioned command, which currently throws the following error message:
Get-S3Bucket : Invalid URI: The format of the URI could not be determined.
We are using self-hosted agents, which are using Blueprint images from Microsoft. The command worked fine with AWSPowershell version 4.1.368, then the version has been updated in the Blueprint image, first to 4.1.487 and then to 4.1.518 and since we receive the mentioned error. The command is called during a custom Powershell script execution and it is connecting to an internal S3 instance. We are behind proxy, but in the script the target endpoint is added to the no_proxy environment variable, so the traffic should not go through the proxy.
Could you please help us investigate this problem?

Expected Behavior

The Get-S3Bucket should list the buckets.

Current Behavior

Currently we receive the following errors for different versions:
4.1.525 -- Invalid URI
4.1.518 -- Invalid URI
4.1.487 -- Bad Gateway
4.1.368 -- Bad Gateway

Get-S3Bucket : Error making request with Error Code BadGateway and Http Status Code BadGateway. No further error information was returned by the service.
At C:\a\1\au_module\scripts\Invoke-SCS3Upload.ps1:55 char:9
+         Get-S3Bucket -EndpointUrl $endpoint -Verbose
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (Amazon.PowerShell.C…3.GetS3BucketCmdlet:GetS3BucketCmdlet) [Get-S3Bucket], InvalidOperationException
+ FullyQualifiedErrorId : Amazon.S3.AmazonS3Exception,Amazon.PowerShell.Cmdlets.S3.GetS3BucketCmdlet
##[error]PowerShell exited with code '1'.

Get-S3Bucket : Invalid URI: The format of the URI could not be determined.
At C:\a\1\au_module\scripts\Invoke-SCS3Upload.ps1:55 char:9
+         Get-S3Bucket -EndpointUrl $endpoint -Verbose
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (Amazon.PowerShell.C…3.GetS3BucketCmdlet:GetS3BucketCmdlet) [Get-S3Bucket], InvalidOperationException
+ FullyQualifiedErrorId : System.UriFormatException,Amazon.PowerShell.Cmdlets.S3.GetS3BucketCmdlet
##[error]PowerShell exited with code '1'.

Reproduction Steps

Since the endpoint is an internal site, therefore I can't provide reproduction steps.

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSPowershell
4.1.525
4.1.518
4.1.487
4.1.368

Targeted .NET Platform

.NET Framework 4.8.1

Operating System and version

Windows Server 2022

[dscpinheiro]

Could this be related to #3198? We added support for http_proxy a while ago, but no_proxy was missing (this was fixed in version 4.1.525 of the AWSPowershell modules).

[KristofKuli]

Hi Daniel,

Yes, we have checked this issue yesterday, but even with 4.1.525, we are getting this "Invalid URI" error and from that message it's hard to decide, where the issue is coming from exactly. In the "Current Behavior" section I have specified the versions being tried and the respective error messages.

[dscpinheiro]

I'm not sure if you can share it, but what does your no_proxy look like? I know we're not ignoring it anymore, but from the PR:

The full implementation of no_proxy would involve DNS resolution to IP address and CIDR matching. That is a much larger scope to integrate into the SDK and for the SDK's use cases unlikely to be needed. This implementation does host name matching with wildcard support matching the Java and Kotlin SDK's implementation.

[KristofKuli]

I can't share the exact details, but I am using an example.
Target endpoint: https://example.target.company.com
Env vars in the script:
$proxy = "http://proxy.company.com:8080"
$noProxy = "127.0.0.1;localhost;example.target.company.com"
$noProxyComma = "127.0.0.1,localhost,example.target.company.com"

We have tried unsetting the $proxy variable as well, but didn't change the outcome.

[normj]

@KristofKuli What is the no_proxy being set to. The value of $noProxy or $noProxyComma?

[KristofKuli]

$noProxyComma

[KristofKuli]

Hi,
is there any update on this?

[afroz429]

Please enable logging to see if it helps in identifying the issue. PowerShell logging can be enabled with below commands.

    Add-AWSLoggingListener -Name "AWS Logs" -LogFilePath "C:\Logs\ps-logs.txt" # or any path
    Set-AWSResponseLogging Always
    Enable-AWSMetricsLogging

[KristofKuli]

Hi @afroz429,

We have enabled the logging, but it doesn't provide any extra information.
Amazon Error: 1 : UriFormatException making request ListBucketsRequest to [https://example.target.company.com/.](https://example.target.company.com/) Attempt 1., System.UriFormatException: Invalid URI: The format of the URI could not be determined. at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind, UriCreationOptions& creationOptions)

[KristofKuli]

Hi All,

Is there anything else, which can be tried? Do you see anything in the source code of AWSPowershell, which could cause this issue?

[dscpinheiro]

Apologies, but we have not been able to reproduce the error you described yet; but from the last error message it looks like the custom endpoint URL is not valid.

Was that URL changed recently? If you run the following command (replacing https://example.target.company.com/ with the actual value you're using), do you see an error? (this is essentially what the SDK does with the provided value - parse it to an System.Uri)

PS> [System.Uri]::new("https://example.target.company.com/")

AbsolutePath   : /
AbsoluteUri    : https://example.target.company.com/
// etc...

[KristofKuli]

Hi,

I have checked the URL with the command you have provided, but it doesn't display any error, it list the details of the URL properly. This issue was reported by one of our customer, who came across with this problem. For testing purpose, I have created an S3 Bucket and currently testing the connection from the Azure Devops pipeline. I am receiving Service Unavailable, when it tries to connect, hence I assume the command in not recognizing the no_proxy env var. I am pasting the task configuration below:

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      $Env:no_proxy="example.target.company.com"
      
      cd $(System.DefaultWorkingDirectory)
      
      Add-AWSLoggingListener -Name "AWS Logs" -LogFilePath "ps-logs.txt"
      
      Set-AWSResponseLogging Always
      
      Enable-AWSMetricsLogging
      
      Install-Module -Name AWSPowerShell -RequiredVersion 4.1.525 -Force
      
      Get-Package -Name AWSPowershell
      
      Set-AWSCredential -AccessKey $(accessKey) -SecretKey $(sharedSecret)
      
      Get-S3Bucket -EndpointUrl https://example.target.company.com

The error I receive is the following:

`Get-S3Bucket : Error making request with Error Code ServiceUnavailable and Http Status Code ServiceUnavailable. No 
further error information was returned by the service.
At C:\a\_temp\d03dbdf9-edcd-4ba4-a685-1df89a5b0f50.ps1:13 char:1
+ Get-S3Bucket -EndpointUrl https://example.target.company.com/
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Amazon.PowerShe...tS3BucketCmdlet:GetS3BucketCmdlet) [Get-S3Bucket],  
   InvalidOperationException
    + FullyQualifiedErrorId : Amazon.S3.AmazonS3Exception,Amazon.PowerShell.Cmdlets.S3.GetS3BucketCmdlet
 
##[error]PowerShell exited with code '1'.`

[KristofKuli]

Update:
I have installed the latest AWSPowershell version (4.1.542) locally and found this strange behavior:

1. If I use the default proxy settings on my workstation, then the Get-S3Bucket command fails with 503 Service Unavailable. The target is an internal site, hence the connection should not use the proxy. By default the no_proxy contains the domain of the target, but despite that I receive the above mentioned error.
2. If I include the target URL, then the connection still not working.
3. If I delete the value of https_proxy and http_proxy env vars and I use the default version of no_proxy value from point 1, then the connection works.
4. If I unset everything, so http_proxy, https_proxy and no_proxy is empty, then the connection is working.

In pipeline:
The same steps have been tried and the only thing to make it work, is to unset everything. So it seems, the no_proxy is getting recognized only, when the http_proxy and https_proxy variables have been unset.

[KristofKuli]

Hi,
has anyone reviewed my previous comment and was able to reproduce the issue? I am trying to figure out, whether there's a bug in AWSPowershell module or I am messing up something.

[dscpinheiro]

Nobody in our team was able to reproduce the problem (we also confirmed the latest version of PowerShell recognizes all of http_proxy, https_proxy, and no_proxy).

We created an internal backlog item to (optionally) turn off checking for those environment variables but it hasn’t been prioritized yet (FYI: For the error you described this is the only report we received, so we believe it’s unfortunately an issue with your setup - especially the internal S3 instance part).

[kellertk]

Hi folks, we're only able to assist you in troubleshooting the use of the .NET SDK with S3 proper, not an internal service that is emulating S3. Since we haven't been able to reproduce this issue, I will be closing it.

If I'm not correct and you are talking to the actual S3 service, please let me know and I'll reopen this.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.