What does setting CreateContainerIfNotExists do?

[lol768]

I don't want this library to try and create buckets. I know the bucket exists.

My IAM user doesn't have permission to create buckets, so it's never going to get a HTTP 409 back from GCP.

[KSemenenko]

this is good point! we doing some updates, and I will add this functionality

[lol768]

Cheers, I have done some sinful things as a workaround for now:

if (storageProvider is GCPStorage baseStorage)
{
    Log.Logger.Debug("Forcing GCP storage container creation to true");
    var field = typeof(GCPStorage).GetField("IsContainerCreated",
        BindingFlags.NonPublic | BindingFlags.Instance);
    field?.SetValue(baseStorage, true);
    Log.Logger.Debug("GCP storage container creation forced to true");
}

[hoylu-royberg]

I also discovered that CreateContainerIfNotExists does nothing in the latest release (2.1.14), so I'm very happy to hear that a fix might come soon for this, as we were also getting a long list of errors in our service logs, due to blob storage account being locked down for security reasons 🙏 Otherwise very happy with this library! Keep up the great work 😊

[KSemenenko]

plese check version '2.1.16'

[KSemenenko]

Is this fixed?

[hoylu-royberg]

Is this fixed?

Almost. Most of the issues I saw before are now handled correctly. However, for Azure there is still an issue, and today I finally had time to debug what the issue is; I think the problem is that in AzureStorage.CreateContainerInternalAsync it does this if CreateContainerIfNotExists is false:

Storage/ManagedCode.Storage.Azure/AzureStorage.cs

Line 152 in eb9a113

IsContainerCreated = await StorageClient.ExistsAsync(cancellationToken);;

We use very restrictive Azure blob storage container settings for our production containers, so an underlying call to GetPropertiesAsync fails. The result is that an Azure.RequestFailedException is thrown, and as a result IsContainerCreated will be set to false, and you will get an ugly XML formatted log message like this one from the Azure client library:

Azure.RequestFailedException: This request is not authorized to perform this operation.
RequestId:6b175b0f-301e-002f-25de-bde1bf000000
Time:2024-06-13T22:12:52.6303820Z
Status: 403 (This request is not authorized to perform this operation.)
ErrorCode: AuthorizationFailure

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
RequestId:6b175b0f-301e-002f-25de-bde1bf000000
Time:2024-06-13T22:12:52.6303820Z</Message></Error>

Headers:
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 6b175b0f-301e-002f-25de-bde1bf000000
x-ms-client-request-id: b5e7b4cc-2a8d-49d8-8daf-317faa19bd53
x-ms-version: 2024-05-04
x-ms-error-code: AuthorizationFailure
Date: Thu, 13 Jun 2024 22:12:52 GMT
Content-Length: 246
Content-Type: application/xml

   at Azure.Storage.Blobs.ContainerRestClient.GetPropertiesAsync(Nullable`1 timeout, String leaseId, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.GetPropertiesInternal(BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.ExistsInternal(Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.ExistsAsync(CancellationToken cancellationToken)
   at ManagedCode.Storage.Azure.AzureStorage.CreateContainerInternalAsync(CancellationToken cancellationToken) in /_/Storages/ManagedCode.Storage.Azure/AzureStorage.cs:line 157

While everything seems to work fine on the surface, this issue slows down all blob storage operations, and pollutes our logs with the above error level messages, when using Azure as the blob storage provider.

So I think it perhaps would be better to always set IsContainerCreated to true in all the cloud storage provider specific CreateContainerInternalAsync implementations if CreateContainerIfNotExists has been set to false. And for that scenario, just leave it to the ManagedCode.Storage library consumer to make sure that the bucket/container exists already.

[KSemenenko]

can you check new version please?

          try
          {
              IsContainerCreated = await StorageClient.ExistsAsync(cancellationToken);
          }
          catch (RequestFailedException e)
          {
              logger.LogException(e);
              //probably we don't have permission to check if container exists
              IsContainerCreated = true;
          }

[hoylu-royberg]

With version 2.1.19 we only get the ugly log statement once per session, so that is acceptable. Thank you! 🙂
I'll leave it to you or issue creator (@lol768) to close this issue.