GSClient auth fails with token-based application credentials.json

[joconnor-ecaa]

The following

gcloud auth application-default login
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/application_default_credentials.json
python -c "from cloudpathlib import GSPath; GSPath('gs://private_file').download_to('.')"

fails with

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/joe/cloudpathlib/cloudpathlib/cloudpath.py", line 171, in __call__
    cls.__init__(new_obj, cloud_path, *args, **kwargs)  # type: ignore[type-var]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/cloudpathlib/cloudpath.py", line 230, in __init__
    client = self._cloud_meta.client_class.get_default_client()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/cloudpathlib/client.py", line 104, in get_default_client
    cls._default_client = cls()
                          ^^^^^
  File "/Users/joe/cloudpathlib/cloudpathlib/gs/gsclient.py", line 88, in __init__
    self.client = StorageClient.from_service_account_json(application_credentials)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 109, in from_service_account_json
    return cls.from_service_account_info(credentials_info, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 76, in from_service_account_info
    credentials = service_account.Credentials.from_service_account_info(info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/oauth2/service_account.py", line 240, in from_service_account_info
    signer = _service_account_info.from_dict(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/auth/_service_account_info.py", line 50, in from_dict
    raise exceptions.MalformedError(
google.auth.exceptions.MalformedError: Service account info was not in the expected format, missing fields token_uri, client_email.

This is because the logic in the GSClient init assumes that if a GOOGLE_APPLICATION_CREDENTIALS file exists, it is in the format of a service account JSON key (i.e. the call to from_service_account_json).

When using workload identity federation GOOGLE_APPLICATION_CREDENTIALS is in a different format (see here).

It is possible to work around this with existing functionality, e.g. explicitly creating a google.storage.client.Client or credentials object. However it would be nice if GSClient and GSPath "just work" with workload identity federation. I've been monkeypatching GSClient and GSPath to achieve this in a few projects.

The simplest workaround is probably to replace the call to from_service_account_json with a call to google.auth.load_credentials_from_file.

[pjbull]

Thanks for the detailed report. From what I read, looks like this wouldn't be a breaking change for users that rely on the way things work now, so happy to take the fix you suggested.

Do you know if there is a good way to get and test the workload identity credentials?

[joconnor-ecaa]

Do you know if there is a good way to get and test the workload identity credentials?

You can set up workload identity federation by following this guide:

https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions

Adding a CI workflow that authenticates using the github action shown in that blog and then runs the live GCS tests should do the trick. Unsure if there's a simpler way to test.

[beazerj]

Any timeline on implementing this?

[pjbull]

@beazerj I don't have a timeline for this—have a few higher priority items I am working on.
Happy to take a PR that has a fix this.

From my reading, I think the fix might be simpler. Remove this block:

cloudpathlib/cloudpathlib/gs/gsclient.py

Lines 80 to 81 in 6bce0f9

	if application_credentials is None:
	application_credentials = os.getenv("GOOGLE_APPLICATION_CREDENTIALS")

I think that if that is done, when we call StorageClient() it will get the default auth which should do the right think in the scenario.

We just need someone to confirm this fix works and submit a PR. I think we probably won't explicitly test the live scenario since getting the config right to do so looks too complicated.

[amardeep]

Just ran into this bug today. Hoping for a fix.

[mil-ad]

Also running into this and am hoping for a fix

[mil-ad]

Can confirm that passing the client as in:

from cloudpathlib import GSClient, GSPath
from google.cloud.storage.client import Client
client = GSClient(storage_client=Client())

foo = GSPath("gs://some/path", client=client)

does work so I think the change you're proposing @pjbull does make sense

[pjbull]

I think this should be resolved in the latest version by #514