bug? titiler.xarray.io.Reader does not accept `s3_credentials` parameter

[abarciauskas-bgse]

In the process of testing CMR collections' compatibility with titiler-cmr, I ran into this error:

  File "/home/jovyan/titiler-cmr-compatibility/titiler_cmr/titiler/cmr/backend.py", line 294, in _reader
    with self.reader(
         ^^^^^^^^^^^^
TypeError: Reader.__init__() got an unexpected keyword argument 's3_credentials'

As background, I am trying to test the tiling capability in titiler-cmr without hitting the API endpoint itself. This is primarily to remove the network call to speed up testing more datasets. Using the code directly also enables me to test using EDL credentials (in theory) so the assignment of "compatible" or "not compatible" has to do with the dataset and the tiling code itself, not having to do with bucket permissions.

So the code I am running on the VEDA JupyterHub to test tiling looks approximately like this:

# AuthSettings in local instance of titiler-cmr are `strategy=environment` and `access=direct`
from titiler_cmr.titiler.cmr.backend import CMRBackend
from titiler.xarray.io import Reader as XarrayReader

auth = earthaccess.login()

# adding for reproducibility, exact values shouldn't matter
concept_id = "C2033151148-GES_DISC"
temporal_extent = ('2007-06-22T03:00:00.000Z', '2007-06-22T03:00:00.000Z')

query = {
    "concept_id": concept_id,
    "temporal": temporal_extent,
}

with CMRBackend(
    reader=XarrayReader,
    auth=auth,
    reader_options={
        "variable": variable,
    },
) as src_dst:
    image, _ = src_dst.tile(
        x,
        y,
        z,
        cmr_query=query,
    )

I think what has happened is that in migrating from a custom reader in titiler-cmr to using titiler.xarray.io.Reader we lost some of the parameters we were previously using in instantiating the reader, such as s3_credentials. I could be totally off base as this change was made a while ago, but looking at the files changed in PR #40 I see we exchanged the custom ZarrReader, which accepted s3_credentials, with from titiler.xarray.io import Reader as XarrayReader and titiler.xarray.io.Reader does not accept s3_credentials.

We do still have the capability to pass s3_credentials to the intended opener which is titiler-cmr's own xarray_open_dataset so I think all that is needed is the ability to pass those credentials through the titiler.xarray.io.Reader initiation.

Let me know if you think this assessment is correct @hrodmn @jbusecke

[jbusecke]

Thanks for tagging me @abarciauskas-bgse. I think this is quite relevant to developmentseed/titiler-multidim#101! It would be great if we could pass both s3_credentials and virtual chunk auth options in a consistent way.

[hrodmn]

Good find @abarciauskas-bgse. The refactor from #40 is the root of the issue. titiler.cmr.reader.xarray_open_dataset is ready to use an arg called s3_credentials but we don't currently have a mechanism to pass those credentials to the opener. If we want to patch it in titiler-cmr we could create a custom Reader class (like we used to have prior to #40) that can do this, but @jbusecke's instinct to sort this out upstream is probably right. The quickest thing could be to add an opener_options arg to titiler.xarray.io.Reader that would let users pass args through to the opener. There is probably a better strategy to take, though, so we should talk it over!

I'll open an issue over in titiler.

[abarciauskas-bgse]

Thank you @hrodmn for addressing this with an upstream fix already! I implemented a temporary fix but I will use the main branch of titiler in a branch of titiler-cmr to incorporate opener_options.

[abarciauskas-bgse]

Just a quick update that opener_options is working to pass through the s3_credentials in CMRBackend functions via this branch of titiler-cmr but I will wait to open a PR until opener_options is part of a titiler release. And also I need to test the feature and part functions.