Add earthaccess.open_icechunk()

[jbusecke]

Over here I am experimenting with building virtual icechunk stores for data stored on NASA cloud buckets. This seems to tie in with a lot of the ideas in #956

For this sort of setup we need to provide two credentials:

• To access the icechunk store data (we are still discussing where this would be hosted)
• To access the "virtual chunks" (which are just references to byte_ranges within files on a NASA cloud bucket)

In the link above you can see that this results in a TON of boilerplate code, and I would like to reduce that at least for this specific use case (data is all accessible via earthaccess, the general case is much thornier and might in fact never work due to security concerns).

What I am proposing is a new .open_icechunk(url_to_icechunk) method which does the following:

1. Use earthaccess.login to create a refreshable credentials to the bucket where the icechunk store lives and open the icechunk repository
2. Inspect the config and determine which buckets hold virtual chunks and repeat step 1 with appropriate credentials
3. Return an icechunk.Repository that is fully authenticated to the user.

[jbusecke]

Reading more into #956...

@TomNicholas after reading #956 (reply in thread) I am very curious if you could elaborate on that. In a recent discussion with @paraseba I figured that there would be no good way to persist credentials (or arbitrary code that could retrieve such) into an icechunk store itself.

Thats why I thought earthaccess is an ideal layer to implement this quite specific but powerful logic. We can assume that users of earthaccess have some form of credentials for earthdata, and with that we can generate the necessary refreshable credentials to supply to the icechunk repo at opening, while hiding a lot of the details from the users.

But if there is a more general way to support this in icechunk, I would love to hear about it.

[paraseba]

I think this makes sense. We'll launch Icechunk 1.0 this week, that has a slightly changed API for managing virtual chunk container credentials. The new API offers more security and control to the Icechunk repo user. It's not a big change and I think everything @jbusecke mentions here still applies. One thing I currently dislike is that to provide credentials to the virtual chunk containers, the user needs at least two trips to the object store: one to learn what are the virtual chunk containers in the repo, and a second one to open with the right credentials. I think this will change very soon after 1.0, without breaking the API, by providing a reopen method than can operate without the second trip to the object store.

@jbusecke let me know if you need help with adapting to the new virtual chunk container API once it's published, I don't think you will, or in general if you need anything else from Icechunk.

[jbusecke]

Much appreciate the offer @paraseba! I will def be in touch as I prototype this feature here (currently not first work priority but very keen to get this working). Ill keep the v1.0 release in mind. Is there a particular branch for the new API that I could use as a starting point?

[TomNicholas]

I think what I wrote in that linked issue is mostly orthogonal to the authentication discussion here. Therefore I'm not quite sure what you want me to elaborate on. Nevertheless I agree with both you and Seba that it might make sense for earthaccess to provide an open function that worries about access credentials for you. Then it's basically the earthaccess team's job to make sure that is done in a secure way.

[paraseba]

@jbusecke I just uploaded the release candidate. The API won't change from this to 1.0. There are some migration notes here . Main differences are:

• now virtual chunk containers must be declared by the repo creater (we used to have a bunch of defaults for urls like s3://, http://, etc).
• readers of the repo must now authorize the virtual chunk container with a new argument to Repository.open.
• virtual chunk containers are no longer identified by a name, now we use their url prefix

[jbusecke]

@TomNicholas this might have simply been a time mismatch then. I interpreted think [Icechunk](https://github.com/earth-mover/icechunk) could be extended to handle NASA Earthdata Login credentials. in a way that icechunk would implement custom logic for Earthdata, but I think we are all in agreement that with current capabilities it is better to tie all of the auth specifics to earthaccess.

[betolink]

Thanks for working on this @jbusecke!! seems very similar to what @TomNicholas presented at CNG but with NASA data! here things get a little intertwined, one could argue that open_icechunk() could be an xarray method.

What I have in mind with Icechunk is that NASA DAACs or missions (e.g. NISAR) will be the ones generating these stores, not necessarily end-users. In any case, it will be great for earthaccess to provide a single line to open an icechunk store. I think there are 2 questions. NASA files can be accessed in 2 ways, in-region we can use the S3 URLs, out of region we need to use HTTPS. It would be great if Icechunk could provide 2 distribution links in their chunk manifest metadata. Something like

store = earthaccess.open_icechunk("s3://icechunk-store", distribution="s3")

I don't think this is a deal breaker, DAACs could generate 2 stores, one that points to S3 URLs and another that uses the HTTPs links so the stores can be used out of region.

@jbusecke if you're available it would be great if you could join one of the earthaccess hacking hours to demo what you did with this!

[jbusecke]

Hey @betolink,

What I have in mind with Icechunk is that NASA DAACs or missions (e.g. NISAR) will be the ones generating these stores, not necessarily end-users.

That would 100% be the end goal, yes. But even as an intermediate step we can probably save users a lot of time by instructing them how to build their own virtual stores (and making it very easy to work with them).

NASA files can be accessed in 2 ways, in-region we can use the S3 URLs, out of region we need to use HTTPS. It would be great if Icechunk could provide 2 distribution links in their chunk manifest metadata.

I think this is something worth dicussing further...If the links are following the same structure it should be trivial to create one store from another by using rename_paths. The more interesting way is how we serve this data. I think this goes back again to @TomNicholas comments in the other discussion about the scope of earthaccess (is this a catalog of sorts?). I think I would prefer to keep this functionality limited to wrapping needed auth, so that discovery/cataloging can be handled independently?

s3_icechunk_url = get_ref_from_some_catalog(type='in_region')
https_icechunk_url = get_ref_from_some_catalog(type='global')

store_s3 = earthaccess.open_icechunk(s3_icechunk_url)
store_https = earthaccess.open_icechunk(https_icechunk_url)

That way it is much more clear that the single purpose of that method is to open a given url to a nasa based virtual icechunk store with the appropriate credentials, and nothing else.

On the other hand I wonder if there is a way to store both datasets in one icechunk store, but whether that is beneficial needs some discussion.

@jbusecke if you're available it would be great if you could join one of the earthaccess hacking hours to demo what you did with this!

Have the next one on my calendar and planning to join!

[itcarroll]

Lolz, I dunno how any of this works and I'm sure it's awesome, but I'll still ungraciously chime in with a user experience concern. The addition of these underscore methods (e.g. open_virtual_dataset) leaves a lot to be desired, in my view. Great tools, absolutely, but a poor experience for a typical earth scientist. Is there a case to be made for having earthaccess.open (which accepts very rich DataGranules) be equipped to default to the best available method (be it icechunk, virtualzarr, or plain fsspec) based on granule metadata? And then xr.open_dataset opens the store (or the URL, or the file, since it can handle all of those)? Maybe some continuation of @betolink's "smart_open"?

[betolink]

A valid concern @itcarroll, we could wrap of these methods in a earthaccess.smart_open(results) where results are either CMR or STAC items, iterate through the "ACCESS" links and setup the best access pattern available for xarray.

[jbusecke]

Thanks for pointing this out @itcarroll .
It seems to me like there are almost two layers of API in earthaccess then?

• A more bare bones (perhaps developer/user focused) layer that simply wraps the auth around other tools in the stack
• A more opaque, smart layer that does a bit of cataloging, but really focuses on supporting end users that just 'want the data'

Is that a fair characterization? If so, it might be useful to more clearly separate them in the API?

[jbusecke]

Started working on this during the hack hour today and got stuck with this. Ill see if I can make more time in the coming days, or for the next hack.