Are we providing a coherent user experience regarding search and open?

[itcarroll]

Transferring to discussion a couple conversations (one taken up in #770, which I'm closing in favor of this Discussion, and one from 2024-01-07 Hackday) regarding top-level earthaccess user experience, particularly search* and open* functions.

We describe our public API within earthaccess.__init__.py as follows.

earthaccess/earthaccess/__init__.py

Lines 33 to 67 in 91866ac

	__all__ = [
	# api.py
	"login",
	"search_datasets",
	"search_data",
	"search_services",
	"get_requests_https_session",
	"get_fsspec_https_session",
	"get_s3fs_session",
	"get_s3_credentials",
	"get_s3_filesystem",
	"get_edl_token",
	"granule_query",
	"collection_query",
	"open",
	"download",
	"auth_environ",
	# search.py
	"DataGranule",
	"DataGranules",
	"DataCollection",
	"DataCollections",
	"DataServices",
	# auth.py
	"Auth",
	# store.py
	"Store",
	# kerchunk
	"consolidate_metadata",
	# virtualizarr
	"open_virtual_dataset",
	"open_virtual_mfdataset",
	"PROD",
	"UAT",
	]

My summary of the current status:

1. We define three search* entry points in the api module: search_datasets, search_data, and search_services. Discussion in consider alternative names for search_data and search_datasets #770 was tending toward a renaming of search_ to find_ (thanks @chuckwondo for the excellent analogy to search_bags), and also towards renaming _datasets to _collections and _data to _items (although there was not consensus on _items) to align with STAC vocabulary.
2. We provide granule_query and collection_query as "lower-level" (i.e. more python-cmr like) access to CMR results.
3. We define one open entry point in the api module and two open_virtual_* entry points in the dmrpp_zarr module.
   • Our open returns a list of file-like objects like you get from with fsspec.open_files(...) as.
   • Our open_virtual_[mf]dataset returns a dataset like you get from xarray.open_[mf]dataset.

Take this thread anywhere you want, but the concern I have is whether each new underscore detracts from the user experience. I'd like to think a user should not even need to be familiar with the difference between CMR collection vs granule queries nor have to know that the data accessed comes from a "virtual" dataset.

The work being done in #898 and towards smart_open should be kept in mind here too.

[itcarroll]

I've been kicking the tires on an idea in my head, so will give others the chance to as well.

Could a single search method that returns a "results" object (i.e. object that behaves like a list of DataCollections) be the only CMR entry point? The "results" object needs a filter method that applies granule query parameters and returns a different "results" object that behaves like a list of DataGranules but also like an fsspec.filesystem. It gives you ls as well as the low-level open. You can also pass this "results" object to smart_open (or, might I suggest access) to return a suitable xarray object.

• A "search" followed by a "filter" matches the EarthData Search flow, where you find a collection, then narrow down the granules you want from a collection, then add it to your "order".
• Using the word "search" here with no suffix avoids the search_bags concern.
• I don't know enough about search_services to say whether that could be worked in too.
• Two ways of "opening" (low-level like fsspec, and high-level like xarray) seem necessary and sufficient, and making one a method and one a top-level entry-point emphasizes the difference.

[chuckwondo]

I like the idea of simplifying things. Here are some obstacles that initially come to mind regarding what you've outlined above:

• Users that already have a collection concept ID or a collection name-version pair (or other unique "key" for a collection) have no need to first produce a collections result before producing a granules result, so this would seem to be counter toward your idea of simplification. Even if we can manage to avoid a collections query/request, this would still be at least slightly less simple, but I don't think it's out of the question. We just need to carefully design the approach.
• This becomes perhaps too tricky if we want to fold in search_services under the singular search umbrella because I don't think we need to first search for collections (I would have to check the docs) in order to find services. Further, even if we did, the "filter" part becomes tricky because an umbrella "filter" function would need to support all possible query parameters, so it would be difficult, if not impossible, to help the user know that if they are searching for granules, they can specify a value for "campaign" (for example), but cannot specify such a param value when searching for services. However, I would question the value of retaining search_services anyway. Perhaps that doesn't belong in earthaccess and should be relegated to python_cmr (and if so, this would perhaps be reason to keep the 2 libs distinct, rather than having earthaccess subsume python_cmr). If we remove search_services, perhaps this is no longer a concern.

[itcarroll]

Even if we can manage to avoid a collections query/request

As in search doesn't bother to query CMR when given a keyword arg for a unique identifier? Makes sense. As for simplicity, if my proposal is to change search_data(short_name="foo", point=(x, y)) to a syntax like search(short_name="foo").filter(point=(x, y)), then the simplicity comes in with multiple collections. In the proposed syntax you would replace "foo" with ["foo", "bar"] and be done.

[chuckwondo]

Yes, I figured that's the direction you were heading, and I do like the idea in general. Under the covers, we could combine everything into a single query.

I think my other concern is now regarding search_services. Offhand, I'm leaning towards moving that to python_cmr so that we keep earthaccess focused solely on collections and granules/items since the primary functionality earthaccess provides beyond python_cmr is the ability to use creds to open/download data. Removing search_services would allow us to keep the api simpler, along the lines of what you're suggesting. Otherwise, if we want a "universal" search function, we would have to support keywords/params for collection searches and for service searches, but there would be no nice way to keep them mutually exclusive (or at least it would likely be clumsy, if there is a way).

To play a bit more devil's advocate, I think my other concern would be wanting to find a convention that is perhaps less ambiguous. Given your example of search(short_name="foo").filter(point=(x, y)), I will reiterate a bit of my reasoning for "find" vs. "search". In this case, my concern would be potential ambiguity regarding what's being filtered. In general, I might think that if search produces a list of collections (even just logically, if not actually making a request under the covers), then filter would be subsetting the found collections, not filtering the contents of the found collections.

Ugh. Now I'm back on the fence with this, thinking that perhaps we stick with find_collections and find_items (or whatever the new names might become), but let's keep the conversation going. I'd like to bat around more ideas here.

[chuckwondo]

To add a bit more, I think the inconsistency of a search without a filter producing a collections result vs. a search with a filter producing a granules result could be very confusing, so I think we need some means of reducing (ideally eliminating) inconsistency/ambiguity.

[ayushnag]

Thanks for opening this discussion @itcarroll! We had some discussion about the open_virtual_dataset location in the hackday last week. The current thinking is to move the functions into a module like earthaccess.zarr.open_virtual_dataset. This way the new function is clearly separated from the standard earthaccess API. This could also be a place for the consolidate_metadata function so it becomes earthaccess.zarr.consolidate_metadata and a place other future zarr/icechunk projects part of earthaccess

[itcarroll]

Thanks, I think that's a great idea. It gives us room to modify open or define something new in the API that will use earthaccess.zarr internally as needed.