STAC ratings ⭐

[gadomski]

Introduction

During a recent STAC Community Meetup, @tylere asked (I'm paraphrasing here):

How the h*** do I navigate the STAC ecosystem?

https://stacindex.org/ is an excellent general-purpose warehouse for STAC resources, but it suffers from a couple of problems:

• It's general purpose, with links to software tools, catalogs, APIs, and documentation. This isn't a bad thing per-se, but makes it a bit of a clumsy tool for someone that's just looking for data sources
• It is (relatively) un-opinionated, meaning that it puts the onus on the user to determine the quality of any given entry
• It's a pain in the butt to maintain (speaking for @m-mohr here)

I think there's a space in the STAC ecosystem for a STAC API-and-Catalog discovery tool with the following characteristics:

1. Focused: the tool helps users find STAC catalogs and APIs, nothing else
2. Opinionated: the tool will have ratings (e.g. out of five stars) that help guide users to "high quality" STAC
3. Automated: to the greatest extent possible, maintenance and updates will be automated. This means that ratings need to be auto-generated from a fixed set of criteria

Bullets one and three are simple design decisions, but the second one (ratings ⭐) is something I think is worth building a consensus around with the community.

Existing art

• json-schema validation is a great 👍🏼 / 👎🏼 rating system, but its output can sometimes be hard to understand. Also, since it's a boolean operation (is_valid) it doesn't on its own provide the granularity needed to build a five star rating system for a given STAC API or catalog.
• stac-check, stac-api-validator, and related tools applies the specs and the STAC best practices to STAC APIs and objects, printing out any violations. As a linter and checker, they are a great tools, but they're less useful for a lay user who may not be able to make an assessment of how "bad" a given violation is relative to others.

A proposed STAC rating ⭐ system

Listed below are the components of my proposed rating system. Each component would have one or more "checks", and each "check" could have a weight to describe its relative importance (implementation dependent).

Component	Description	Relative importance	Notes
Validity	Is the STAC value valid under its core json-schema?	High
Link accessibility	Are all of the STAC value's links accessible?	High
Geometry validity	Is the STAC value's geometry valid? (if applicable)	High
Extension validity	Is the STAC value valid under all of its extensions?	Medium	I weight this one less because extensions are more volatile than the core spec
Version	Is the STAC value a "modern" version	Medium	Right now v1.0.0 and v1.1.0 would be "modern"
Conformance classes	Does this STAC API implement the main STAC API conformance classes?	Medium	I want to be sure an API that has queryables is rated higher than one that doesn't
Cloud-native assets	When applicable, are the assets in a cloud-native format?	Low	We don't want require an implementer to touch the assets to generate the rating (IMO) so we'll just use the type field. That's why I rank this one low, because a lot of folks don't set the type field correctly.

Extension

Ratings could be an extension!

Field name	Type	Description
rating:stars	number	Required The number of stars, between zero and five, in this STAC value's rating. This should be equal to 5 * rating:score / rating:total.
rating:score	number	This STAC value's score, calculated during the rating process.
rating:total	number	The highest possible score from the rating process.
rating:checks	[Check object]	A list of checks that were run on this STAC value

Check object

Field name	Type	Description
name	str	Required The name of the check (e.g. "is_valid")
ok	bool	Required Did the check succeed?
description	str	Any notes about the check (e.g. the stringified validation output)
weight	integer	The relative weight of this check (implementation dependent)

Relation types

We should probably pick a relation type to link to a "rating report" or rating tooling page so folks know where the rating values came from.

Notes

• In my head, the "rating" of a STAC API would be a roll-up of the ratings of all of its collections plus (e.g.) one sample item for each collection. This could be implementation-dependent.
• I want to make sure this rating could be calculated relatively quickly from a generic client without any special authentication information, hence the non-requirement to touch assets.
• There could be a json-schema to enumerate the possible check names, but that seems extra complex as a place to start from.
• The rating components could obviously be extended to include everything in the Best Practices and beyond, but I think there's some high value things we can do without having to Check All the Things.

What do you think? Would a rating system like this be useful to you at all? What other components should be included in a rating system?

Update 2024-11-19

I've got a first-gen command-line interface (CLI) for ratings here: https://pypi.org/project/heystac/

$ pip install heystac
$ heystac rate https://raw.githubusercontent.com/radiantearth/stac-spec/refs/tags/v1.0.0/examples/simple-item.json
5.0 ★★★★★

[jkeifer]

This is great.

The first thing that comes to mind in reviewing your ranking components, beyond generally agreeing with what you propose, is a vague idea that it might also be good to have a way to measure how well items are "covered" by the core spec and extensions. Said differently, how much of the content of an item is not part of core STAC or an extension? It seems like this is some value that could be measured and might provide some insight into usability/quality of that additional metadata.

Or maybe such a measure is not actually reflective of any level of quality. I don't know. Just an idea I figured I'd throw out there.

[gadomski]

how much of the content of an item is not part of core STAC or an extension

Yeah, that's a really interesting idea. It should be discoverable via json-schema but I don't know if it's a solved problem or not. But I like it.

[m-mohr]

You can't really count via JSON Schema, so this needs probably a different solution.

[gadomski]

I guess you could count the dictionary keys that have a prefix: in stac_extensions, and add that to the core spec fields count. It'd be a little kludgey but it'd get you part of the way there.

[m-mohr]

Yeah, although GEE for example uses not a lot of extension, but has a lot of gee: prefixed fields ;-) Planet has something similar (although they at least have an extension for it now).

[hrodmn]

How about some kind of availability score. I'm not sure what the test would look like (maybe a stac-api-validator thing?) but a way to evaluate the reliability and availability of an API would be useful. Something like x / 100 requests return a 500 error.

[gadomski]

I think this is really useful too, but I'm not sure it belongs here. I see a "rating" as something that can live on a piece of STAC metadata to describe those metadata — a "success percentage" metric is inherently more dynamic than that?

A "STAC API dashboard" that monitors the top X STAC APIs would be an excellent complementary tool, though.

[j08lue]

The primary benefit from such a rating would, IMO, be to clearly define best practice and create an incentive to follow it.

Whether it is also helpful for discovery? 🤷

[m-mohr]

Will people use such a tool? Great question. Anecdotally, people do use the Planetary Computer (PC) Explorer to search for and discover datasets, and people found it valuable (IMO) in a large part due to the "curation" — you could trust that the data and metadata in the PC had been vetted

I'm wondering how much of that is data vs metadata. Even if the metadata is great, the data can be really bad. So just by curating the STAC metadata, you don't necessarily get good data. Just a thought...

[gadomski]

Yup, this is something that's coming up in internal conversations as well. STAC is just one part of the picture, so applying a rating to an API just based on its STAC could be misleading or incomplete. Rating the assets themselves feels much tricker, but not impossible.

[m-mohr]

Yeah, it goes into the direction of CEOS ARD in the end, but much of that can't be just derived from the pure numbers. You could just check whether the file follow certain structural best practices (like cloud-optimized, no data reported, maybe statistics included, ...). But generally difficult.

I'd like to sneak into your internal conversations. That sounds fun and interesting 😅

[gadomski]

I'd like to sneak into your internal conversations. That sounds fun and interesting 😅

🤣 not that much different than out here TBH, and probably a similar number of GIFs.

[m-mohr]

and probably a similar number of GIFs.

I have a suspect here... someone who's liking raccoons... 😂

[tylere]

I agree that (detailed) ratings would provide an incentive for following best practices, particularly if there were a rating checking tool that STAC metadata developers could easily run while populating their metadata. I would lean toward listing "All the things" developers should consider, even if only some off the checks are implemented and contribute to the rating. More could be implemented over time, if feasible.

[m-mohr]

It's a pain in the butt to maintain (speaking for @m-mohr here)

Well, it could be easier to maintain if I'd have more time to actually build a management UI and a crawler for it. I'm just overwhelmed with work and then everything is a pain in the butt :-D

Validity - Is the STAC value valid under its core json-schema?

Generally a good idea, it just is always misleading that "valid" entities are still invalid because we can't check everything properly in the schemas.

Link accessibility - Are all of the STAC value's links accessible?

Is this only links or also assets? What does that mean more specifically? If you use the auth extension and something returns e.g. a 401/403 for example, is it really okay to "downgrade"? Also, sometimes you link to external sources and if these are "out of my control" offline temporarily my STAC gets a worse grade?

Geometry validity - Is the STAC value's geometry valid? (if applicable)

"(if applicable)" - actually, I'd downgrade STAC entities that have no spatial information (I assume that's what you meant here?)

Extension validity

Not really validity, but making sure some extension are available would also help. Like raster, eo, etc. But it's hard to judge in which cases those are actually applicable. Raster for vector data doesn't make a lot of sense, EO for SAR neither.

Conformance classes Does this STAC API implement the main STAC API conformance classes?

What are the main conformance classes?

Cloud-native assets - When applicable, are the assets in a cloud-native format? Low

What files formats are cloud-native in your checks? I assume: COG, COPC, ZARR (what's the type though?), FGB (no media type?), GeoParquet, ...?

We don't want require an implementer to touch the assets to generate the rating (IMO) so we'll just use the type field. That's why I rank this one low, because a lot of folks don't set the type field correctly.

Well, but then it's their fault and it's worth downgrading it. So I'd actually think checking the type field and making it medium prio is fine. (On the other hand: uncertainties for media types, see above.)

Ratings could be an extension!

I don't like this idea, honestly. I've seen people manipulating things in STAC entites, would be equal to just put rating = 5 stars manually although it might be 1 star.

Some more ideas for criteria:

• CORS enabled (medium/high?)
• Landing page has reasonable id, title and description (e.g. not "stac-fastapi" as many have) (medium?)
• Check for open license? That might be difficult though. (low?)
• Check whether there's recent data for a collection that has an open-ended temporal extent. (low?)
• Collection has item_assets and summaries. (medium?)
• Check for latest extension versions? (low?)
• Item has overview, thumbnail or visual asset? (medium?)
• Collection actually has items (high?)
• links all have type and title (maybe except for self)?
• assets all have type and roles?
• assets have storage and/or auth extension when not accessible?

[gadomski]

Thanks for the additional ideas, I really like them. I think with these comments and those ideas, I can write up something more formal that can be PR'd and reviewed so we have a point of reference, rather than links into discussion comments. Appreciate the review @m-mohr 🙇🏼

Some inline responses:

Is this only links or also assets? What does that mean more specifically? If you use the auth extension and something returns e.g. a 401/403 for example, is it really okay to "downgrade"? Also, sometimes you link to external sources and if these are "out of my control" offline temporarily my STAC gets a worse grade?

Couple of parts here:

• Yes, I am proposing just checking links, not assets. Assets could also be another check, but IMO they're different. Many public STAC APIs have "private" (or otherwise restricted) assets, and I don't think that is necessarily a bad thing.
• In my opinion, using the "auth" extension on link should be discouraged, at least for "non-structural" links (so, not items, search, etc). Links are supposed to help people learn more about the STAC, and putting that information behind auth kind of defeats the purpose. This is where the ratings start veeing into "opinionated" rather than "objective"
• And yes, I do think things should get a worse grade if their links sometimes break. If links are flakey, the STAC itself is less useful to a user. This could be implementation dependent, of course — a certain rater could decide to "come back later" and re-rate if it wanted to.

"(if applicable)" - actually, I'd downgrade STAC entities that have no spatial information (I assume that's what you meant here?)

Yeah, I like that 👍🏼. Here, I meant an "invalid geometry", e.g. one that doesn't have a closing point on a polygon.

Not really validity, but making sure some extension are available would also help. Like raster, eo, etc. But it's hard to judge in which cases those are actually applicable. Raster for vector data doesn't make a lot of sense, EO for SAR neither.

I agree, but that starts to be less automateable. Maybe "if you have a GeoTIFF asset, you should have the raster extension" sort of thing?

What files formats are cloud-native in your checks? I assume: COG, COPC, ZARR (what's the type though?), FGB (no media type?), GeoParquet, ...?

Yeah, this one's less thought out (and tbh I haven't implemented it yet). I'd probably start with anything listed on https://guide.cloudnativegeo.org/ ... and this would be low-importance.

What are the main conformance classes?

Good question. Off the top of my head:

• search is pretty critical for most applications, so I think I'd 👎🏼 things that advertise items but not search
• I'd want to 👎🏼 APIs that have query but not filter
• Probably 👎🏼 search without sort

But again, pretty subjective.

I don't like this idea, honestly. I've seen people manipulating things in STAC entites, would be equal to just put rating = 5 stars manually although it might be 1 star.

Fair enough. I have a half-baked idea of "hashing" a rating report to make it verifiable but you're right that in the end someone's just going to do item.rating = "5 stars" and walk away.

[m-mohr]

Couple of additional remarks:

• Links: Fair, just keep in mind there's now also headers, method and body in the links. So just a GET request will not always work.
• Geometries: Is a Point or LineString good or bad?
• Extensions: Yes, it can probably only be in assets and depending on the type field. raster file formats => raster extension, vector formats => table extension?, copc => point cloud extension?, netcdf => datacube extension?, ... Wondering whether we have some extensions that should always be present, e.g. projection?
• Conformance classes: I have seen quite a variety with regards to the filter extension, sometimes weird queryables or none, sometimes mixture of CQL and methods. Maybe something to have a closer look to ensure how it is implemented really works. And then there's these weird APIs like CMR which have search on "sub-APIs" which drives STAC Browser crazy... What about that?

[gadomski]

Is a Point or LineString good or bad?

Seems fine to me? Feels like favoring Polygon is a very EO-specific approach

Wondering whether we have some extensions that should always be present, e.g. projection?

I kind of think so, that's the whole point of geospatial data, right? To know where it is in space and time?

I have seen quite a variety with regards to the filter extension, sometimes weird queryables or none, sometimes mixture of CQL and methods.

Agreed, and this is where ratings (and an associated web app) might end up helping — someone could go and see "oh this API has landsat and does a good job with filters, which means I can build more complex stuff on top of it".

Maybe something to have a closer look to ensure how it is implemented really works.

Yeah, I thought about including stac-api-validator in a ratings run to check exactly this, but discarded it because it takes a little while even on a good API. Could be something to add once I've got the core ratings infrastructure built out and automated.

And then there's these weird APIs like CMR which have search on "sub-APIs" which drives STAC Browser crazy... What about that?

It's a great question. I think we'd want to eventually support them, as long as they're "legal" (they aren't breaking the spec, they're just doing weird things with it).

[m-mohr]

Filter extension: I was more thinking about APIs that expose the filter extension but not e.g. CQL Text and JSON, which is weird... Also recently we had a mixup of requirement classes and conformance classes. OGC things. That also shouldn't take too long when it's just checking conformance classes being complete...

Look at https://cmr.earthdata.nasa.gov/stac for example: Exposes an "API" with filter conformace classes, but has no search link? just child links. Is that meant to apply to sub APIs? That should be downgraded for sure, having conformance classes but no respective links. That actually is breaking the spec, I'd say. The landing page should probably just be a Catalog.