Seeking Recommendations for Maintaining Consistency Between STAC JSON and GeoParquet

[drewm-swe]

I’m designing geospatial workflows for a non-static STAC catalog where new records/assets are added frequently. My goal is to maintain strong consistency between STAC JSON and GeoParquet formats while leveraging:

• STAC JSON: For metadata standardization, extensibility, and external data discovery via STAC APIs.
• GeoParquet: For efficient internal processing workflows and analytics.

Question:

• What are best practices for ensuring strong consistency between STAC JSON and GeoParquet in a catalog with frequent updates?
  • I’ve considered synchronous, on-the-fly conversion behind the scenes of STAC API endpoints.
  • Another option I've considered is periodic conversions from STAC JSON to GeoParquet, though this would provide weaker consistency compared to real-time updates.

Looking forward to your input!

[scottyhq]

Also interested in this topic! @TomAugspurger, perhaps could you comment on how this is done by microsoft planetary computer for datasets like Sentinel-1 or Sentinel-2? According to this thread microsoft/PlanetaryComputer#368 the catalog is updated every 6 hours, but it looks like the geoparquet dump hasn't been updated since 2024-06-24?

import pystac_client
import planetary_computer
import adlfs

catalog = pystac_client.Client.open(
    "https://planetarycomputer.microsoft.com/api/stac/v1/",
    # NOTE: autosigns URLs with SAS tokens, and adds 'credential' to 'table:storage_options'
    modifier=planetary_computer.sign_inplace, 
)

collection_id = "sentinel-2-l2a"

# STAC API
collection = catalog.get_collection(collection_id)
most_recent_item = next(collection.get_items())
print(most_recent_item.datetime)
# 2024-12-23 03:23:09.024000+00:00

# STAC GEOPARQUET
gpq = collection.assets["geoparquet-items"]
fs = adlfs.AzureBlobFileSystem(**gpq.extra_fields["table:storage_options"])
print(fs.ls(gpq.href)[-1:])
# ['items/sentinel-2-l2a.parquet/part-0469_2024-06-17T10:25:31+00:00_2024-06-24T10:25:31+00:00.parquet']

[TomAugspurger]

Large collections had a msft:partition_info property that defined how the data was temporally partitioned. We had a job run ~weekly that checked which partitions should exist by the current time, and would read from the database & write the geoparquet to Blob Storage to ensure those partitions existed. Not sure why that hasn't been updated recently.

As Kyle mentions, you'll probably want to use Iceberg or Delta instead of just plain parquet. That should make appending a bit more efficient. We didn't try to keep the geoparquet files up to date for low-latency workflows since we didn't want to be constantly rewriting the most recent partition as new data came in. The table formats might have better support for that.

[kylebarron]

You'd probably want to test out something like Iceberg or Delta Lake to manage ACID transactions on top of Parquet. Otherwise the simplest option would be to regenerate the Parquet files every X hours/days.

[drewm-swe]

Thanks for the feedback everyone.

In the case of a real-time data pipeline, I’ve been thinking that maybe the source of truth should be the GeoParquet and that the STAC JSON Catalog should be the one to lag behind. This would enable processing algorithms, responsible for generating low-latency derived data products, to efficiently access input data in the form of GeoParquet via Delta Lake, Apache Iceberg, etc. Then at some later point, the STAC JSON catalog would be updated based on the GeoParquet containing the most up to date data.

Have others used stac-geoparquet in this way? Am I thinking about this all wrong?