Data volume difference between Microsoft Planetary Computer and E84 for Sentinel-2?

[Kongstad]

Hello,

First of all let me just say I appreciate the work you guys are doing here and I really love using Microsoft Planetary Computer!

Now I have recently been working on a light weight client that handles loading of data from several STAC API's into one easy to use tool. Whilst this is working really well and I am very happy with MPC, E84, NASA LPDAAC and all the other providers out there. I am trying to understand why there is such a large discrepancy in the available data volume from Sentinel-2 when requested from MPC vs. E84.

I know that E84 recently started migrating to the new baseline release as Copernicus are processing and deploying it. I also know that E84 have had a lot of issues with duplicate files, because they stored different baselines in such a way that any baseline-blind query would return several baselines of the same image. So I always just assumed MPC had fewer images available because you guys kept a tighter ship on the data in terms of duplicates/baseline control. But as I am diving in to understand this descrepancy better and running large tests, I was hoping I could get some feedback here from MPC team and other users like me, to help me understand why I seem to get several thousand more pictures from E84 than I do from MPC.

I can of course provide specific query parameters if desired. But for now this is meant more as a theoretical question and to draw on others experience.

Thank you for your time!

[Kongstad]

Just to add a little more information.
For 1000 farmers field boundaries across the UK from the period of 2021-08-01 to 2022-06-01 these bands ['B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B09', 'B11', 'B12'], I get the following numbers:

Provider Satellite Items found
E84 Sentinel-2 98481
MPC Sentinel-2 76654

[TomAugspurger]

Thanks for opening this up. I'm partway through a "census" of our sentinel-2 inventory that's trying to get to the bottom of things. I don't have anything definitive yet, but I can confirm that we are missing some scenes. There are a few classes of issues, each of which might have a few causes. Some preliminary results:

1. Some scenes are missing from both our STAC API and Blob Storage. These might have failed because
   a.) The transfer (and all retries) from our upstream data provider failed. This shouldn't typically happen, but some exceptional circumstances might have led to missing scenes
   b.) The sen2cor process failed. Up until recently we ran sen2cor ourselves to produce the L2A data. This was... finnicky, and sen2cor would occasionally fail (sometimes retries on the same scene would succeed, sometimes not). We're now pulling L2A scenes from our upstream data provider, so I expect this to be less of an issue going forward.
2. Assets are present in Blob Storage, but not the STAC API. This might be because
   a.) Our processing pipeline, for some reason, failed to schedule STAC item creation for some particular scene
   b.) stactools.sentinel2 failed, for some reason, errored while creating a STAC item

Like I mentioned, this is in progress but I'll post here as we get a better understanding of the causes, and then address some of the issues by backfilling the missing scenes.

[Kongstad]

Happy to help Tom.
I am glad to hear you are looking into and I appreciate your feedback. Looking forward to seeing the result.

[Kongstad]

Hey,

I was wondering if you are nearing any solution or answers to this issue?

[777arc]

It seems like it's not just the number of items that is different, but also the actual items being returned for the same time/spatial query. Eg the following query gives 136 and 187 items for MPC and E84 respectively, less on MPC like you have been seeing, but using the value from an arbitrary field as sort of a unique identifier (the item id for each provider uses a different format) eg s2:medium_proba_clouds_percentage shows only two items that match between the two sets. The fact that there are a couple matches in the relatively small pool, not zero, tell me that MPC and E84 are using the same upstream data provider.

import pystac_client
import planetary_computer

catalog_MPC = pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1", modifier=planetary_computer.sign_inplace)
catalog_E84 = pystac_client.Client.open("https://earth-search.aws.element84.com/v1/", modifier=planetary_computer.sign_inplace)
collection = "sentinel-2-l2a"
time_of_interest = "2021-08-01/2022-06-01"
area_of_interest = {
    "type": "Polygon",
    "coordinates": [
        [
            [-148.56536865234375, 60.80072385643073],
            [-147.44338989257812, 60.80072385643073],
            [-147.44338989257812, 61.18363894915102],
            [-148.56536865234375, 61.18363894915102],
            [-148.56536865234375, 60.80072385643073],
        ]
    ],
}
items_MPC = catalog_MPC.search(collections=[collection], intersects=area_of_interest, datetime=time_of_interest).item_collection()
items_E84 = catalog_E84.search(collections=[collection], intersects=area_of_interest, datetime=time_of_interest).item_collection()
print(f"MPC: {len(items_MPC)} Items")
print(f"E84: {len(items_E84)} Items")

uris_MPC = [item.properties["s2:product_uri"] for item in items_MPC]
uris_E84 = [item.properties["s2:product_uri"] for item in items_E84]
print(len(uris_MPC))
print(len(uris_E84))
uris_MPC = set(uris_MPC)
uris_E84 = set(uris_E84)

MPC = set(item.properties["s2:medium_proba_clouds_percentage"] for item in items_MPC)
E84 = set(item.properties["s2:medium_proba_clouds_percentage"] for item in items_E84)
print(len(MPC))
print(len(E84))
print(len(MPC.intersection(E84)))
print(MPC.intersection(E84))

MPC: 136 Items
E84: 187 Items
136
187
136
187
2
{99.903607, 99.444407}