Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support searching by TEMPORAL_BASELINE_DAYS attribute for S1 ARIA GUNW products #198

Closed
asjohnston-asf opened this issue May 30, 2023 · 12 comments · Fixed by #199
Closed
Labels
enhancement New feature or request

Comments

@asjohnston-asf
Copy link
Member

New S1 ARIA GUNW products will soon include a TEMPORAL_BASELINE_DAYS attribute in CMR. @cmarshak is interested in being able to search by this parameter using asf_search.

For more context, see ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#136

This applies to the five CMR collections with ASF_PLATFORM=Sentinel-1 Interferogram (BETA); these CMR collection short names:

  • SENTINEL-1_INTERFEROGRAMS
  • SENTINEL-1_INTERFEROGRAMS_AMPLITUDE
  • SENTINEL-1_INTERFEROGRAMS_COHERENCE
  • SENTINEL-1_INTERFEROGRAMS_CONNECTED_COMPONENTS
  • SENTINEL-1_INTERFEROGRAMS_UNWRAPPED_PHASE
@asjohnston-asf asjohnston-asf added the enhancement New feature or request label May 30, 2023
@cmarshak
Copy link

I would be happy to help if there was some guidance to do this.

@asjohnston-asf
Copy link
Member Author

@cmarshak are you interested in exact matches only, e.g. "show me all products with a temporal baseline of 12?" Or do you have a use case for range searches, e.g. "show me all products with a temporal baseline between 12 and 36, inclusive?"

@cmarshak
Copy link

cmarshak commented May 30, 2023

I think exact matches because we want to associate a unique GUNW from:

  1. Range start-time
  2. Exact frame_id
  3. Exact temporal_baseline_days

@SpicyGarlicAlbacoreRoll
Copy link
Contributor

Opened a PR with temporalBaselineDays as a searchable keyword, allows searching via temporal baseline, or a list of temporal baselines that match the exact values provided (rather than ranges)

@cmarshak
Copy link

cmarshak commented May 30, 2023

For reference, on the UAT venue, we should be able to get a unique GUNW from

  • frame_id = 25502
  • temporal_baseline_days = 12
  • start_time in 2022-02-24 to 2022-02-25

@asjohnston-asf
Copy link
Member Author

@Cameronsplaze Cameronsplaze linked a pull request May 31, 2023 that will close this issue
@Cameronsplaze Cameronsplaze linked a pull request May 31, 2023 that will close this issue
@cmarshak
Copy link

Please let me know when this can be tested @SpicyGarlicAlbacoreRoll.

@SpicyGarlicAlbacoreRoll
Copy link
Contributor

SpicyGarlicAlbacoreRoll commented Jun 1, 2023

While asfPlatform isn't available in asf-search yet, it is possible to get the S1 GUNW products for the time being without it by setting platform to SENTINEL1.

using your example parameters, the following examples yields the 5 products for a single unique scene (S1-GUNW-A-R-164-tops-20220224_20220212-222854-00071W_00047N-PP-6c60-v3_0_0):

import asf_search as asf

opts = asf.ASFSearchOptions(host='cmr.uat.earthdata.nasa.gov', platform=asf.SENTINEL1, frame=25502, start=datetime(2022, 2, 24), end=datetime(2022, 2, 25), temporalBaselineDays=[12])
results = asf.search(opts=opts)

for f in results:
    print(f.properties['fileID'])

we are still considering whether or not to enable asfPlatform as a keyword argument, however if results do need to be narrowed to ASF_PLATFORM=Sentinel-1 Interferogram (BETA) products (say TEMPORAL_BASELINE_DAYS finds its way to other S1 collection products), either set the collections (available in the next release) or the processingLevel params like so:

(concept-ids)

collections = ['C1225776654-ASF','C1225776655-ASF','C1225776657-ASF','C1225776658-ASF','C1225776659-ASF'] # UAT S1 gunw concept-ids
opts = asf.ASFSearchOptions(host='cmr.uat.earthdata.nasa.gov', frame=25502, collections=collections, platform=asf.SENTINEL1, start=datetime(2022, 2, 24), end=datetime(2022, 2, 25), temporalBaselineDays=[12])

(processing level)

opts = asf.ASFSearchOptions(host='cmr.uat.earthdata.nasa.gov', frame=25502, processingLevel=['GUNW_UNW', 'GUNW_CON', 'GUNW_COH', 'GUNW_AMP', 'GUNW_STD'], platform=asf.SENTINEL1, start=datetime(2022, 2, 24), end=datetime(2022, 2, 25), temporalBaselineDays=[12])

Personally I'd recommend the collections keyword method once it's made available, since it will dodge the subquery system (less requests to CMR), unlike processingLevel.

@SpicyGarlicAlbacoreRoll
Copy link
Contributor

This should be available for you to test on the feature-temporal-baseline-days-search branch (PR #199 )

@SpicyGarlicAlbacoreRoll
Copy link
Contributor

@cmarshak feature-temporal-baseline-days-search is ready for you to try out. We likely won't go through with adding the asfPlatform keyword in a future release, but I've provided some alternative methods of limiting results to products from Sentinel-1 Interferogram (BETA) collections.

@cmarshak
Copy link

cmarshak commented Jun 5, 2023

Thanks, @SpicyGarlicAlbacoreRoll - this is what I will use:

import asf_search as asf
from datetime import datetime

opts = asf.ASFSearchOptions(host='cmr.uat.earthdata.nasa.gov', 
                            platform=asf.SENTINEL1,
                            processingLevel=asf.constants.GUNW_STD,
                            frame=25502, 
                            start=datetime(2022, 2, 24), 
                            end=datetime(2022, 2, 25), 
                            temporalBaselineDays=[12],
                            )
results = asf.search(opts=opts)

for f in results:
    print(f.properties['fileID'])

I used %%timeit to check the speed and found:

1.01 s ± 75.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

If I want to check about 250k products, is there any way to increase performance? I was going to multithread 5 requests at a time as I was told not to run more than that? Could I thread more?

Let me know your thoughts.

Also, the GUNWs are now official NASA products managed in part by @asjohnston-asf - no longer "Beta".

@SpicyGarlicAlbacoreRoll
Copy link
Contributor

I will say I haven't tried using asf-search's general search methods with multithreading (that use case to my knowledge remains untested), but I don't believe there's a particular limit on threads while using search() beyond what your machine can handle and how many simultaneously connections CMR will tolerate from a single IP address (again, this is untested).

If I were to try, I'd split my threads up by date ranges. Also, Including concept-ids with the collections keyword may yield faster results when querying CMR (at least this is behavior we've observed when searching for S1 granule_ur[]s with concept-ids in SearchAPI, though this hasn't been benchmarked with other more complex query params).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants