diff --git a/CHANGELOG.md b/CHANGELOG.md index ef7cc1d9..7726d27f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/) and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.3.1] + +### Added +* Provides CLI options for: + - output resolution: 30 meters or 90 meters (the latter is default and the standardized GUNW resolution) + - unfiltered coherence layer (True/False) + - Goldstein filtering power - power that phase is raised to in patch FFT - default .5 (can be any number >= 0) + - Dense offsets layers (True/false) +* Codifies (in documentation and in plugin) what is meant by "standard" GUNW with respect to exposed parameters including: + - 90 m resolution + - .5 value in the Goldstein filter for InSAR phase + - No ESD or dense offsets + - Additional layers: ionosophere, SET, and unfiltered coherence. + - uses pydantic to record relevant topsapp parameters for "standard" GUNW +* Records parameters in the product including the CLI command to regenerate said product +* If parameters are not standard uses prefix `S1-GUNW_CUSTOM-...` +* Pydantic dependency for parameter accounting + +### Changed +* The CLI now *requires* `frame_id` (use `frame_id = -1` for old API and what is now considered a "non"-standard product) + + + ## [0.3.0] ### Added @@ -81,7 +104,6 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). * Fixes write of start/stop sensing times due to changes in ASF Search v5.0.0 (see #79) - ## [0.2.0] ### Added @@ -95,6 +117,7 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixed * Uses dem-stitcher>=v2.3.0, which by default, fills in `glo-30` tiles that are missing over Armenia and Azerbaijan with the available `glo-90` tiles (upsampled). * Uses dem-stitcher>=v2.3.1 to fix URLs for `glo-30` and `srtm_v3` + ## [0.1.2] ### Fixed diff --git a/README.md b/README.md index d92afa54..85300fd1 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DockerizedTopsApp (aka ISCE2 TopsApp Hyp3-Plugin) -This repository represents a dockerized science processor for generating an ARIA Sentinel-1 [Geocoded Unwrapped Interferogram](https://aria.jpl.nasa.gov/products/standard-displacement-products.html) (GUNW) product from a collection of valid Sentinel-1 IW-mode Single Look Complex (SLC) IDs across a date pair using [ISCE2](https://github.com/isce-framework/isce2). The GUNW is a NISAR beta-product. The initial development of the GUNW was done under the Getting Ready for NISAR initiative and a collection of related ARIA-funded projects. This work has continued under the Project Enabling Cloud-Based InSAR Science for an Exploding NASA InSAR Data Archive (ACCESS19-0023) funded under the ACCESS program. +This repository represents a dockerized science processor for generating an ARIA Sentinel-1 [Geocoded Unwrapped Interferogram](https://asf.alaska.edu/data-sets/derived-data-sets/sentinel-1-interferograms/) (ARIA-S1-GUNW) product from a collection of valid Sentinel-1 IW-mode Single Look Complex (SLC) IDs across a date pair using [ISCE2](https://github.com/isce-framework/isce2). The ARIA-S1-GUNW (or simply a GUNW) is an official NASA product. The initial development of the GUNW was done under the Getting Ready for NISAR initiative and a collection of related ARIA-funded projects. This work has continued under the Project Enabling Cloud-Based InSAR Science for an Exploding NASA InSAR Data Archive (ACCESS19-0023) funded under the ACCESS program. A description of the product can be found here: https://aria.jpl.nasa.gov/products/standard-displacement-products.html This processor plugs into the [HyP3](https://hyp3-docs.asf.alaska.edu/v2-transition/) platform and therefore can spawn processing at scale from an API. All of the necessary datasets required for processing are determined from the input SLC IDS and then downloaded from public APIs. Thus, this repository accomplishes two goals: @@ -37,68 +37,92 @@ We note all the input datasets are publicly available using a NASA Earthdata acc ``` The first `username`/`password` pair are the appropriate Earthdata Login credentials that are used to access NASA data. The second pair are your credentials for the [Copernicus Data Space Ecosystem](https://dataspace.copernicus.eu). This file is necessary for downloading the Sentinel-1 files, and auxiliary data. Additionally, the [`requests`](https://docs.python-requests.org/en/latest/) library automatically uses credentials stored in the `~/.netrc` for authentification when none are supplied. -## Generate a GUNW +## Generate an ARIA-S1-GUNW -Make sure you have `~/.netrc`. Run the following command: +Make sure you have `~/.netrc` as described above. Run the following command: ``` -isce2_topsapp --reference-scenes S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9 \ - --secondary-scenes S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D \ - S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404 \ - S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C +isce2_topsapp --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E \ + S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 \ + --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F \ + S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 \ + --frame-id 25502 ``` Add `> topsapp_img.out 2> topsapp_img.err` to avoid unnecessary output to your terminal and record the stdout and stderr as files. This is reflected in the [`sample_run.sh`](sample_run.sh). To be even more explicity, you can use [`tee`](https://en.wikipedia.org/wiki/Tee_(command)) to record output to both including `> >(tee -a topsapp_img.out) 2> >(tee -a topsapp_img.err >&2)`. -## Customizations +## What makes an ARIA-S1-GUNW Product *standard*? -### Estimating Ionospheric Phase Delay and ESD +Each ARIA-S1-GUNW at the ASF that ensures that down-stream analysis by [ARIA-Tools](https://github.com/aria-tools/ARIA-tools) and [Mintpy](https://github.com/insarlab/MintPy) is done consistently and reproducibly. There are a number of exposed parameters in this plugin that we require to be set in a certain manner for a product to be considered "standard". We now discuss the standard parameters with respect to this plugin. -This example shows how to obtain a layer with ionsopheric phase delay. The SLCs are over the Arabian peninusula where the ionosphere can be seen: +Since v3+, in addition to reference and secondary scenes, a `frame-id` must be supplied for a *standard* product to be generated. This effectively restricts processing and the resulting product to be within this frame (technically, all bursts within the frame are included in the standard product). The geojson of spatially-fixed frames with their ids can be downloaded [here](https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/data/s1_frames_latitude_aligned.geojson.zip). These are derived from ESA's burst [map](https://sar-mpc.eu/test-data-sets/). More information about finding SLC pairs and their corresponding pairs can be found [here](https://github.com/ACCESS-Cloud-Based-InSAR/s1-frame-enumerator) and the generation of our spatially fixed-frames is discussed [here](https://github.com/ACCESS-Cloud-Based-InSAR/s1-frame-generation). -``` -isce2_topsapp --reference-scenes S1B_IW_SLC__1SDV_20171117T145926_20171117T145953_008323_00EBAB_AFB8 \ - --secondary-scenes S1A_IW_SLC__1SDV_20171111T150004_20171111T150032_019219_0208AF_EE89 \ - --estimate-ionosphere-delay True \ - --esd-coherence-threshold .5 \ - > topsapp_img.out 2> topsapp_img.err -``` -Not including `--esd-coherence-threshold` means no ESD correction will be applied. The ESD threshold refers to a coherence value and therefore must be in $[0, 1]$. +All standard products have the following layers: ++ Data Layers (0.00083333333 deg or ~90 m at the equator) + + Unwrapped phase + + Coherence + + Connected compenents + + Unfiltered coherence - *new* in version 3❗ + + InSAR amplitude ++ Correction Layers + + Ionosphere (0.00916 deg or ~1 km at the equator) - *new* in version 3❗ + + Solid earth tide (.1 deg or ~11 km at the equator) - *new* in version 3❗ + + Tropo correction layers if HRRR available (see [RAiDER](https://github.com/dbekaert/RAiDER)) ++ Geometry Layers (.1 deg or ~11 km) + + Incidence angle + + Azimuth angle + + Parallel baseline + + Perpendicular baseline + + Lat/lon grids -### Apply Solid Earth Tide Correction +Again, tropo corrections are controlled via a separate step-function so is not included above. The repository is [here](https://github.com/dbekaert/RAiDER). Turning off certain layers or adding available layers using the CLI arguments are permissible but will produce *custom* products (indicated with a prefix `S1-GUNW_CUSTOM...`). The parameters are often simply exposing certain topsApp parameters discussed [here](https://github.com/isce-framework/isce2/blob/main/applications/topsApp.py). Our template for topsapp that is utilized for ISCE is found [here](https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/templates/topsapp_template.xml). -``` -isce2_topsapp --reference-scenes S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9 \ - --secondary-scenes S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D \ - S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404 \ - S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C \ - --compute-solid-earth-tide True \ - > topsapp_img_set.out 2> topsapp_img_set.err -``` +The command line string and relevant plugin version used to generate every product is included in the product itself and can be used to reproduce a product. These are attributes in the top level netcdf group. + +We note that the ionosphere correction layer is the (hard) work of [Marin Govorcin](https://github.com/mgovorcin) and David Bekaert, which utilizes ISCE2 in a creative fashion. Users should refer to [this file](https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/iono_proc.py) for the process. -### Using "fixed frames" (experimental) +Below indicates all available arguments for product generation and parameters required for *standard product* generation (again, for a given pairing and frame, one must use the enumeration of pairs described [here](https://github.com/ACCESS-Cloud-Based-InSAR/s1-frame-enumerator)). Use `isce2_topsapp --help` for more information of available arguments. -Sentinel-1 Frames are not constant over passes. We generate fixed frames [here](https://github.com/ACCESS-Cloud-Based-InSAR/s1-frame-generation) and enumerate interferograms using this [repo](https://github.com/ACCESS-Cloud-Based-InSAR/s1-frame-enumerator). This is highly experimental. We then ensure ISCE processes only over the frame. The key is overlap. We provide some examples of the additional options (you will need to run this in *two* separate directories because ISCE2 outputs are organized with respect to the working directory of the processing). For one frame over CA: ``` -isce2_topsapp --reference-scenes S1A_IW_SLC__1SDV_20230125T135954_20230125T140021_046941_05A132_D35C \ - S1A_IW_SLC__1SDV_20230125T140019_20230125T140046_046941_05A132_82DF \ - --secondary-scenes S1A_IW_SLC__1SDV_20221220T135956_20221220T140023_046416_058F77_B248 \ - S1A_IW_SLC__1SDV_20221220T140020_20221220T140047_046416_058F77_5213 \ - --frame-id 22438 \ - > topsapp_img_f22438.out 2> topsapp_img_f22438.err +isce2_topsapp --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E \ + S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 \ + --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F \ + S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 \ + --frame-id 25502 # latitude aligned ARIA spatially fixed frame\ + --estimate-ionosphere-delay True # ionosphere correction layers\ + --esd-coherence-threshold -1. # if -1, ESD is not used; else should be a value in (0, 1)\ + --compute_solid_earth_tide True \ + --goldstein-filter-power 0.5 # the power of the patch FFT filter used in the Goldstein filter\ + --output-resolution 90 # either 30 or 90 meters\ + --unfiltered-coherence True # this adds an unfiltered coherence layer\ + --dense-offsets False # adds layers that compute patch wise correlation measurement done in range and azimuth which are helpful after significant surface changes\ ``` -and an overlapping frame: +or as a json: ``` -isce2_topsapp --reference-scenes S1A_IW_SLC__1SDV_20230125T140019_20230125T140046_046941_05A132_82DF \ - S1A_IW_SLC__1SDV_20230125T140044_20230125T140111_046941_05A132_59E7 \ - --secondary-scenes S1A_IW_SLC__1SDV_20221220T140020_20221220T140047_046416_058F77_5213 \ - S1A_IW_SLC__1SDV_20221220T140045_20221220T140112_046416_058F77_7692 \ - --frame-id 22439 \ - > topsapp_img_f22439.out 2> topsapp_img_f22439.err +{ + "reference_scenes": [ + "S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E", + "S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2" + ], + "secondary_scenes": [ + "S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F", + "S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7" + ], + "frame_id": 25502, + "estimate_ionosphere_delay": true, + "compute_solid_earth_tide": true, + "output_resolution": 90, + "unfiltered_coherence": true, + "goldstein_filter_power": 0.5, + "dense_offsets": false, + "wrapped_phase_layer": false, + "esd_coherence_threshold": -1.0 +} ``` + # Running with Docker (locally or on a server) 1. When running locally with root privileges (i.e. at your local workstation), build the docker image using: @@ -113,14 +137,11 @@ isce2_topsapp --reference-scenes S1A_IW_SLC__1SDV_20230125T140019_20230125T14004 3. Create a directory to mount the data files so you can inspect them outside of your docker container. Call it `topsapp_data`. Navigate to it. Copy the `sample_run.sh` in this directory, modifying it to add your Earthdata username and password e.g. ``` - isce2_topsapp --username \ - --password \ - --esa-username \ - --esa-password \ - --reference-scenes S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9 \ - --secondary-scenes S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D \ - S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404 \ - S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C \ + isce2_topsapp --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E \ + S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 \ + --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F \ + S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 \ + --frame-id 25502 \ > topsapp_img.out 2> topsapp_img.err ``` @@ -140,10 +161,11 @@ Create a new directory (for all the intermediate files) and navigate to it. ``` docker run -ti -v $PWD:/home/ops/topsapp_data topsapp_img \ - --reference-scenes S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9 \ - --secondary-scenes S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D \ - S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404 \ - S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C \ + --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E \ + S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 \ + --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F \ + S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 \ + --frame-id 25502 \ --username --password --esa-username \ diff --git a/Untitled.ipynb b/Untitled.ipynb new file mode 100644 index 00000000..2725ca2a --- /dev/null +++ b/Untitled.ipynb @@ -0,0 +1,368 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "43a70bf0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "This is the Open Source version of ISCE.\n", + "Some of the workflows depend on a separate licensed package.\n", + "To obtain the licensed package, please make a request for ISCE\n", + "through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.\n", + "Alternatively, if you are a member, or can become a member of WinSAR\n", + "you may be able to obtain access to a version of the licensed sofware at\n", + "https://winsar.unavco.org/software/isce\n", + "2023-11-08 12:49:09,028 - matplotlib.font_manager - DEBUG - Using fontManager instance from /Users/cmarshak/.matplotlib/fontlist-v330.json\n" + ] + } + ], + "source": [ + "from isce2_topsapp.__main__ import get_slc_parser, true_false_string_argument\n", + "from argparse import ArgumentParser\n", + "import xarray as xr\n", + "import h5py" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "49e167bb", + "metadata": {}, + "outputs": [], + "source": [ + "path = 'S1-GUNW-A-R-064-tops-20210723_20210711-015000-00119W_00033N-PP-6267-v2_0_6.nc'" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "486971e2", + "metadata": {}, + "outputs": [], + "source": [ + "with h5py.File(path, mode='a') as file:\n", + " \n", + " file.attrs.update(dict(oy_vey=1))" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "55d6a081", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['product_type',\n", + " 'Conventions',\n", + " 'title',\n", + " 'version',\n", + " 'author',\n", + " 'institution',\n", + " 'references',\n", + " '_NCProperties',\n", + " 'ogr_geometry_field',\n", + " 'ogr_layer_name',\n", + " 'ogr_layer_type',\n", + " 'source',\n", + " 'testing',\n", + " 'test',\n", + " 'oy_vey']" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "f2 = h5py.File(path)\n", + "list(f2.attrs)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2ef90ce1", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "407841d2", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\n", + "root group (NETCDF4 data model, file format HDF5):\n", + " product_type: UNW GEO IFG\n", + " Conventions: CF-1.6\n", + " title: ARIA standard product UNW GEO IFG\n", + " version: 1\n", + " author: David Bekaert, Charlie Marshak, Simran Sangha, Joe Kennedy, Andrew Johnston\n", + " institution: Jet Propulsion Laboratory\n", + " references: https://aria.jpl.nasa.gov/\n", + " ogr_geometry_field: productBoundingBox\n", + " ogr_layer_name: productBoundingBox\n", + " ogr_layer_type: POLYGON\n", + " source: Contains modified Copernicus Sentinel data processed by ESA and ARIA NASA/JPL using the DockerizedTopsApp HyP3 plugin version 0.2.2.dev40+g679dfc6\n", + " testing: 1\n", + " dimensions(sizes): matchup(2), wkt_length(464), wkt_count(1)\n", + " variables(dimensions): |S1 productBoundingBox(wkt_count, wkt_length), int32 crs_polygon()\n", + " groups: science" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "ds = netCDF4.Dataset(path, mode='a')\n", + "ds" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "cf18ec99", + "metadata": {}, + "outputs": [ + { + "ename": "NameError", + "evalue": "name 'setncattr' is not defined", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[13], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43msetncattr\u001b[49m\n", + "\u001b[0;31mNameError\u001b[0m: name 'setncattr' is not defined" + ] + } + ], + "source": [ + "ds.setncattr()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8b68feb0", + "metadata": {}, + "outputs": [], + "source": [ + "ds.setncattr('testing', '1')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1d558deb", + "metadata": {}, + "outputs": [], + "source": [ + "#ds.close()\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "a1e09885", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Conventions',\n", + " '__class__',\n", + " '__delattr__',\n", + " '__dir__',\n", + " '__doc__',\n", + " '__enter__',\n", + " '__eq__',\n", + " '__exit__',\n", + " '__format__',\n", + " '__ge__',\n", + " '__getattr__',\n", + " '__getattribute__',\n", + " '__getitem__',\n", + " '__gt__',\n", + " '__hash__',\n", + " '__init__',\n", + " '__init_subclass__',\n", + " '__le__',\n", + " '__lt__',\n", + " '__ne__',\n", + " '__new__',\n", + " '__orthogonal_indexing__',\n", + " '__reduce__',\n", + " '__reduce_ex__',\n", + " '__repr__',\n", + " '__setattr__',\n", + " '__sizeof__',\n", + " '__str__',\n", + " '__subclasshook__',\n", + " '_close',\n", + " '_close_mem',\n", + " '_enddef',\n", + " '_getname',\n", + " '_grpid',\n", + " '_isopen',\n", + " '_ncstring_attrs__',\n", + " '_redef',\n", + " 'author',\n", + " 'close',\n", + " 'cmptypes',\n", + " 'createCompoundType',\n", + " 'createDimension',\n", + " 'createEnumType',\n", + " 'createGroup',\n", + " 'createVLType',\n", + " 'createVariable',\n", + " 'data_model',\n", + " 'delncattr',\n", + " 'dimensions',\n", + " 'disk_format',\n", + " 'enumtypes',\n", + " 'file_format',\n", + " 'filepath',\n", + " 'fromcdl',\n", + " 'get_variables_by_attributes',\n", + " 'getncattr',\n", + " 'groups',\n", + " 'has_blosc_filter',\n", + " 'has_bzip2_filter',\n", + " 'has_szip_filter',\n", + " 'has_zstd_filter',\n", + " 'institution',\n", + " 'isopen',\n", + " 'keepweakref',\n", + " 'name',\n", + " 'ncattrs',\n", + " 'ogr_geometry_field',\n", + " 'ogr_layer_name',\n", + " 'ogr_layer_type',\n", + " 'parent',\n", + " 'path',\n", + " 'product_type',\n", + " 'references',\n", + " 'renameAttribute',\n", + " 'renameDimension',\n", + " 'renameGroup',\n", + " 'renameVariable',\n", + " 'set_always_mask',\n", + " 'set_auto_chartostring',\n", + " 'set_auto_mask',\n", + " 'set_auto_maskandscale',\n", + " 'set_auto_scale',\n", + " 'set_fill_off',\n", + " 'set_fill_on',\n", + " 'set_ncstring_attrs',\n", + " 'setncattr',\n", + " 'setncattr_string',\n", + " 'setncatts',\n", + " 'source',\n", + " 'sync',\n", + " 'title',\n", + " 'tocdl',\n", + " 'variables',\n", + " 'version',\n", + " 'vltypes']" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dir(ds)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "f37e82b2", + "metadata": {}, + "outputs": [], + "source": [ + "with xr.open_dataset(path, group='input_parameters') as ds:\n", + " ds = ds.assign_attrs(test_1 = 1)\n", + "ds.to_netcdf(path)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "89023a90", + "metadata": {}, + "outputs": [ + { + "ename": "OSError", + "evalue": "[Errno group not found: input_parameters] 'input_parameters'", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:187\u001b[0m, in \u001b[0;36m_nc4_require_group\u001b[0;34m(ds, group, mode, create_group)\u001b[0m\n\u001b[1;32m 186\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 187\u001b[0m ds \u001b[38;5;241m=\u001b[39m \u001b[43mds\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgroups\u001b[49m\u001b[43m[\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m]\u001b[49m\n\u001b[1;32m 188\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n", + "\u001b[0;31mKeyError\u001b[0m: 'input_parameters'", + "\nDuring handling of the above exception, another exception occurred:\n", + "\u001b[0;31mOSError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[9], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m ds \u001b[38;5;241m=\u001b[39m \u001b[43mxr\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mopen_dataset\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpath\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43minput_parameters\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 2\u001b[0m ds\n", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/api.py:566\u001b[0m, in \u001b[0;36mopen_dataset\u001b[0;34m(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)\u001b[0m\n\u001b[1;32m 554\u001b[0m decoders \u001b[38;5;241m=\u001b[39m _resolve_decoders_kwargs(\n\u001b[1;32m 555\u001b[0m decode_cf,\n\u001b[1;32m 556\u001b[0m open_backend_dataset_parameters\u001b[38;5;241m=\u001b[39mbackend\u001b[38;5;241m.\u001b[39mopen_dataset_parameters,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 562\u001b[0m decode_coords\u001b[38;5;241m=\u001b[39mdecode_coords,\n\u001b[1;32m 563\u001b[0m )\n\u001b[1;32m 565\u001b[0m overwrite_encoded_chunks \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mpop(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124moverwrite_encoded_chunks\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[0;32m--> 566\u001b[0m backend_ds \u001b[38;5;241m=\u001b[39m \u001b[43mbackend\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mopen_dataset\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 567\u001b[0m \u001b[43m \u001b[49m\u001b[43mfilename_or_obj\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 568\u001b[0m \u001b[43m \u001b[49m\u001b[43mdrop_variables\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdrop_variables\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 569\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mdecoders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 570\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 571\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 572\u001b[0m ds \u001b[38;5;241m=\u001b[39m _dataset_from_backend_dataset(\n\u001b[1;32m 573\u001b[0m backend_ds,\n\u001b[1;32m 574\u001b[0m filename_or_obj,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 584\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[1;32m 585\u001b[0m )\n\u001b[1;32m 586\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m ds\n", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:590\u001b[0m, in \u001b[0;36mNetCDF4BackendEntrypoint.open_dataset\u001b[0;34m(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose)\u001b[0m\n\u001b[1;32m 569\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mopen_dataset\u001b[39m( \u001b[38;5;66;03m# type: ignore[override] # allow LSP violation, not supporting **kwargs\u001b[39;00m\n\u001b[1;32m 570\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m 571\u001b[0m filename_or_obj: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m|\u001b[39m os\u001b[38;5;241m.\u001b[39mPathLike[Any] \u001b[38;5;241m|\u001b[39m BufferedIOBase \u001b[38;5;241m|\u001b[39m AbstractDataStore,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 587\u001b[0m autoclose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m,\n\u001b[1;32m 588\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Dataset:\n\u001b[1;32m 589\u001b[0m filename_or_obj \u001b[38;5;241m=\u001b[39m _normalize_path(filename_or_obj)\n\u001b[0;32m--> 590\u001b[0m store \u001b[38;5;241m=\u001b[39m \u001b[43mNetCDF4DataStore\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mopen\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 591\u001b[0m \u001b[43m \u001b[49m\u001b[43mfilename_or_obj\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 592\u001b[0m \u001b[43m \u001b[49m\u001b[43mmode\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmode\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 593\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mformat\u001b[39;49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mformat\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 594\u001b[0m \u001b[43m \u001b[49m\u001b[43mgroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgroup\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 595\u001b[0m \u001b[43m \u001b[49m\u001b[43mclobber\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mclobber\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 596\u001b[0m \u001b[43m \u001b[49m\u001b[43mdiskless\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdiskless\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 597\u001b[0m \u001b[43m \u001b[49m\u001b[43mpersist\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpersist\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 598\u001b[0m \u001b[43m \u001b[49m\u001b[43mlock\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlock\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 599\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoclose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoclose\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 600\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 602\u001b[0m store_entrypoint \u001b[38;5;241m=\u001b[39m StoreBackendEntrypoint()\n\u001b[1;32m 603\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m close_on_error(store):\n", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:391\u001b[0m, in \u001b[0;36mNetCDF4DataStore.open\u001b[0;34m(cls, filename, mode, format, group, clobber, diskless, persist, lock, lock_maker, autoclose)\u001b[0m\n\u001b[1;32m 385\u001b[0m kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mdict\u001b[39m(\n\u001b[1;32m 386\u001b[0m clobber\u001b[38;5;241m=\u001b[39mclobber, diskless\u001b[38;5;241m=\u001b[39mdiskless, persist\u001b[38;5;241m=\u001b[39mpersist, \u001b[38;5;28mformat\u001b[39m\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mformat\u001b[39m\n\u001b[1;32m 387\u001b[0m )\n\u001b[1;32m 388\u001b[0m manager \u001b[38;5;241m=\u001b[39m CachingFileManager(\n\u001b[1;32m 389\u001b[0m netCDF4\u001b[38;5;241m.\u001b[39mDataset, filename, mode\u001b[38;5;241m=\u001b[39mmode, kwargs\u001b[38;5;241m=\u001b[39mkwargs\n\u001b[1;32m 390\u001b[0m )\n\u001b[0;32m--> 391\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mcls\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mmanager\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgroup\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmode\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmode\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlock\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlock\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mautoclose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoclose\u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:338\u001b[0m, in \u001b[0;36mNetCDF4DataStore.__init__\u001b[0;34m(self, manager, group, mode, lock, autoclose)\u001b[0m\n\u001b[1;32m 336\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_group \u001b[38;5;241m=\u001b[39m group\n\u001b[1;32m 337\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_mode \u001b[38;5;241m=\u001b[39m mode\n\u001b[0;32m--> 338\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mformat \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mds\u001b[49m\u001b[38;5;241m.\u001b[39mdata_model\n\u001b[1;32m 339\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_filename \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mds\u001b[38;5;241m.\u001b[39mfilepath()\n\u001b[1;32m 340\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mis_remote \u001b[38;5;241m=\u001b[39m is_remote_uri(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_filename)\n", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:400\u001b[0m, in \u001b[0;36mNetCDF4DataStore.ds\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 398\u001b[0m \u001b[38;5;129m@property\u001b[39m\n\u001b[1;32m 399\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mds\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[0;32m--> 400\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_acquire\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:395\u001b[0m, in \u001b[0;36mNetCDF4DataStore._acquire\u001b[0;34m(self, needs_lock)\u001b[0m\n\u001b[1;32m 393\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_acquire\u001b[39m(\u001b[38;5;28mself\u001b[39m, needs_lock\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m):\n\u001b[1;32m 394\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_manager\u001b[38;5;241m.\u001b[39macquire_context(needs_lock) \u001b[38;5;28;01mas\u001b[39;00m root:\n\u001b[0;32m--> 395\u001b[0m ds \u001b[38;5;241m=\u001b[39m \u001b[43m_nc4_require_group\u001b[49m\u001b[43m(\u001b[49m\u001b[43mroot\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_group\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_mode\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 396\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m ds\n", + "File \u001b[0;32m~/mambaforge/envs/topsapp_env/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:193\u001b[0m, in \u001b[0;36m_nc4_require_group\u001b[0;34m(ds, group, mode, create_group)\u001b[0m\n\u001b[1;32m 190\u001b[0m ds \u001b[38;5;241m=\u001b[39m create_group(ds, key)\n\u001b[1;32m 191\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 192\u001b[0m \u001b[38;5;66;03m# wrap error to provide slightly more helpful message\u001b[39;00m\n\u001b[0;32m--> 193\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mOSError\u001b[39;00m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mgroup not found: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mkey\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, e)\n\u001b[1;32m 194\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m ds\n", + "\u001b[0;31mOSError\u001b[0m: [Errno group not found: input_parameters] 'input_parameters'" + ] + } + ], + "source": [ + "ds = xr.open_dataset(path, group='input_parameters')\n", + "ds" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "48688cd3", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "topsapp_env", + "language": "python", + "name": "topsapp_env" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.16" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/environment.yml b/environment.yml index bc50e176..43161fec 100644 --- a/environment.yml +++ b/environment.yml @@ -31,6 +31,7 @@ dependencies: - pysolid - papermill - pytest + - pydantic - pytest-cov - pytest-mock - rasterio diff --git a/isce2_topsapp/__init__.py b/isce2_topsapp/__init__.py index 671a71fd..661f1e51 100644 --- a/isce2_topsapp/__init__.py +++ b/isce2_topsapp/__init__.py @@ -3,12 +3,14 @@ from isce2_topsapp.delivery_prep import prepare_for_delivery from isce2_topsapp.localize_aux_cal import download_aux_cal -from isce2_topsapp.localize_burst import BurstParams, download_bursts, get_region_of_interest +from isce2_topsapp.localize_burst import (BurstParams, download_bursts, + get_region_of_interest) from isce2_topsapp.localize_dem import download_dem_for_isce2 from isce2_topsapp.localize_mask import download_water_mask from isce2_topsapp.localize_orbits import download_orbits from isce2_topsapp.localize_slc import download_slcs, get_asf_slc_objects from isce2_topsapp.packaging import package_gunw_product +from isce2_topsapp.topsapp_params import topsappParams from isce2_topsapp.topsapp_proc import topsapp_processing try: @@ -30,6 +32,7 @@ 'download_bursts', 'BurstParams', 'topsapp_processing', + 'topsappParams', 'package_gunw_product', 'prepare_for_delivery', '__version__', diff --git a/isce2_topsapp/__main__.py b/isce2_topsapp/__main__.py index d2ef84a7..aa435693 100644 --- a/isce2_topsapp/__main__.py +++ b/isce2_topsapp/__main__.py @@ -3,27 +3,27 @@ import netrc import os import sys -from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser +from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace from importlib.metadata import entry_points from pathlib import Path from platform import system from typing import Optional -from isce2_topsapp import ( - BurstParams, - aws, - download_aux_cal, - download_bursts, - download_dem_for_isce2, - download_orbits, - download_slcs, - download_water_mask, - get_asf_slc_objects, - get_region_of_interest, - package_gunw_product, - prepare_for_delivery, - topsapp_processing, -) +from isce2_topsapp import (BurstParams, + aws, + download_aux_cal, + download_bursts, + download_dem_for_isce2, + download_orbits, + download_slcs, + download_water_mask, + get_asf_slc_objects, + get_region_of_interest, + package_gunw_product, + prepare_for_delivery, + topsappParams, + topsapp_processing, + ) from isce2_topsapp.iono_proc import iono_processing from isce2_topsapp.json_encoder import MetadataEncoder from isce2_topsapp.packaging import update_gunw_internal_version_attribute @@ -39,6 +39,8 @@ def localize_data( frame_id: int = -1, dry_run: bool = False, water_mask_flag: bool = True, + geocode_resolution: int = 90 + ) -> dict: """The dry-run prevents gets necessary metadata from SLCs and orbits. @@ -57,7 +59,9 @@ def localize_data( out_dem = {} out_aux_cal = {} if not dry_run: - out_dem = download_dem_for_isce2(out_slc["extent"]) + out_dem = download_dem_for_isce2(out_slc['extent'], + geocode_resolution=geocode_resolution) + out_water_mask = {"water_mask": None} # For ionospheric correction computation if water_mask_flag: @@ -152,44 +156,46 @@ def true_false_string_argument(s: str) -> bool: def esd_threshold_argument(threshold: str) -> float: - threshold = float(threshold) + threshold_float = float(threshold) - if math.isclose(threshold, -1.0): - return threshold + if math.isclose(threshold_float, -1.0): + return threshold_float - if (0.0 > threshold) or (threshold > 1.0): + if (0.0 > threshold_float) or (threshold_float > 1.0): raise ValueError( "ESD coherence threshold should be a value between 0 and 1," " or -1 for no ESD correction" ) - return threshold + return threshold_float -def gunw_slc(): +def get_slc_parser(): parser = ArgumentParser() - parser.add_argument("--username") - parser.add_argument("--password") - parser.add_argument("--esa-username") - parser.add_argument("--esa-password") - parser.add_argument("--bucket") - parser.add_argument("--bucket-prefix", default="") - parser.add_argument("--dry-run", action="store_true") - parser.add_argument("--reference-scenes", - type=str.split, nargs="+", required=True) - parser.add_argument("--secondary-scenes", - type=str.split, nargs="+", required=True) - parser.add_argument( - "--estimate-ionosphere-delay", type=true_false_string_argument, default=False - ) - parser.add_argument("--frame-id", type=int, default=-1) - parser.add_argument( - "--compute-solid-earth-tide", type=true_false_string_argument, default=False - ) - parser.add_argument("--esd-coherence-threshold", type=float, default=-1.0) - args = parser.parse_args() - - ensure_earthdata_credentials(args.username, args.password) - check_esa_credentials(args.esa_username, args.esa_password) + parser.add_argument('--username', ) + parser.add_argument('--password') + parser.add_argument('--bucket') + parser.add_argument('--bucket-prefix', default='') + parser.add_argument('--dry-run', action='store_true') + parser.add_argument('--reference-scenes', type=str.split, nargs='+', required=True) + parser.add_argument('--secondary-scenes', type=str.split, nargs='+', required=True) + parser.add_argument('--estimate-ionosphere-delay', type=true_false_string_argument, default=True) + parser.add_argument('--frame-id', type=int, default=-1, required=True, + help=('If -1 is specified, no frame is used and a non-standard product generated. ' + 'See examples in repository. For generating SLC pairs and a fixed frame, see:' + 'https://github.com/ACCESS-Cloud-Based-InSAR/s1-frame-enumerator')) + parser.add_argument('--compute-solid-earth-tide', type=true_false_string_argument, default=True) + parser.add_argument('--esd-coherence-threshold', type=float, default=-1.) + parser.add_argument('--output-resolution', type=int, default=90, required=False) + parser.add_argument('--unfiltered-coherence', type=true_false_string_argument, default=True) + parser.add_argument('--dense-offsets', type=true_false_string_argument, default=False) + parser.add_argument('--goldstein-filter-power', type=float, default=.5, + help="The power applied to the patch FFT of the phase filter") + parser.add_argument("--esa-username", help='Username (i.e. email) for "https://dataspace.copernicus.eu/"') + parser.add_argument("--esa-password", help='Password for "https://dataspace.copernicus.eu/"') + return parser + + +def update_slc_namespace(args: Namespace) -> Namespace: args.reference_scenes = [ item for sublist in args.reference_scenes for item in sublist @@ -198,29 +204,51 @@ def gunw_slc(): item for sublist in args.secondary_scenes for item in sublist ] + args.esd_coherence_threshold = esd_threshold_argument(args.esd_coherence_threshold) + + if args.goldstein_filter_power < 0: + raise ValueError('Goldstein filter power must be non-negative') + + return args + + +def gunw_slc(): + cmd_line_str = 'isce2_topsapp ++' + ' '.join(sys.argv) + + parser = get_slc_parser() + args = parser.parse_args() + args = update_slc_namespace(args) + + # Validation + ensure_earthdata_credentials(args.username, args.password) + check_esa_credentials(args.esa_username, args.esa_password) + cli_params = vars(args).copy() + [cli_params.pop(key) for key in ['username', 'password', 'bucket', 'bucket_prefix', 'dry_run']] + topsapp_params_obj = topsappParams(**cli_params) + + # serialize input + json.dump(topsapp_params_obj.model_dump(), + open('topsapp_input_params.json', 'w'), + indent=2) + # Region of interest becomes 'extent' in loc_data loc_data = localize_data( args.reference_scenes, args.secondary_scenes, dry_run=args.dry_run, + geocode_resolution=args.output_resolution, frame_id=args.frame_id, water_mask_flag=args.estimate_ionosphere_delay, ) - loc_data["frame_id"] = args.frame_id + loc_data['frame_id'] = args.frame_id + loc_data['cmd_line_str'] = cmd_line_str + loc_data['tops_app_params'] = topsapp_params_obj.model_dump() # Allows for easier re-inspection of processing, packaging, and delivery # after job completes json.dump(loc_data, open("loc_data.json", "w"), indent=2, cls=MetadataEncoder) - # Turn-off ESD when using ionospheric computation - # NOTE: note sure if this needs to be off - # esd is calculated with iono only when - # considerBurstProperties is on which off - # by default - if args.estimate_ionosphere_delay: - args.esd_coherence_threshold = -1 - topsapp_processing( reference_slc_zips=loc_data["ref_paths"], secondary_slc_zips=loc_data["sec_paths"], @@ -233,6 +261,9 @@ def gunw_slc(): dem_for_proc=loc_data["full_res_dem_path"], dem_for_geoc=loc_data["low_res_dem_path"], dry_run=args.dry_run, + do_dense_offsets=args.dense_offsets, + goldstein_filter_power=args.goldstein_filter_power, + output_resolution=args.output_resolution ) # Run ionospheric correction @@ -242,26 +273,43 @@ def gunw_slc(): correct_burst_ramps=True, ) + additional_2d_layers_for_packaging = [] + additional_attributes_for_packaging = {} + if args.estimate_ionosphere_delay: + additional_2d_layers_for_packaging.append('ionosphere') + additional_2d_layers_for_packaging.append('ionosphereBurstRamps') + # Keys need to be the same as layer names; + # specifically ionosphere and ionosphereBurstRamps are keys + additional_attributes_for_packaging.update(**iono_attr) + if args.dense_offsets: + additional_2d_layers_for_packaging.append('rangePixelOffsets') + additional_2d_layers_for_packaging.append('azimuthPixelOffsets') + if args.unfiltered_coherence: + additional_2d_layers_for_packaging.append('unfilteredCoherence') + + # Serialize additional layer data to replicate packaging + with open('additional_2d_layers.txt', 'w') as file: + file.write('\n'.join(additional_2d_layers_for_packaging)) + json.dump(additional_attributes_for_packaging, + open("additional_attributes_for_packaging.json", "w"), + indent=2) + ref_properties = loc_data["reference_properties"] sec_properties = loc_data["secondary_properties"] extent = loc_data["extent"] + product_geometry_wkt = loc_data['gunw_geo'].wkt - additional_2d_layers = [] - additional_attributes = [] - if args.estimate_ionosphere_delay: - additional_2d_layers.append("ionosphere") - additional_attributes.append(iono_attr['ionosphere']) - additional_2d_layers.append("ionosphereBurstRamps") - additional_attributes.append(iono_attr['ionosphereBurstRamps']) - - additional_2d_layers = additional_2d_layers or None nc_path = package_gunw_product( isce_data_directory=Path.cwd(), reference_properties=ref_properties, secondary_properties=sec_properties, extent=extent, - additional_2d_layers=additional_2d_layers, - additional_attributes=additional_attributes + additional_2d_layers=additional_2d_layers_for_packaging, + additional_attributes=additional_attributes_for_packaging, + standard_product=topsapp_params_obj.is_standard_gunw_product(), + cmd_line_str=cmd_line_str, + product_geometry_wkt=product_geometry_wkt, + topaspp_params=topsapp_params_obj.model_dump() ) if args.compute_solid_earth_tide: diff --git a/isce2_topsapp/localize_dem.py b/isce2_topsapp/localize_dem.py index 302be5db..617b6b25 100644 --- a/isce2_topsapp/localize_dem.py +++ b/isce2_topsapp/localize_dem.py @@ -40,6 +40,7 @@ def fix_image_xml(isce_raster_path: str) -> str: def download_dem_for_isce2(extent: list, dem_name: str = 'glo_30', + geocode_resolution: int = 90, full_res_dem_dir: Path = None, low_res_dem_dir: Path = None, buffer: float = .4) -> dict: @@ -58,6 +59,9 @@ def download_dem_for_isce2(extent: list, ------- dict """ + if geocode_resolution not in [30, 90]: + raise ValueError('Geocode resolution must be "30" or "90"') + full_res_dem_dir = full_res_dem_dir or Path('.') low_res_dem_dir = low_res_dem_dir or Path('.') @@ -91,14 +95,23 @@ def download_dem_for_isce2(extent: list, with rasterio.open(full_res_dem_path, 'w', **dem_profile_isce) as ds: ds.write(dem_array, 1) - geocode_res = dem_res * 3 - dst_profile = update_profile_resolution(dem_profile_isce, geocode_res) - dem_geocode_arr, dem_geocode_profile = reproject_arr_to_match_profile(dem_array, - dem_profile_isce, - dst_profile, - num_threads=5, - resampling='bilinear') - dem_geocode_arr = dem_geocode_arr[0, ...] + # Standard GUNW + if geocode_resolution == 90: + geocode_res = dem_res * 3 + + dst_profile = update_profile_resolution(dem_profile_isce, geocode_res) + dem_geocode_arr, dem_geocode_profile = reproject_arr_to_match_profile(dem_array, + dem_profile_isce, + dst_profile, + num_threads=5, + resampling='bilinear') + dem_geocode_arr = dem_geocode_arr[0, ...] + + else: + # if 30 meters the array and geometadata should be the same + dem_geocode_arr = dem_array + dem_geocode_profile = dem_profile.copy() + low_res_dem_path = low_res_dem_dir / 'low_res.dem.wgs84' dem_geocode_profile['driver'] = 'ISCE' diff --git a/isce2_topsapp/packaging.py b/isce2_topsapp/packaging.py index 4f6ccbc9..a4045d71 100644 --- a/isce2_topsapp/packaging.py +++ b/isce2_topsapp/packaging.py @@ -7,17 +7,19 @@ from typing import Union import h5py +import numpy as np +import rasterio from dateparser import parse import isce2_topsapp from isce2_topsapp.packaging_utils.additional_layers import add_2d_layer -from isce2_topsapp.packaging_utils.ionosphere import format_iono_burst_ramps, format_ionosphere_for_gunw +from isce2_topsapp.packaging_utils.ionosphere import ( + format_iono_burst_ramps, format_ionosphere_for_gunw) from isce2_topsapp.templates import read_netcdf_packaging_template DATASET_VERSION = '3.0.0' - - -PERMISSIBLE_2D_LAYERS = ['ionosphere', 'ionosphereBurstRamps'] +STANDARD_PROD_PREFIX = 'S1-GUNW' +CUSTOM_PROD_PREFIX = 'S1-GUNW_CUSTOM' """Warning: the packaging scripts were written as command line scripts and @@ -79,6 +81,7 @@ def get_center_time(properties: list) -> datetime.datetime: def get_gunw_id(reference_properties: list, secondary_properties: list, extent: list, + standard_product: bool = True, ) -> str: # asc_or_desc: will be "A" or "D" @@ -116,7 +119,8 @@ def get_gunw_id(reference_properties: list, version = DATASET_VERSION.replace('.', '_') version = f'v{version}' - ids = ['S1-GUNW', + gunw_prefix = STANDARD_PROD_PREFIX if standard_product else CUSTOM_PROD_PREFIX + ids = [gunw_prefix, asc_or_desc, # right looking 'R', @@ -218,7 +222,7 @@ def perform_netcdf_packaging(*, subprocess.check_call(cmd, shell=True) os.chdir(cwd) - out_nc_file = merged_dir/f'{gunw_id}.nc' + out_nc_file = merged_dir / f'{gunw_id}.nc' # Check if the netcdf file was created assert out_nc_file.exists() @@ -228,14 +232,16 @@ def perform_netcdf_packaging(*, def package_additional_layers_into_gunw(gunw_path: Path, isce_data_directory: Path, additional_2d_layers: list, - additional_attributes: list): + additional_attributes: dict): # Current workflow of additional layers # 1. Do any additional processing/formatting outside of GUNW # 2. Add layer into GUNW # 3. Update Version - if not set(additional_2d_layers).issubset(set(PERMISSIBLE_2D_LAYERS)): - raise RuntimeError('Additional 2d layers must be subset of ' - f'{PERMISSIBLE_2D_LAYERS}') + + # in case additional attributes is None + additional_attributes = additional_attributes or {} + if not set(additional_attributes.keys()).issubset(additional_2d_layers): + raise ValueError('Additional attributes dict must be within additional_2d_layers') if 'ionosphere' in additional_2d_layers: # current working directory is ISCE directory @@ -245,8 +251,13 @@ def package_additional_layers_into_gunw(gunw_path: Path, _ = format_iono_burst_ramps(isce_data_directory, gunw_path) # Assumes ionosphere raster is written to specific path - additional_dataset = zip(additional_2d_layers, additional_attributes) - [add_2d_layer(layer, gunw_path, attributes) for layer, attributes in additional_dataset] + additional_attributes_lst = [additional_attributes.get(layer_name, None) + for layer_name in additional_2d_layers] + zipped_data = zip(additional_2d_layers, additional_attributes_lst) + [add_2d_layer(layer, + gunw_path, + additional_attrs=add_attrs) + for (layer, add_attrs) in zipped_data] # Update with h5py.File(gunw_path, mode='a') as file: @@ -254,13 +265,71 @@ def package_additional_layers_into_gunw(gunw_path: Path, return gunw_path +def get_layer_mean(netcdf_path: Path, + groups: list, + layers: list) -> dict: + if len(groups) != len(layers): + raise ValueError('groups and layers must have same length') + data = {} + for group, layer in zip(groups, layers): + with rasterio.open(f'netcdf:{netcdf_path}:{group}/{layer}') as ds: + X = ds.read() + if np.isnan(ds.nodata): + mask = np.isnan(X) + else: + mask = ds.nodata == X + raster_data = X[~mask] + data[f'mean_{layer}'] = np.mean(raster_data) + return data + + +def record_stats(*, + netcdf_path: Path) -> Path: + groups = ['science/grids/imagingGeometry'] * 2 + ['science/grids/data'] + layers = ['perpendicularBaseline', 'parallelBaseline', 'coherence'] + mean_attrs = get_layer_mean(netcdf_path, + groups, + layers) + with h5py.File(netcdf_path, mode='a') as file: + file.attrs.update(**mean_attrs) + return netcdf_path + + +def record_params(*, + netcdf_path: Path, + cmd_line_str: str, + topsapp_params: dict) -> Path: + + with h5py.File(netcdf_path, mode='a') as file: + file.attrs.update(aria_frame_id=topsapp_params['frame_id']) + file.attrs.update(topsapp_command_line_string=cmd_line_str) + file.attrs.update(isce2_topsapp_version=f'{isce2_topsapp.__version__}') + file['science/grids'].attrs.update(**topsapp_params) + return netcdf_path + + +def record_wkt_geometry(*, + netcdf_path: Path, + product_geometry_wkt: str + ) -> Path: + + with h5py.File(netcdf_path, mode='a') as file: + file.attrs.update(product_geometry_wkt=product_geometry_wkt) + return netcdf_path + + def package_gunw_product(*, isce_data_directory: Union[str, Path], reference_properties: list, secondary_properties: list, extent: list, + topaspp_params: dict, + cmd_line_str: str, + product_geometry_wkt: str, additional_2d_layers: list = None, - additional_attributes: list = None) -> Path: + standard_product: bool = True, + additional_attributes: dict = None, + ) -> Path: """Creates a GUNW standard product netcdf from the ISCE outputs and some initial metadata. @@ -276,9 +345,13 @@ def package_gunw_product(*, List of extents ([xmin, ymin, xmax, ymax]) additional_2d_layers: list List of 2d layers to add. Currently, supported is ionosphere. - additional_attributes: list + additional_attributes: dict List of attributs dicts for additional layers o add. Currently, supported only for ionosphere. + standard_product: bool + Whether the package is a GUNW standard product or not. Will use the + the prefix `S1-GUNW` for standard products and `S1-GUNW_CUSTOM` + otherwise Returns ------- @@ -289,14 +362,22 @@ def package_gunw_product(*, gunw_id = get_gunw_id(reference_properties=reference_properties, secondary_properties=secondary_properties, - extent=extent) + extent=extent, + standard_product=standard_product) out_nc_file = perform_netcdf_packaging(isce_data_dir=isce_data_directory, gunw_id=gunw_id) if additional_2d_layers is not None: + isce_data_directory = Path(isce_data_directory) package_additional_layers_into_gunw(out_nc_file, isce_data_directory, additional_2d_layers, additional_attributes) + out_nc_file = record_params(netcdf_path=out_nc_file, + topsapp_params=topaspp_params, + cmd_line_str=cmd_line_str) + out_nc_file = record_stats(netcdf_path=out_nc_file) + out_nc_file = record_wkt_geometry(netcdf_path=out_nc_file, + product_geometry_wkt=product_geometry_wkt) return out_nc_file diff --git a/isce2_topsapp/packaging_utils/additional_layers.json b/isce2_topsapp/packaging_utils/additional_layers.json index 095d30cd..b7b4f4c2 100644 --- a/isce2_topsapp/packaging_utils/additional_layers.json +++ b/isce2_topsapp/packaging_utils/additional_layers.json @@ -1,20 +1,61 @@ -{"ionosphere": {"dst_group": "/science/grids/corrections/derived/ionosphere", +{ + "ionosphere": { + "dst_group": "/science/grids/corrections/derived/ionosphere", "dst_variable": "ionosphere", - "input_relative_path": "merged/ionosphere_for_gunw.geo", - "attrs": { - "standard_name": "ionospherePhaseCorrection", - "long_name": "ionospherePhaseCorrection", - "description": "Estimated ionosphere phase correction" - } - }, - "ionosphereBurstRamps": {"dst_group": "/science/grids/corrections/derived/ionosphereBurstRamps", - "dst_variable": "ionosphereBurstRamps", - "input_relative_path": "merged/ionosphereBurstRamps_for_gunw.geo", - "attrs": { - "standard_name": "ionosphereBurstRamps", - "long_name": "ionosphereBurstRamps", - "description": "Estimated burst ramps correction due to ionosphere" - } - } - + "src_band_number": 1, + "input_relative_path": "merged/ionosphere_for_gunw.geo", + "attrs": { + "standard_name": "ionospherePhaseCorrection", + "long_name": "ionospherePhaseCorrection", + "units": "rad", + "description": "Estimated ionosphere phase correction" + } + }, + "ionosphereBurstRamps": { + "dst_group": "/science/grids/corrections/derived/ionosphereBurstRamps", + "dst_variable": "ionosphereBurstRamps", + "src_band_number": 1, + "input_relative_path": "merged/ionosphereBurstRamps_for_gunw.geo", + "attrs": { + "standard_name": "ionosphereBurstRamps", + "long_name": "ionosphereBurstRamps", + "description": "Estimated burst ramps correction due to ionosphere" + } + }, + "unfilteredCoherence": { + "dst_group": "/science/grids/data", + "dst_variable": "unfilteredCoherence", + "src_band_number": 2, + "nodata": 0, + "input_relative_path": "merged/topophase.cor.geo", + "attrs": { + "standard_name": "unfilteredCoherence", + "long_name": "unfilteredCoherence", + "description": "Coherence without filters applied" + } + }, + "rangePixelOffsets": { + "dst_group": "/science/grids/data", + "dst_variable": "rangePixelOffsets", + "input_relative_path": "merged/dense_offsets.bil.geo", + "src_band_number": 2, + "attrs": { + "standard_name": "rangePixelOffsets", + "long_name": "rangePixelOffsets", + "units": "range radar pixel", + "description": "Range pixel offsets measured through patch correlation. To translate into meters, multiply by 2.3 as indicated here: https://sentinels.copernicus.eu/web/sentinel/technical-guides/sentinel-1-sar/products-algorithms/level-1/single-look-complex/interferometric-wide-swath" + } + }, + "azimuthPixelOffsets": { + "dst_group": "/science/grids/data", + "dst_variable": "azimuthPixelOffsets", + "input_relative_path": "merged/dense_offsets.bil.geo", + "src_band_number": 1, + "attrs": { + "standard_name": "azimuthPixelOffsets", + "long_name": "azimuthPixelOffsets", + "units": "azimuth radar pixel", + "description": "Azimuth pixel offsets measured through patch correlation. To translate into meters, multiply by 14.1 as indicated here: https://sentinels.copernicus.eu/web/sentinel/technical-guides/sentinel-1-sar/products-algorithms/level-1/single-look-complex/interferometric-wide-swath" + } + } } \ No newline at end of file diff --git a/isce2_topsapp/packaging_utils/additional_layers.py b/isce2_topsapp/packaging_utils/additional_layers.py index d6359dc0..ca5de99f 100644 --- a/isce2_topsapp/packaging_utils/additional_layers.py +++ b/isce2_topsapp/packaging_utils/additional_layers.py @@ -5,51 +5,78 @@ import xarray as xr LAYER_JSON = Path(__file__).parents[0] / 'additional_layers.json' -ADDITIONAL_LAYERS = json.load(open(LAYER_JSON)) +ADDITIONAL_LAYERS_DATA = json.load(open(LAYER_JSON)) -def add_2d_layer(layer_name: str, gunw_netcdf_path: Path, additional_attrs: dict = None) -> Path: +def add_2d_layer(layer_name: str, + gunw_netcdf_path: Path, + additional_attrs: dict = None) -> Path: """ Combines a lot of standard formatting of the netcdf via rioxarray and - deletes the previous placeholder (we assume it exists via the placeholder). + deletes the previous placeholder if there is one. - We also assume any additional processing specific to GUNW is done outside of + We assume any additional processing specific to GUNW is done outside of this function. """ - layer_data = ADDITIONAL_LAYERS[layer_name] + layer_data = ADDITIONAL_LAYERS_DATA[layer_name] dst_group = layer_data['dst_group'] dst_variable = layer_data['dst_variable'] + band_number = layer_data['src_band_number'] + nodata_val = layer_data.get('nodata') + possible_layers = list(ADDITIONAL_LAYERS_DATA.keys()) + + if layer_name not in possible_layers: + ValueError('layer_name must be in {", ".join(possible_layers)}') + + if not isinstance(band_number, int) or (band_number < 1): + ValueError('Layers must select individual layers from outputs i.e ' + '1, 2, ...') + if additional_attrs: layer_data['attrs'].update(additional_attrs) - # The layers generally already exist within the file + # If the layers already exist within the file, we need to delete them otherwise the dummy placeholder + # Causes type errors when attempting to overwrite with h5py.File(gunw_netcdf_path, mode='a') as file: if dst_group in file: # Delete the variable to be written to if dst_variable in file[dst_group]: del file[dst_group][dst_variable] - # Delete the group if there are no variables left + # Delete the group if there are no variables left to ensure correct type of arrays. + # If there are variables left, this routine appends the new array to the group (assumes same dims + # as existing arrays) if len(file[dst_group].keys()) == 0: del file[dst_group] ds = xr.open_dataset(layer_data['input_relative_path'], engine='rasterio') - # Renaming ensures correct geo-referencing with spatial_ref grid mapping - ds = ds.rename({'x': 'longitude', + # Ensures of grid_mapping_name as 'crs' + # This allows us to append to /science/grids/data to have consistent CRS + ds = ds.drop_vars('spatial_ref') + ds.rio.write_crs(4326, inplace=True, grid_mapping_name='crs') + + # Renaming ensures correct geo-referencing with grid mapping + ds = ds.rename({ + # x, y are the coordinate names + 'x': 'longitude', 'y': 'latitude', 'band_data': dst_variable}) + # removes channel (what rioxarray band) dimension in the coordinates + ds = ds.sel(band=band_number, drop=True) + ds['latitude'].attrs.update({'long_name': 'latitude', 'standard_name': 'latitude'}) ds['longitude'].attrs.update({'long_name': 'longitude', 'standard_name': 'longitude'}) - # removes channel (aka band) dimension - ds = ds.squeeze(['band'], drop=True) ds[layer_name].attrs.update(layer_data['attrs']) + if nodata_val is not None: + ds[layer_name] = ds[layer_name].rio.write_nodata(nodata_val) + ds.to_netcdf(gunw_netcdf_path, group=layer_data['dst_group'], mode='a') diff --git a/isce2_topsapp/packaging_utils/ionosphere.py b/isce2_topsapp/packaging_utils/ionosphere.py index f7cdb55f..126a10d6 100644 --- a/isce2_topsapp/packaging_utils/ionosphere.py +++ b/isce2_topsapp/packaging_utils/ionosphere.py @@ -8,6 +8,19 @@ def format_ionosphere_for_gunw(isce_directory: Path, gunw_netcdf_path: Path) -> Path: + """Resamples the ionosphere to approximately 1 km using origin of product + + Parameters + ---------- + isce_directory : Path + The so-called "scratch" directory + gunw_netcdf_path : Path + + Returns + ------- + Path + gunw_path + """ with rasterio.open(isce_directory / "merged/topophase.ion.geo") as ds: X_ion = ds.read(1) p_ion = ds.profile diff --git a/isce2_topsapp/templates/nc_packaging_template.json b/isce2_topsapp/templates/nc_packaging_template.json index c00538c3..961c56c8 100755 --- a/isce2_topsapp/templates/nc_packaging_template.json +++ b/isce2_topsapp/templates/nc_packaging_template.json @@ -14,7 +14,7 @@ "value" : "1b" },{ "name" : "author", - "value" : "David Bekaert, Charlie Marshak, Simran Sangha, Joe Kennedy, Andrew Johnston" + "value" : "David Bekaert, Grace Bato, Marin Govorcin, Andrew Johnston, Joe Kennedy, Charlie Marshak, Simran Sangha and ARIA-JPL" },{ "name" : "institution", "value" : "Jet Propulsion Laboratory" diff --git a/isce2_topsapp/templates/topsapp_iono_template.xml b/isce2_topsapp/templates/topsapp_iono_template.xml deleted file mode 100644 index a79c562a..00000000 --- a/isce2_topsapp/templates/topsapp_iono_template.xml +++ /dev/null @@ -1,37 +0,0 @@ - - - - - {{ orbit_directory }} - aux_cal - {{ output_reference_directory }} - {{ ref_zip_file }} - - - {{ orbit_directory }} - aux_cal - {{ output_secondary_directory }} - {{ sec_zip_file }} - - {{ swaths }} - {{ range_looks }} - {{ azimuth_looks }} - {{ filter_strength }} - {{ region_of_interest }} - {{ demFilename }} - {{ geocodeDemFilename }} - True - snaphu_mcf - {{ estimate_ionosphere_delay }} - {{ ion_burstProperties }} - {{ ion_startStep }} - {{ ion_stopStep }} - {{ ion_polyFit }} - {{ ion_correctAzimuthshift }} - False - {{ do_esd }} - {{ esd_coherence_threshold }} - {{ use_virtual_files }} - {{ geocode_list }} - - \ No newline at end of file diff --git a/isce2_topsapp/templates/topsapp_template.xml b/isce2_topsapp/templates/topsapp_template.xml index eb04f68b..4b1330d1 100755 --- a/isce2_topsapp/templates/topsapp_template.xml +++ b/isce2_topsapp/templates/topsapp_template.xml @@ -16,7 +16,6 @@ {{ swaths }} {{ range_looks }} {{ azimuth_looks }} - {{ filter_strength }} {{ region_of_interest }} {{ demFilename }} {{ geocodeDemFilename }} @@ -26,7 +25,11 @@ False {{ do_esd }} {{ esd_coherence_threshold }} + {{ do_dense_offsets }} + {{ ampcor_width }} + {{ ampcor_height }} {{ use_virtual_files }} + {{ goldstein_filter_power }} {{ geocode_list }} diff --git a/isce2_topsapp/topsapp_params.py b/isce2_topsapp/topsapp_params.py new file mode 100644 index 00000000..d5d8b475 --- /dev/null +++ b/isce2_topsapp/topsapp_params.py @@ -0,0 +1,30 @@ +from typing import List + +from pydantic import BaseModel + + +class topsappParams(BaseModel): + reference_scenes: List[str] + secondary_scenes: List[str] + frame_id: int + estimate_ionosphere_delay: bool = True + compute_solid_earth_tide: bool = True + output_resolution: int = 90 + unfiltered_coherence: bool = True + goldstein_filter_power: float = .5 + dense_offsets: bool = False + esd_coherence_threshold: float = -1 + + def is_standard_gunw_product(self) -> bool: + """Version 3+""" + checks = [ + self.estimate_ionosphere_delay, + self.frame_id != -1, + self.compute_solid_earth_tide, + not self.dense_offsets, + self.unfiltered_coherence, + self.esd_coherence_threshold == -1, + self.goldstein_filter_power == .5, + self.output_resolution == 90, + ] + return all(checks) diff --git a/isce2_topsapp/topsapp_proc.py b/isce2_topsapp/topsapp_proc.py index 974e000b..d378f88b 100644 --- a/isce2_topsapp/topsapp_proc.py +++ b/isce2_topsapp/topsapp_proc.py @@ -42,17 +42,28 @@ def topsapp_processing(*, extent: list, dem_for_proc: str, dem_for_geoc: str, - estimate_ionosphere_delay: bool, - azimuth_looks: int = 7, - range_looks: int = 19, + estimate_ionosphere_delay: bool = False, swaths: list = None, dry_run: bool = False, do_esd: bool = False, - esd_coherence_threshold: float = .7): + esd_coherence_threshold: float = .7, + output_resolution: int = 90, + do_dense_offsets: bool = False, + goldstein_filter_power: float = .5, + ampcor_window_size: int = 64): swaths = swaths or [1, 2, 3] # for [ymin, ymax, xmin, xmax] extent_isce = [extent[k] for k in [1, 3, 0, 2]] + if output_resolution == 90: + azimuth_looks = 7 + range_looks = 19 + elif output_resolution == 30: + azimuth_looks = 3 + range_looks = 7 + else: + raise ValueError('Output resolution must be "30" or "90"') + # Update PATH with ISCE2 applications isce_application_path = Path(f'{site.getsitepackages()[0]}' '/isce/applications/') @@ -65,6 +76,9 @@ def topsapp_processing(*, if estimate_ionosphere_delay: geocode_list.append('merged/topophase.ion') + if do_dense_offsets: + geocode_list.append('merged/dense_offsets.bil') + topsApp_xml = template.render(orbit_directory=orbit_directory, output_reference_directory='reference', output_secondary_directory='secondary', @@ -73,7 +87,6 @@ def topsapp_processing(*, region_of_interest=extent_isce, demFilename=dem_for_proc, geocodeDemFilename=dem_for_geoc, - filter_strength=.5, do_unwrap=True, use_virtual_files=True, do_esd=do_esd, @@ -82,7 +95,11 @@ def topsapp_processing(*, azimuth_looks=azimuth_looks, range_looks=range_looks, swaths=swaths, - geocode_list=geocode_list + geocode_list=geocode_list, + do_dense_offsets=do_dense_offsets, + goldstein_filter_power=goldstein_filter_power, + ampcor_height=ampcor_window_size, + ampcor_width=ampcor_window_size ) with open('topsApp.xml', "w") as file: file.write(topsApp_xml) diff --git a/sample_run.sh b/sample_run.sh index c7914183..257e434f 100644 --- a/sample_run.sh +++ b/sample_run.sh @@ -2,8 +2,9 @@ isce2_topsapp --username \ --password \ --esa-username \ --esa-password \ - --reference-scenes S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9 \ - --secondary-scenes S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D \ - S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404 \ - S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C \ + --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E \ + S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 \ + --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F \ + S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 \ + --frame-id 25502 \ > topsapp_img.out 2> topsapp_img.err diff --git a/setup.py b/setup.py index e0e1a316..7c417118 100644 --- a/setup.py +++ b/setup.py @@ -12,8 +12,10 @@ url='https://github.com/dbekaert/DockerizedTopsApp', - author='Charlie Marshak, David Bekaert, Grace Bato, Simran Sangha Joseph H. Kennedy Brett Buzzunga, and others', - author_email='charlie.z.marshak@jpl.nasa.gov', + author=('David Bekaert, Grace Bato, ' + 'Marin Govorcin, Andrew Johnston, Joe Kennedy, Charlie Marshak, ' + 'Simran Sangha and ARIA-JPL'), + author_email='access_cloud_based_insar@jpl.nasa.gov', license='Apache-2.0', classifiers=[ diff --git a/tests/localize-data.ipynb b/tests/localize-data.ipynb index dae8da48..125764a5 100644 --- a/tests/localize-data.ipynb +++ b/tests/localize-data.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "sapphire-destruction", "metadata": { "ExecuteTime": { @@ -18,7 +18,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "civilian-arabic", "metadata": { "ExecuteTime": { @@ -26,9 +26,18 @@ "start_time": "2022-01-19T02:29:31.117277Z" } }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2023-11-16 18:00:22,356 - matplotlib.font_manager - DEBUG - Using fontManager instance from /Users/cmarshak/.matplotlib/fontlist-v330.json\n" + ] + } + ], "source": [ - "from isce2_topsapp import download_slcs, download_orbits, download_dem_for_isce2, download_aux_cal" + "from isce2_topsapp import download_slcs, download_orbits, download_dem_for_isce2, download_aux_cal\n", + "from isce2_topsapp.__main__ import check_esa_credentials" ] }, { @@ -41,7 +50,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "id": "failing-building", "metadata": { "ExecuteTime": { @@ -63,7 +72,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "id": "amended-aircraft", "metadata": { "ExecuteTime": { @@ -95,7 +104,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "id": "third-trinity", "metadata": { "ExecuteTime": { @@ -103,7 +112,200 @@ "start_time": "2022-01-19T02:29:32.271241Z" } }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2023-11-16 17:35:54,152 - fiona.env - DEBUG - PROJ data files are available at built-in paths.\n", + "2023-11-16 17:35:54,172 - fiona.env - DEBUG - Entering env context: \n", + "2023-11-16 17:35:54,173 - fiona.env - DEBUG - Starting outermost env\n", + "2023-11-16 17:35:54,173 - fiona.env - DEBUG - No GDAL environment exists\n", + "2023-11-16 17:35:54,174 - fiona.env - DEBUG - New GDAL environment created\n", + "2023-11-16 17:35:54,198 - fiona._env - DEBUG - GDAL_DATA found in environment.\n", + "2023-11-16 17:35:54,199 - fiona._env - DEBUG - PROJ data files are available at built-in paths.\n", + "2023-11-16 17:35:54,199 - fiona._env - DEBUG - Started GDALEnv: self=.\n", + "2023-11-16 17:35:54,200 - fiona.env - DEBUG - Updated existing with options {}\n", + "2023-11-16 17:35:54,200 - fiona.env - DEBUG - Entered env context: \n", + "2023-11-16 17:35:54,203 - fiona.env - DEBUG - Exiting env context: \n", + "2023-11-16 17:35:54,203 - fiona.env - DEBUG - Cleared existing options\n", + "2023-11-16 17:35:54,204 - fiona._env - DEBUG - Stopping GDALEnv .\n", + "2023-11-16 17:35:54,204 - fiona._env - DEBUG - Error handler popped.\n", + "2023-11-16 17:35:54,205 - fiona._env - DEBUG - Stopped GDALEnv .\n", + "2023-11-16 17:35:54,205 - fiona.env - DEBUG - Exiting outermost env\n", + "2023-11-16 17:35:54,205 - fiona.env - DEBUG - Exited env context: \n", + "2023-11-16 17:35:54,228 - fiona.env - DEBUG - Entering env context: \n", + "2023-11-16 17:35:54,228 - fiona.env - DEBUG - Starting outermost env\n", + "2023-11-16 17:35:54,229 - fiona.env - DEBUG - No GDAL environment exists\n", + "2023-11-16 17:35:54,230 - fiona.env - DEBUG - New GDAL environment created\n", + "2023-11-16 17:35:54,230 - fiona._env - DEBUG - GDAL_DATA found in environment.\n", + "2023-11-16 17:35:54,231 - fiona._env - DEBUG - PROJ data files are available at built-in paths.\n", + "2023-11-16 17:35:54,231 - fiona._env - DEBUG - Started GDALEnv: self=.\n", + "2023-11-16 17:35:54,231 - fiona.env - DEBUG - Updated existing with options {}\n", + "2023-11-16 17:35:54,232 - fiona.env - DEBUG - Entered env context: \n", + "2023-11-16 17:35:54,242 - fiona.ogrext - DEBUG - Got coordinate system\n", + "2023-11-16 17:35:54,243 - fiona.ogrext - DEBUG - Got coordinate system\n", + "2023-11-16 17:35:54,244 - fiona.ogrext - DEBUG - OLC_FASTSETNEXTBYINDEX: 1\n", + "2023-11-16 17:35:54,245 - fiona.ogrext - DEBUG - OLC_FASTFEATURECOUNT: 1\n", + "2023-11-16 17:35:54,245 - fiona.ogrext - DEBUG - Next index: 0\n", + "2023-11-16 17:35:54,246 - fiona.ogrext - DEBUG - Next index: 1\n", + "2023-11-16 17:35:54,515 - fiona.ogrext - DEBUG - Next index: 2\n", + "2023-11-16 17:35:54,536 - fiona.ogrext - DEBUG - Next index: 3\n", + "2023-11-16 17:35:54,567 - fiona.ogrext - DEBUG - Next index: 4\n", + "2023-11-16 17:35:54,605 - fiona.ogrext - DEBUG - Next index: 5\n", + "2023-11-16 17:35:54,618 - fiona.ogrext - DEBUG - Next index: 6\n", + "2023-11-16 17:35:54,639 - fiona.ogrext - DEBUG - Next index: 7\n", + "2023-11-16 17:35:54,653 - fiona.ogrext - DEBUG - Next index: 8\n", + "2023-11-16 17:35:54,739 - fiona.ogrext - DEBUG - Next index: 9\n", + "2023-11-16 17:35:54,744 - fiona.ogrext - DEBUG - Next index: 10\n", + "2023-11-16 17:35:54,753 - fiona.ogrext - DEBUG - Next index: 11\n", + "2023-11-16 17:35:54,761 - fiona.collection - DEBUG - Flushed buffer\n", + "2023-11-16 17:35:54,762 - fiona.collection - DEBUG - Stopped session\n", + "2023-11-16 17:35:54,762 - fiona.env - DEBUG - Exiting env context: \n", + "2023-11-16 17:35:54,762 - fiona.env - DEBUG - Cleared existing options\n", + "2023-11-16 17:35:54,763 - fiona._env - DEBUG - Stopping GDALEnv .\n", + "2023-11-16 17:35:54,763 - fiona._env - DEBUG - Error handler popped.\n", + "2023-11-16 17:35:54,763 - fiona._env - DEBUG - Stopped GDALEnv .\n", + "2023-11-16 17:35:54,763 - fiona.env - DEBUG - Exiting outermost env\n", + "2023-11-16 17:35:54,764 - fiona.env - DEBUG - Exited env context: \n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Downloading SLCs: 100%|████████████████████████████████████████████████████| 4/4 [00:05<00:00, 1.29s/it]\n" + ] + }, + { + "data": { + "text/plain": [ + "{'ref_paths': ['S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9.zip'],\n", + " 'sec_paths': ['S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C.zip',\n", + " 'S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404.zip',\n", + " 'S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D.zip'],\n", + " 'extent': [-119.080994, 33.405682, -115.988029, 35.43090952856198],\n", + " 'gunw_geo': ,\n", + " 'processing_extent': [-119.080994, 33.405682, -115.988029, 35.43090952856198],\n", + " 'reference_properties': [{'beamModeType': 'IW',\n", + " 'browse': None,\n", + " 'bytes': 4334545363,\n", + " 'centerLat': 34.4266,\n", + " 'centerLon': -117.5237,\n", + " 'faradayRotation': None,\n", + " 'fileID': 'S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9-SLC',\n", + " 'flightDirection': 'ASCENDING',\n", + " 'groupID': 'S1B_IWDV_0107_0114_027915_064',\n", + " 'granuleType': 'SENTINEL_1B_FRAME',\n", + " 'insarStackId': None,\n", + " 'md5sum': 'fac5678b4b27b4321cde8953a95d0416',\n", + " 'offNadirAngle': None,\n", + " 'orbit': 27915,\n", + " 'pathNumber': 64,\n", + " 'platform': 'Sentinel-1B',\n", + " 'pointingAngle': None,\n", + " 'polarization': 'VV+VH',\n", + " 'processingDate': '2021-07-23T01:49:47.000Z',\n", + " 'processingLevel': 'SLC',\n", + " 'sceneName': 'S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9',\n", + " 'sensor': 'C-SAR',\n", + " 'startTime': '2021-07-23T01:49:47.000Z',\n", + " 'stopTime': '2021-07-23T01:50:14.000Z',\n", + " 'url': 'https://datapool.asf.alaska.edu/SLC/SB/S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9.zip',\n", + " 'pgeVersion': '003.31',\n", + " 'fileName': 'S1B_IW_SLC__1SDV_20210723T014947_20210723T015014_027915_0354B4_B3A9.zip',\n", + " 'frameNumber': 108}],\n", + " 'secondary_properties': [{'beamModeType': 'IW',\n", + " 'browse': None,\n", + " 'bytes': 4435118066,\n", + " 'centerLat': 35.9201,\n", + " 'centerLon': -117.85,\n", + " 'faradayRotation': None,\n", + " 'fileID': 'S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C-SLC',\n", + " 'flightDirection': 'ASCENDING',\n", + " 'groupID': 'S1B_IWDV_0112_0119_027740_064',\n", + " 'granuleType': 'SENTINEL_1B_FRAME',\n", + " 'insarStackId': None,\n", + " 'md5sum': '53c785599671f23bebc9493e7c5113c4',\n", + " 'offNadirAngle': None,\n", + " 'orbit': 27740,\n", + " 'pathNumber': 64,\n", + " 'platform': 'Sentinel-1B',\n", + " 'pointingAngle': None,\n", + " 'polarization': 'VV+VH',\n", + " 'processingDate': '2021-07-11T01:50:11.000Z',\n", + " 'processingLevel': 'SLC',\n", + " 'sceneName': 'S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C',\n", + " 'sensor': 'C-SAR',\n", + " 'startTime': '2021-07-11T01:50:11.000Z',\n", + " 'stopTime': '2021-07-11T01:50:38.000Z',\n", + " 'url': 'https://datapool.asf.alaska.edu/SLC/SB/S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C.zip',\n", + " 'pgeVersion': '003.31',\n", + " 'fileName': 'S1B_IW_SLC__1SDV_20210711T015011_20210711T015038_027740_034F80_376C.zip',\n", + " 'frameNumber': 113},\n", + " {'beamModeType': 'IW',\n", + " 'browse': None,\n", + " 'bytes': 4345308811,\n", + " 'centerLat': 34.4262,\n", + " 'centerLon': -117.5234,\n", + " 'faradayRotation': None,\n", + " 'fileID': 'S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404-SLC',\n", + " 'flightDirection': 'ASCENDING',\n", + " 'groupID': 'S1B_IWDV_0107_0114_027740_064',\n", + " 'granuleType': 'SENTINEL_1B_FRAME',\n", + " 'insarStackId': None,\n", + " 'md5sum': '56b2004a93ad86b82e128e5ac9f04a94',\n", + " 'offNadirAngle': None,\n", + " 'orbit': 27740,\n", + " 'pathNumber': 64,\n", + " 'platform': 'Sentinel-1B',\n", + " 'pointingAngle': None,\n", + " 'polarization': 'VV+VH',\n", + " 'processingDate': '2021-07-11T01:49:47.000Z',\n", + " 'processingLevel': 'SLC',\n", + " 'sceneName': 'S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404',\n", + " 'sensor': 'C-SAR',\n", + " 'startTime': '2021-07-11T01:49:47.000Z',\n", + " 'stopTime': '2021-07-11T01:50:13.000Z',\n", + " 'url': 'https://datapool.asf.alaska.edu/SLC/SB/S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404.zip',\n", + " 'pgeVersion': '003.31',\n", + " 'fileName': 'S1B_IW_SLC__1SDV_20210711T014947_20210711T015013_027740_034F80_D404.zip',\n", + " 'frameNumber': 108},\n", + " {'beamModeType': 'IW',\n", + " 'browse': None,\n", + " 'bytes': 4150793626,\n", + " 'centerLat': 32.9309,\n", + " 'centerLon': -117.2057,\n", + " 'faradayRotation': None,\n", + " 'fileID': 'S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D-SLC',\n", + " 'flightDirection': 'ASCENDING',\n", + " 'groupID': 'S1B_IWDV_0102_0109_027740_064',\n", + " 'granuleType': 'SENTINEL_1B_FRAME',\n", + " 'insarStackId': None,\n", + " 'md5sum': '8cad65891ffcbb32218cece857104ed3',\n", + " 'offNadirAngle': None,\n", + " 'orbit': 27740,\n", + " 'pathNumber': 64,\n", + " 'platform': 'Sentinel-1B',\n", + " 'pointingAngle': None,\n", + " 'polarization': 'VV+VH',\n", + " 'processingDate': '2021-07-11T01:49:22.000Z',\n", + " 'processingLevel': 'SLC',\n", + " 'sceneName': 'S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D',\n", + " 'sensor': 'C-SAR',\n", + " 'startTime': '2021-07-11T01:49:22.000Z',\n", + " 'stopTime': '2021-07-11T01:49:49.000Z',\n", + " 'url': 'https://datapool.asf.alaska.edu/SLC/SB/S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D.zip',\n", + " 'pgeVersion': '003.31',\n", + " 'fileName': 'S1B_IW_SLC__1SDV_20210711T014922_20210711T014949_027740_034F80_859D.zip',\n", + " 'frameNumber': 103}]}" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "out_slc = download_slcs(reference_scenes,\n", " secondary_scenes,\n", @@ -122,7 +324,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 10, "id": "suitable-hardware", "metadata": { "ExecuteTime": { @@ -130,8 +332,22 @@ "start_time": "2022-01-19T02:29:37.369332Z" } }, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "{'orbit_directory': 'orbits',\n", + " 'reference_orbits': [\"https://scihub.copernicus.eu/gnss/odata/v1/Products('b1085082-4941-4d7e-8e15-4ab9fa6bd118')/$value\"],\n", + " 'secondary_orbits': [\"https://scihub.copernicus.eu/gnss/odata/v1/Products('0a5b7c24-01de-4c15-a313-2ff02102f995')/$value\"]}" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ + "check_esa_credentials(None, None)\n", "out_orbit = download_orbits(reference_scenes,\n", " secondary_scenes,\n", " dry_run=dry_run)\n", @@ -158,7 +374,15 @@ "start_time": "2022-01-19T02:29:44.690665Z" } }, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Reading glo_30 Datasets: 100%|█| 15/15 [00:20<00:00,\n" + ] + } + ], "source": [ "out_dem = download_dem_for_isce2(out_slc['extent'])\n", "out_dem" @@ -264,7 +488,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.5" + "version": "3.9.16" } }, "nbformat": 4,