From 168d3f7a4a83bc57b3d18b56134b139d73fd013f Mon Sep 17 00:00:00 2001 From: Alice Fage Date: Mon, 11 Sep 2023 11:07:43 +1200 Subject: [PATCH] feat: publish to ODR TDE-801 (#282) Updated documentation and GitHub Action to switch over to ODR. --------- Co-authored-by: Daniel Silk --- .github/workflows/publish.yml | 11 ++++++++++- README.md | 12 ++++++------ docs/naming.md | 4 ++-- docs/tools.md | 2 +- docs/usage.md | 18 +++++++++--------- template/catalog.json | 4 ++-- 6 files changed, 30 insertions(+), 21 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 2cc5b05c14..5fb7d69a85 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -35,9 +35,18 @@ jobs: mask-aws-account-id: true role-to-assume: ${{ secrets.AWS_CI_ROLE }} + - name: AWS Configure ODR + if: github.ref == 'refs/heads/master' + uses: aws-actions/configure-aws-credentials@v2 + with: + aws-region: ap-southeast-2 + mask-aws-account-id: true + role-to-assume: ${{ secrets.AWS_ODR_DATAMANAGER_ROLE }} + role-chaining: true + # Sync STAC files only on push to 'master' - name: Sync STAC uses: docker://ghcr.io/linz/argo-tasks:v2 if: github.ref == 'refs/heads/master' with: - args: stac-sync /github/workspace/stac/ s3://linz-imagery/ + args: stac-sync /github/workspace/stac/ s3://nz-imagery/ diff --git a/README.md b/README.md index 0aba4f3d3e..119481f48e 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,21 @@ # New Zealand Imagery -Toitū Te Whenua makes New Zealand’s most up-to-date publicly owned aerial imagery freely available to use under an open licence. You can access this through the [LINZ Data Service](https://data.linz.govt.nz/data/category/aerial-photos/?s=n), [LINZ Basemaps](https://basemaps.linz.govt.nz/#@-41.8899962,174.0492437,z5) or Amazon AWS. +Toitū Te Whenua makes New Zealand's most up-to-date publicly owned aerial imagery freely available to use under an open licence. You can access this through the [LINZ Data Service](https://data.linz.govt.nz/data/category/aerial-photos/?s=n), [LINZ Basemaps](https://basemaps.linz.govt.nz/#@-41.8899962,174.0492437,z5) or the [Registry of Open Data on AWS](https://registry.opendata.aws/nz-imagery/). This repository contains STAC Collection metadata for each imagery dataset, as well as some guidance documentation: -- [Naming](docs/naming.md) covers the s3://linz-imagery bucket naming structure + +- [Naming](docs/naming.md) covers the s3://nz-imagery bucket naming structure - [Tools](docs/tools.md) covers some of the STAC ecosystem tools that can be used to interact with our STAC Catalog - [Usage](docs/usage.md) shows how TIFFs can be interacted with from S3 using GDAL, QGIS, etc - ## AWS Access -Toitū Te Whenua provides a public requester pays bucket `s3://linz-imagery` in `ap-southeast-2`. +Toitū Te Whenua owns and maintains a public bucket which is sponsored and shared via the [Registry of Open Data on AWS](https://registry.opendata.aws/nz-imagery/) `s3://nz-imagery` in `ap-southeast-2`. -Using the [AWS CLI](https://aws.amazon.com/cli/) anyone with a AWS account can access all of the imagery mentioned inside of this repository. +Using the [AWS CLI](https://aws.amazon.com/cli/) anyone can access all of the imagery specified in this repository. ``` -aws s3 ls s3://linz-imagery --request-payer requester +aws s3 ls --no-sign-request s3://nz-imagery/ ``` ## License diff --git a/docs/naming.md b/docs/naming.md index 1a2ce12455..4f6e4b4d13 100644 --- a/docs/naming.md +++ b/docs/naming.md @@ -16,13 +16,13 @@ Imagery is named according to the majority region that imagery covers while also 7.5cm RGB imagery covering Auckland captured in 2022 ``` -s3://linz-imagery/auckland/auckland_2022_0.075m/rgb/2193/collection.json +s3://nz-imagery/auckland/auckland_2022_0.075m/rgb/2193/collection.json ``` 10m Satellite RGB imagery covering the North Island captured in 2023 ``` -s3://linz-imagery/new-zealand/north-island_2023_10m/rgb/2193/collection.json +s3://nz-imagery/new-zealand/north-island_2023_10m/rgb/2193/collection.json ``` ### Characters diff --git a/docs/tools.md b/docs/tools.md index 51d8f07614..e2c44f1dc7 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -8,7 +8,7 @@ A list of tools is available on [the STAC website](https://stacspec.org/en/about To deploy STAC Browser, you can follow [this documentation](https://github.com/radiantearth/stac-browser#get-started). -For a quick start, you can use [the demo instance](https://radiantearth.github.io/stac-browser/), pass the link to the New Zealand Imagery STAC Catalog `https://linz-imagery.s3-ap-southeast-2.amazonaws.com/catalog.json` and click on the `Load` button. +For a quick start, you can use [the demo instance](https://radiantearth.github.io/stac-browser/), pass the link to the New Zealand Imagery STAC Catalog `https://nz-imagery.s3-ap-southeast-2.amazonaws.com/catalog.json` and click on the `Load` button. ### Filtering the data diff --git a/docs/usage.md b/docs/usage.md index 94f16250cc..455b92e3d8 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1,8 +1,8 @@ # Get the link to the TIFF -The STAC Catalog, https://linz-imagery.s3-ap-southeast-2.amazonaws.com/catalog.json, is the entry point of the LINZ imagery data. It points to the different STAC Collections representing a dataset. +The STAC Catalog, https://nz-imagery.s3-ap-southeast-2.amazonaws.com/catalog.json, is the entry point of the LINZ imagery data. It points to the different STAC Collections representing a dataset. -> **_Note:_** The files on our AWS S3 bucket `linz-imagery` can be accessible with a `s3` path `s3://linz-imagery` or a `https` URL `https://linz-imagery.s3-ap-southeast-2.amazonaws.com/`. +> **_Note:_** The files on the AWS S3 bucket `nz-imagery` can be accessible with a `s3` path `s3://nz-imagery` or a `https` URL `https://nz-imagery.s3-ap-southeast-2.amazonaws.com/`. ## Manually navigate through the STAC Catalog to a get a TIFF @@ -20,13 +20,13 @@ This is an example using the "Wellington 0.075m Urban Aerial Photos (2021)" data } ``` -2. The Collection can be accessible from this GitHub repository, `stac/wellington/wellington_2021_0.075m/rgb/2193/collection.json` or from `https://linz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/collection.json`. From the Collection get an Item link in the `links` list: +2. The Collection can be accessible from this GitHub repository, `stac/wellington/wellington_2021_0.075m/rgb/2193/collection.json` or from `https://nz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/collection.json`. From the Collection get an Item link in the `links` list: ```json { "rel": "item", "href": "./BQ31_500_040071.json", "type": "application/json" } ``` -3. The Item can be accessible from `https://linz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.json`. A list of assets can be found: +3. The Item can be accessible from `https://nz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.json`. A list of assets can be found: ```json "assets": { @@ -39,7 +39,7 @@ This is an example using the "Wellington 0.075m Urban Aerial Photos (2021)" data ``` -In this case, the TIFF can be accessible from `https://linz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.tiff` +In this case, the TIFF can be accessible from `https://nz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.tiff` > **_Note:_** Our TIFF files and STAC Item files use the same base name. Having the link to the STAC Item, you can determine the TIFF link by changing its suffix from `.json` to `.tiff`. @@ -55,13 +55,13 @@ This tool is described in [this page](tools.md). GDAL allows you to run command on a file that is not on your local machine using a virtual file systems path: -- `/vicurl/https://linz-imagery.s3-ap-southeast-2.amazonaws.com/[PATH]` -- `/vsis3/s3://linz-imagery/[PATH]` +- `/vicurl/https://nz-imagery.s3-ap-southeast-2.amazonaws.com/[PATH]` +- `/vsis3/s3://nz-imagery/[PATH]` ## gdalinfo ```bash -gdalinfo /vsicurl/https://linz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.tiff +gdalinfo /vsicurl/https://nz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.tiff ```
@@ -150,7 +150,7 @@ Band 4 Block=512x512 Type=Byte, ColorInterp=Alpha 1. In QGIS, open the "Data Source Manager" (press `ctrl+L`) 2. Select "Protocol: HTTP(S)" as the "Source Type" 3. Select "HTTP/HTTPS/FTP" as the "Protocol Type" -4. Paste the HTTPS URL to the TIFF, for example `https://linz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.tiff` +4. Paste the HTTPS URL to the TIFF, for example `https://nz-imagery.s3-ap-southeast-2.amazonaws.com/wellington/wellington_2021_0.075m/rgb/2193/BQ31_500_040071.tiff` 5. Click on the "Add" button and wait for the file to load. ![Data Source Manager](img/usage/qgis_data-source-manager.png) diff --git a/template/catalog.json b/template/catalog.json index d3ec053aaf..4efb8a24ec 100644 --- a/template/catalog.json +++ b/template/catalog.json @@ -1,10 +1,10 @@ { "stac_version": "1.0.0", "type": "Catalog", - "id": "linz-imagery", + "id": "nz-imagery", "description": "Toitū Te Whenua Land Information New Zealand makes New Zealand's publicly owned aerial and satellite imagery archive freely available to use under an open licence. This public S3 bucket has been made available to enable bulk access and cloud-based data processing. You can also access the imagery through the LINZ Data Service or LINZ Basemaps.", "links": [ - { "rel": "self", "href": "https://linz-imagery.s3.ap-southeast-2.amazonaws.com/catalog.json" }, + { "rel": "self", "href": "https://nz-imagery.s3.ap-southeast-2.amazonaws.com/catalog.json" }, { "rel": "root", "href": "./catalog.json" } ] }