-
Notifications
You must be signed in to change notification settings - Fork 18
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add more background information about how basemaps works. BM-775 (
#2988) #### Motivation Basemaps from the outside looks to be a very complicated set of moving parts and has little documetion to shed light on why it was created and what it is good for. #### Modification Add some background information on the configuration, processing and tile serving aspects of the service. #### Checklist _If not applicable, provide explanation of why._ - [ ] Tests updated - [ ] Docs updated - [ ] Issue linked in Title --------- Co-authored-by: Daniel Silk <[email protected]>
- Loading branch information
Showing
13 changed files
with
210 additions
and
98 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,73 @@ | ||
| # Overview | ||
| # LINZ Basemaps | ||
|
|
||
| ## What is LINZ Basemaps? | ||
| [LINZ Basemaps](https://basemaps.linz.govt.nz) is a collection of tools to create and serve vector and raster basemaps using open source and open standards. It is designed to be light weight, cost efficient and fast. | ||
|
|
||
| A digital basemap provides a consistent background detail necessary to orient location and add to aesthetic appeal. Basemaps can be made up of streets, parcels, boundaries (country, regional, and city boundaries), shaded relief of a digital elevation model, waterways, hydrography, aerial and satellite imagery. Basemaps can be used as desktop, website or mobile phone application components, or as a 3rd party layers within a GIS or desktop mapping application. | ||
| Basemaps currently supports both [Imagery](#aerial--satellite-imagery) and [Vector data](#vector-data) | ||
|
|
||
| ## Background | ||
|
|
||
| Land Information New Zealand has a large archive of aerial and satellite imagery freely available from [LINZ Data Service](https://data.linz.govt.nz) and the [Registry of Open Data on AWS](https://registry.opendata.aws/nz-imagery/). This imagery was a primary driver for the creation of the LINZ Basemaps product and service. For more information on how LINZ acquires, ingests and publishes aerial and satellite imagery see [linz/imagery](https://github.com/linz/imagery). | ||
|
|
||
| While linz/basemaps is public, its primary reason is to [work in the open](https://www.digital.govt.nz/standards-and-guidance/digital-service-design-standard/principles/work-in-the-open/), so that we can share knowledge and foster collaboration with peers across the world. While the LINZ Basemaps product and services can be customised and deployed we are primarily focused on our own use cases of sharing New Zealand's Aerial Imagery archive and Topographic vector data. | ||
|
|
||
| ## Basemaps Service | ||
|
|
||
|  The core of the LINZ Basemaps Service consists of | ||
|
|
||
| - AWS S3 - COG Storage location | ||
| - AWS Lambda function - Tile Server - [@basemaps/lambda-tiler](../packages/lambda-tiler/README.md) | ||
| - AWS Cloudfront - Content distribution and tile cache | ||
| - [linz/basemaps-config](https://github.com/linz/basemaps) - Basemaps configuration | ||
|
|
||
| ### Imagery Tile Requests | ||
|
|
||
| A tile request flows a number of basemaps packages: | ||
|
|
||
| Given a request `/v1/tiles/aerial/WebMercatorQuad/6/1/40.webp`, | ||
| Breaking down this request url `lambda-tiler` needs to serve a `webp` image that for the tile `{z: 6, x: 1, y: 40}` from the `WebMercatorQuad` tile matrix from the tile set named `aerial` in the default configuration. | ||
|
|
||
| Which will then utlize the following packages | ||
|
|
||
| - [@basemaps/config](../packages/config/README.md) - Load configuration ([`?config`](./configuration.md)) | ||
| - [@basemaps/lambda-tiler](../packages/lambda-tiler/README.md) Convert tile XYZ into output coordinates bounding box | ||
| - [@basemaps/lambda-tiler](../packages/lambda-tiler/README.md) Determine which tiffs intersect the output coordinates | ||
| - [@basemaps/tiler](../packages/tiler/README.md) Determine which parts of the tiffs need to be loaded and resampled | ||
| - [@basemaps/tiler-sharp](../packages/tiler-sharp/README.md) - Load the bytes from the tiff and render it as a output image. | ||
|
|
||
| Imagery is stored as cloud optimised geotiffs by survey then layered together dynamically based on a [configured priority order](./configuration.md). | ||
|
|
||
| [@basemaps/tiler](../packages/tiler/README.md) takes the collection of tiffs and determines how resample (resize/crop) and paint them onto the output tile. | ||
|
|
||
|  | ||
|
|
||
| [@basemaps/tiler-sharp](../packages/tiler-sharp/README.md) uses [sharp/libvips](https://github.com/lovell/sharp) to take the output of [@basemaps/tiler](../packages/tiler/README.md) then load the raw tiff bytes and render them into output image. | ||
|
|
||
| ### Vector tile requests | ||
|
|
||
| Vector tiles are pre-rendered as [Mapbox Vector Tiles](https://docs.mapbox.com/data/tilesets/guides/vector-tiles-introduction/) and served directly from a [Cloud Optimised Tar](https://github.com/linz/cotar). Since there is no merging or dynamic configuration the tile server will just request the specified tile directly from the tar file. | ||
|
|
||
| ### Additional Processing | ||
|
|
||
| #### Aerial & Satellite Imagery Processing | ||
|
|
||
| To efficiently serve aerial imagery to huge number of consumers, it should be optimised into a format that makes it easy to serve. All of LINZ's imagery is stored as [Cloud Optimised GeoTIFFs (COGs)](https://www.cogeo.org/), and to ensure they are served as efficiently as possible some additional processing steps are generally pre-applied to the imagery. | ||
|
|
||
| - Reprojection: reprojecting the imagery into common consumption formats (EPSG:3857 and EPSG:2193) will greatly reduce the amount of load and complexity of the basemaps service service | ||
| - Tile Alignment: By aligning the COGs to the output XYZ tile grid | ||
| - Additional overviews: It is hard to fetch data from 1,000s of COGs to create a single tile, so we create additional overviews up to z0. | ||
|
|
||
| These additional processing steps are part of the basemaps [import process](#TODO) and are controlled by [@basemaps/cogify](../packages/cogify/README.md) | ||
|
|
||
| The additional steps are optional but do **greatly improve** the tile serving performance of the service. | ||
|
|
||
| #### Vector Data | ||
|
|
||
| Vector data is processed from the LINZ topographic vector datasets using tippecanoe, this process is still a work in progress and will. | ||
|
|
||
| ## Development | ||
|
|
||
| Deployments are handled by Github Actions see [Deployments](./deployment.md) for more information. | ||
|
|
||
| ### Diagrams | ||
|
|
||
| All diagrams across this repository are created with [excalidraw](https://excalidraw.com/) and embed the excalidraw configuration inside them, so can be re-edited on https://excalidraw.com/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| ## Configuration | ||
|
|
||
| Basemaps has two main components to its configuration | ||
|
|
||
| - Imagery - List of tiffs that make up a imagery set | ||
| - TileSet - List of imagery or vector layers that make up a output layer. | ||
|
|
||
| The configuration can be stored as individual AWS DynamoDB objects or can be stored as a bundled JSON object. | ||
|
|
||
| The configuration is generally generated directly from the source tiffs and associated STAC metadata using the [@basemaps/cli](../packages/cli/README.md) import process, with LINZ's specific configuration being stored in [linz/basemaps-config](https://github.com/linz/basemaps-config). | ||
|
|
||
| This allows the configuration objects to be somewhat short, below is a snippet from the [aerial](https://github.com/linz/basemaps-config/blob/master/config/tileset/aerial.json) tile set configuration which layers 100+ aerial imagery layers together. | ||
|
|
||
| ```json5 | ||
| { | ||
| // Raster or Vector layer | ||
| "type": "raster", | ||
| // all id's are prefixed by type eg 'ts_' is TileSet | ||
| "id": "ts_aerial", | ||
| // Human friendly name | ||
| "title": "Aerial Imagery Basemap", | ||
| "category": "Basemaps", | ||
| // If there is no data create a background color as hex RGBA | ||
| "background": "dce9edff", | ||
| "layers": [ | ||
| { | ||
| // Source location for EPSG:2193 Imagery | ||
| "2193": "s3://linz-basemaps/2193/gebco_2020_Nztm2000Quad_305-75m/01F1BFJN8R8P7BXN3XTHC5MT5G", | ||
| // Source location for EPSG:3857 Imagery | ||
| "3857": "s3://linz-basemaps/3857/gebco_2020_305-75m/01EDMTM3P563P06TWYQAZRA9F6", | ||
| // url/slug friendly name of the imagery set | ||
| "name": "gebco-2020-305.75m", | ||
| // At what level should the imagery be turned off | ||
| "maxZoom": 15, | ||
| // Human friendly name for the imagery | ||
| "title": "GEBCO Gridded Bathymetry (2020)", | ||
| "category": "Bathymetry" | ||
| }, | ||
| ... | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| When this configuration is imported it will list all of the source locations looking for `collection.json` and `*.tiff|*.tif` files. | ||
|
|
||
| ### Dynamic configuration | ||
|
|
||
| Configuration can be stored as single JSON object in s3 and then passed directly to the basemaps service with `?config=s3://linz-basemaps/config/config-latest.json.gz` this allows basemaps to preview a configuration before being deployed. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Deployment | ||
|
|
||
| Deployments of Basemaps are managed with github actions. | ||
|
|
||
| To trigger a deployment, make sure your branch is up to date and run the version bump script [../scripts/version.bump.sh](../scripts/version.bump.sh) | ||
| This script will create a `release:` commit and branch, please review the commit then create a pull request from it. | ||
|
|
||
| Once the release pull request is merged the CI system will deploy the released version into dev then into production. | ||
|
|
||
| ## Deployment Rollback | ||
|
|
||
| If a deployment breaks production, don't try to fix it on the fly, this risks introducing more errors and downtime. The fastest way to ensure production is stable again is to roll back to the previous release immediately and then focus on fixing the problem before the next release. | ||
|
|
||
| As Basemaps deployments are managed with GitHub Actions, every release will bundle the release packages and deployment in the GitHub Action run automatically. So, it is very simple for us to roll back to a previous release as all the previous deployments remain in history. Please use the following steps to trigger a roll back when needed. | ||
|
|
||
| - Open the Basemaps Deployments page - https://github.com/linz/basemaps/deployments | ||
| - Choose the environment that needs to be rolled back | ||
| - Open the workflow run for an older release | ||
|
|
||
|  | ||
|
|
||
| - Select the `Build / deploy-prod (push)` workflow and re-run it to roll back to a previous release. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.