ODC SC Meeting 2020 01 30

Attendees

Apologies

Minutes

See Google Drive document.

Agenda

Welcome
Previous Actions
State of the Cube
- CSIRO
  - Continuing work on CSIRO data-pipelines (live ingester):
    - Generic prepare scripts.
      - Loads all product definitions into memory.
      - Auto matches a source dataset to a product definition and picks the one with the highest score.
      - User mostly needs to provide metadata extraction for the dataset.
    - Testing real time operation on a single node before deploying to EASI.
      - Subscription based and launches a series of daily orders.
      - Decouples ordering from download/processing.
      - Historical catch up, then switch to real time.
- Catapult
  - Continuing work on ARD processing workflows:
    - Landsat 5-8
    - Sentinel-2
    - Sentinel-1 (including complications around the anti-meridian over Fiji)
  - Continuing work on CommonSensing Notebooks.
  - Working on approaches to leverage our Dask cluster in the above mentioned Notebooks.
  - Working on a Cube REST API for product generation.
- GA
  - Alex Leith attended the Radiant Earth ML Workshop and had a discussion with some folks from Pangeo there. Alex would like to develop a dynamic STAC API off the ODC database, so that we can integrate in a light way with the Pangeo stack.
  - Work is underway preparing data for the ODC Hackathon over Ghana. It will be Sentinel-1, Sentinel-2 and Landsat collection 1.
  - We're working with Element 84 on converting Sentinel-2 data over Africa to COGs with STAC in us-west-2
  - We continue to work with datacube-alchemist, which is a scene-to-scene WOfS and FC processor (but can be used for more algorithms). Current focus is doing Sentinel-2 data in Australia on AWS.
  - And we're transitioning super computers, migrating processes to the new one called Gadi.
  - Test infrastructure is now fully containerised (./check-code.sh --with-docker will run all the tests on dev machine with only dev machine requirement being -- "docker is installed and user has permissions to run it")
  - Using Github Actions instead of TravisCI for continuous intergation testing, typical CI tests time is down to around 5 minutes (15-20 was common previously)
  - Improvements for dockerized workflows: datacube can be configured via single environment variable now, no need to write out config files via entry point or other orchestration means.
- USGS
- AMA
  - Open source release of updated UI, and several new notebooks.
  - Working to download LS7 & LS8 2018-2019 data for the 5 ARDC countries.
  - Working to deploy a light-weight ODC Sandbox with demo notebooks.
  - Working on creating a connection between Google Earth Engine and Open Data Cube.
  - Providing support to ARDC training with AWS infra. and up-to-date UN SDG notebooks.
  - Working to move ARDC from ingested data to indexed data.
  - Working with other countries wanting to use ODC, e.g., Peru, El Salvador, etc.
  - Other projects - California Cube, Virginia Cube, etc.
- FrontierSI
Enhancement Proposals
- New Proposals
- Committed Proposals
ODC V1.8 and V2.0
Hack-a-thon
- 24-28 February 2020
Other Business
Next Meeting and Close
- TBA