ODC SC Meeting 2019 07 25

Attendees

Apologies

Minutes

See Google Doc.

Agenda

Welcome
Previous Actions
- See below
Enhancement Proposals
- Proposals wiki
State of the Cube Highlights
- CSIRO
  - All of the CSIRO staff have taken significant leave during this period so a smaller number of outcomes this month
  - CSIRO Data pipelines (live ingester) and ODC/DEA indexing components have been successfully run on the CSIRO AWS EASI EO hub deployment:
    - DEA indexes had a dozen or so containers fail during indexing. No logs were returned, the pods just hung, with s3-find and some other processes just spinning small amounts of CPU time. The (vast majority) of other pods ran to completion successfully.
    - CSIRO data pipelines successfully ordered data from multiple global source archives and indexed/ingested into the CSIRO environment for small scale tests.
    - Next step: Scale up the CSIRO data pipelines robots in a manner similar to the DEA indexing approach (k8s jobs) and hopefully debug the DEA indexing issue
  - The CSIRO AWS EASI EO hub deployment has gained interest from a group of Pangeo connected folks in CSIRO Oceans and Atmosphere. A demonstration of the capability was provided and we are now looking at join development opportunity. The Pangeo stack, as an ODC would expect is basically the same as the ODC one but with additional emphasis on netcdf support.
    - One issue exists with holoviews - holoviews is currently crashing the jupyter notebook kernel for unknown reasons. It's probably an issue with library versions but if anyone has a working ODC with holoview (or more specifically geoviews) running we'd a appreciate a look at your image build so we can compare the library and OS package versions. The O&A folks normally use Conda to manage this but ODC is using pip so its a little more convoluted to resolve. - CSIRO EASI EO k8s environment was updated to support multiple-instance types when scheduling SPOT nodes, significantly increasing the SPOT node availability. PR has been posted to dataube-k8s-eks.
- Catapult
  - EO Team:
    - Amendment of [WOFS (or equivalent), Land cover classification, Coastal change] algorithms to work in the ODC Sandbox for the CommonSensing project
    - Amendment of [WOFS (or equivalent), Land cover classification, Coastal change] algorithms to work for Sentinel-2 and Sentinel-1 datasets
  - Engineering Team:
    - Set up of the Catapult's PDS in the form of an AWS S3 bucket, currently public, but IP whitelisting required to access data in order to control costs
    - Miscellaneous activities around the standalone ODC instance that can run on Binder
- GA
  - Collection Upgrade for the Australian data is well underway with some test data available in the DEA Sandbox
  - Digital Earth Africa work has begun, with data being prepared for a number of countries now and a water classifier is being run over this data, with web services and some example notebooks in the works.
  - Planning still underway regarding ODC 2.0 and the changes this entails.
- USGS
  - USGS is focused on delivery data over Africa to prototype Collection 2 - Level 2
  - GA will then index this data into an Open Data Cube and look at continental scale applications in the cloud
  - Tony Butzer and Randy Sunne will be largely offline for Steering Council activities during this very large data load operation
- AMA
  - TBA
- FrontierSI
  - Working with AMA setting up ODC Cluster
  - Working with GA on DEA sandbox
  - Working on learning portal for DEA
Other Business
- TBA
Next Meeting and Close
- Date