ODC SC Meeting 2020 07 09

Attendees

• Luigi di Fraia (SA Catapult)
• Federica Moscato (SA Catapult)
• Davide De Lerma (SA Catapult)
• Robert Woodcock (CSIRO)
• Syed Rizvi (AMA)
• Paul Haesler (CSIRO)
• Kirill Kouzoubov (GA)
• Alex Leith (GA)
• Tony Butzer (USGS)
• Tom Butler (GA)
• Andrew Hicks (FrontierSI)
• Randall Sunne (USGS)
• Andrew Cherry (AMA)
• Chris Morgan (FrontierSI)
• Caitlin Adams (FrontierSI)
• Peter Wang (CSIRO)
• George Dyke (Symbios)

Apologies

Minutes

See Google Drive document.

Agenda

• Welcome
• Previous Actions
• State of the Cube
  • CSIRO
    • 2D Zarr Driver
      • Examples with EO and EO3 e.g. product definition, Zarrify, & prepare scripts.
      • Currently working on examples and tutorials.
    • EO3 Assembler (CSIRO edition) developed and producing valid EO3. Based on the GA version and will be seeking to merge with that for general ODC use but the in development GA version wasn't easily modified to a general form (and am not suggesting the CSIRO one is! :-D ).
    • Trial run of JupyterHub 0.9.0 official release - Cognito auth provider has been deprecated as the generic oauth provider now supports Cognito. Configuration changes required, aside that initial testing doesn't look like a problem if you are already on 0.9.0.beta 3
    • Argo workflows - now have running scalable argo workflows for indexing DEA data on AWS and CSIRO data on AWS. Also have Cogify and prepare workflows at scale. Uses a combination of ODC and CSIRO data pipelines code on the running Pods but the orchestration is all done in Argo. Still building an understanding on where the bottlenecks are to scaling (network congestion to S3 and database writes) but Argo will quite happily flood the k8s cluster with as many Pods as you might like (yes, you can set limits).
    • Updated datacube-ows in EASI Hub - long overdue and only in dev at the moment but its running again. Did run into an issue with the AWS s3 'no-sign' configuration which prevent access to AWS s3 buckets that require permissions alongside the DEA public ones. Looks like the issue is related to assumptions in the datacube-ows default configuration code and the developer was going to log an issue for this to prevent recurrence.
  • Catapult
  • GA
    • Work continues on Datacube Statistician for summary data processing
    • DE Africa has supported the COG and STAC transforms for Sentinel-2 data (available at s3://sentinel-cogs)
    • 1.8.1 Datacube Core release including bug fixes and AWS S3 'no-sign' and 'request-payer' reads
  • USGS
  • AMA
  • FrontierSI
• Enhancement Proposals
  • New Proposals
  • Committed Proposals
• ODC V2.0
• Other Business
  • ODC Bug Squash Days aiming for August 18-19?
• Next Meeting and Close
  • TBA

Actions

See the action tracking project.

Outstanding Actions

• Example