From 3e45a050a15eae4ea9382ca2e967685f9ce2e5de Mon Sep 17 00:00:00 2001 From: Paul Haesler Date: Thu, 12 Dec 2024 12:04:49 +1100 Subject: [PATCH] Update and cross-reference migration notes. (#1686) * Juicing up migration notes, first pass. * Fix doc links. * Fix section ref links. * Still trying to fix section ref links. * Sphinx is weird. * Fixed errant copypasta. * whats_new.rst * Update docs/installation/MIGRATION-1.8-to-1.9.rst Co-authored-by: Damien Ayers * Update docs/installation/MIGRATION-1.8-to-1.9.rst Co-authored-by: Damien Ayers --------- Co-authored-by: Damien Ayers --- docs/about-core-concepts/datasets.rst | 4 +- docs/about/index.rst | 1 + docs/about/whats_new.rst | 2 + docs/index.rst | 6 +- .../MIGRATION-1.8-to-1.9.rst | 70 ++++++++++++------- docs/installation/database/setup.rst | 2 +- docs/installation/index.rst | 8 ++- 7 files changed, 59 insertions(+), 34 deletions(-) rename docs/{ => installation}/MIGRATION-1.8-to-1.9.rst (79%) diff --git a/docs/about-core-concepts/datasets.rst b/docs/about-core-concepts/datasets.rst index f23d89a34..96cbd60d9 100644 --- a/docs/about-core-concepts/datasets.rst +++ b/docs/about-core-concepts/datasets.rst @@ -5,8 +5,8 @@ Datasets are a fundamental part of the Open Data Cube. A dataset is *“The smal .. admonition:: Examples of Datasets :class: important - - a Landsat Scene​ - - an Albers Equal Area tile portion of a Landsat Scene​ + - a Landsat Scene + - an Albers Equal Area tile portion of a Landsat Scene Dataset metadata format diff --git a/docs/about/index.rst b/docs/about/index.rst index dd0cecab1..fc09dd785 100644 --- a/docs/about/index.rst +++ b/docs/about/index.rst @@ -9,3 +9,4 @@ About glossary license release_process + MIGRATION-1.8-to-1.9 diff --git a/docs/about/whats_new.rst b/docs/about/whats_new.rst index 8d82736d3..4e491603b 100644 --- a/docs/about/whats_new.rst +++ b/docs/about/whats_new.rst @@ -8,6 +8,8 @@ What's New v1.9.next ========= +- Update and cross-reference 1.8 to 1.9 migration notes (:pull:`1686`) + v1.9.0-rc12 (9th December 2024) =============================== diff --git a/docs/index.rst b/docs/index.rst index d69a01891..4cc5191ba 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -4,7 +4,7 @@ The Open Data Cube The Open Data Cube is an open-source software ecosystem for managing and processing gridded Earth observation data. -It has been designed to manage decades of analysis ready +It has been designed to manage decades of analysis ready Earth observation satellite data and related data from multiple sources. For more information see the following sections: @@ -16,11 +16,11 @@ For more information see the following sections: :doc:`installation/index` -* This section contains information on how to install the Open Data Cube. +* This section contains information on how to install, maintain and upgrade the Open Data Cube. :doc:`data-access-analysis/index` -* This section contains information on how to access and analyse data that +* This section contains information on how to access and analyse data that is managed by an Open Data Cube. :doc:`api/index` diff --git a/docs/MIGRATION-1.8-to-1.9.rst b/docs/installation/MIGRATION-1.8-to-1.9.rst similarity index 79% rename from docs/MIGRATION-1.8-to-1.9.rst rename to docs/installation/MIGRATION-1.8-to-1.9.rst index 4c20338f3..1b2df09d2 100644 --- a/docs/MIGRATION-1.8-to-1.9.rst +++ b/docs/installation/MIGRATION-1.8-to-1.9.rst @@ -15,8 +15,9 @@ approaches becoming the standard, but some minor backwards incompatible changes This document describes the changes between 1.8.x and 1.9.x, with a particular focus on backwards incompatible changes and new features. -After the release of 1.9.0, focus will shift to updating secondary ODC libraries to work with ODC-1.9. (Explorer -and OWS in particular will require major changes.) We will continue to support and release 1.8.x versions after +1.9 compatible releases of `eo-datasets`_, `explorer`_ and `OWS`_ will be released at the same time as datacube-1.9.0. +After the release of 1.9.0, focus will shift to updating secondary ODC libraries to work with ODC-1.9. (e.g. +`Statistician`_ and `Alchemist`_). We will continue to support and release 1.8.x versions after the release of 1.9.0, until the 1.9.x releases have stabilised and all secondary libraries are up to date. Smaller ODC installations will probably prefer to stick with the 1.8.x releases for the time being, but if you can @@ -24,26 +25,32 @@ spare the resources we encourage you to set up a 1.9.x installation to test your against the new release, and open issues on github for any problems that you come up against, especially any that are not documented here. +.. _`eo-datasets`: https://github.com/opendatacube/eo-datasets +.. _`explorer`: https://github.com/opendatacube/datacube-explorer +.. _`OWS`: https://github.com/opendatacube/datacube-ows +.. _`Statistician`: https://github.com/opendatacube/odc-stats +.. _`Alchemist`: https://github.com/opendatacube/datacube-alchemist + Major Changes between 1.8.x and 1.9.x ------------------------------------- -1. Integration with ``odc-geo``. +1. Integration with `odc-geo`_. - The old ``datacube.utils.geometry`` library has been replaced by ``odc-geo``. + The old ``datacube.utils.geometry`` library has been replaced by `odc-geo`_. - If you have already used ``odc-geo`` you will appreciate the additional power and flexibility that this brings to + If you have already used `odc-geo`_ you will appreciate the additional power and flexibility that this brings to core. If you have not, please take the time to have a read through the `odc-geo documentation `_ and especially the `migration notes `_. In particular, you should familiarise - yourself with ``.odc`` accessor which ``odc-geo`` dynamically adds to all xarray ``DataArray`` and ``Dataset`` + yourself with ``.odc`` accessor which `odc-geo`_ dynamically adds to all xarray ``DataArray`` and ``Dataset`` objects. - Note that ``dc.load()`` now preferentially accepts ``odc-geo`` data types for passing ``GeoBox`` via the ``like`` + Note that ``dc.load()`` now preferentially accepts `odc-geo`_ data types for passing ``GeoBox`` via the ``like`` parameter, as well as ``resolution`` and ``align`` values, although backwards compatible behaviour with the old types is available with a deprecation warning. The classes and methods in ``datacube.utils.geometry`` are still available, but raise a deprecation warning when - used. Please migrate all code to use the equivalent methods and classes in ``odc-geo``. + used. Please migrate all code to use the equivalent methods and classes in `odc-geo`_. 2. A new configuration engine has replaced the configuration engine used previously. @@ -51,27 +58,36 @@ Major Changes between 1.8.x and 1.9.x continue to work as previously with minimal changes. The behaviour of the new configuration engine (and the reasoning behind the changes) is fully documented in + :doc:`database/passing-configuration`. + + The reasoning behind the changes is described in `ODC Enhancement Proposal 10 `_ a. Previously multiple config files could be read and merged to generate the final effective configuration file. - From 1.9.0 only a single config file is ever read at a time. Managed instances which have previously allowed + From 1.9.0 only a single config file is ever read at a time + (see :ref:`Merging-multiple-config-files`). + + Managed instances which have previously allowed user customisation by the user creating a minimal config file which was loaded merged on top of a default system configuration will have to migrate to a system whereby users take a copy of the default system configuration file and edit that copy for their needs. - b. The "user" section no longer has a special meaning, as the old special meaning is irrelevant now that config + b. The "user" section no longer has a special meaning, as the old special meaning is redundant now that config files are not merged. c. Previously only the INI file format was supported for configuration files. The JSON and YAML formats are now also - supported. + supported (see :ref:`format`). - d. Previously configuration by Environment Variables was implemented in an inconsistent and ad hoc way that resulted + d. Previously configuration by Environment Variables was implemented in an inconsistent ad hoc way that resulted in complex interactions that were impossible to predict without intimate knowledge of the source code that - implemented it. There is now a consistent and systematic approach taken to the interaction between the - active configuration file and environment variables. Partial backwards compatibility is attempted, but + implemented it. + + There is now a consistent and systematic approach taken to the interaction between the + active configuration file and environment variables + (see :ref:`4. Generic Environment Variable Overrides`). Partial backwards compatibility is attempted, but full backwards compatibility is not possible due to the ad hoc nature of the previous implementation. - The new (preferred) environment variable names are of the form ``$ODC__`` + The new (preferred) environment variable names are of the form ``$ODC__``. e. Tighter restrictions are applied to environment names. This is required to ensure consistent interaction between config files and environment variables. Environment names can now only contain alphanumeric characters. @@ -94,7 +110,7 @@ Major Changes between 1.8.x and 1.9.x metadata, such tools may not be possible in all cases. (The legacy postgres driver will continue to support non-EO3 metadata types until it is dropped in 2.0) - The postgis driver will support the creation of PostGIS spatial indexes for arbitrary CRSes. This will improve + The postgis driver supports the creation of PostGIS spatial indexes for arbitrary CRSes. This will improve efficiency and accuracy of database searches, particularly when working with data covering regions where conversions to/from EPSG:4326 lat/long coordinates are highly non-linear (e.g. the Pacific around the anti-meridian and the north and south polar regions). @@ -102,11 +118,9 @@ Major Changes between 1.8.x and 1.9.x The postgis driver uses Alembic for managing schema migrations, so future changes to the postgis database schema will be much easier to roll out than in the past. - See below for more information about migrating to the Postgis index driver. + See below for more `information about migrating to the Postgis index driver<#the-new-postgis-index-driver>`_. - Note that many other libraries in the ODC ecosystem may not work well with the Postgis driver at first. As noted - above, Explorer and Datacube-OWS in particular will need extensive changes before they can be used with the new - index driver. + Note that many other libraries in the ODC ecosystem may not work well with the Postgis driver at first. 5. New Lineage API (Postgis driver only) @@ -123,13 +137,19 @@ Major Changes between 1.8.x and 1.9.x The handling of lineage in the legacy postgres index driver has not changed - the postgres driver does NOT support the new lineage API. -6. Support for multi-dimensional loading of hyperspectral datasets (Coming Soon) +6. Support for multi-dimensional loading of hyperspectral datasets - This is a work in progress and will not be available in 1.9.0. It will appear in a later 1.9.x release. + Use of this feature currently requires installing `odc-loader`_. + + If `odc-loader`_ is installed, you may specify `driver='rio'` or `driver='zarr'` when loading to access the new hyperspectral loading and + dask-chunking features. 7. The long-deprecated "ingestion" workflow and "executor" API have both been removed. -8. Multiple locations per dataset is now deprecated. +8. Multiple locations per dataset is now deprecated, and is not supported by the ``postgis`` index driver. + +.. _`odc-geo`: https://github.com/opendatacube/odc-geo +.. _`odc-loader`: https://github.com/opendatacube/odc-loader The New Postgis Index Driver ---------------------------- @@ -171,6 +191,8 @@ for EPSG:3577:: datacube -E new spindex create 3577 +For more information, see :ref:`Create Spatial Indexes (Postgis Driver Only)`. + Migrating (Cloning) Data From a Postgres Index ++++++++++++++++++++++++++++++++++++++++++++++ @@ -207,5 +229,3 @@ geopolygon. In the postgis driver, the polygon is passed directly to Postgis fo * Only datasets whose extents overlap the geopolygon will be loaded. * Geopolygons whose CRS does NOT have a native spatial index available will be projected to EPSG:4326 for search purposes. -* Datasets whose projected extents are not contained within a given CRSes "valid area" will NOT be included in that - CRSes spatial index. diff --git a/docs/installation/database/setup.rst b/docs/installation/database/setup.rst index 1e87b35c4..c11b7f0f2 100644 --- a/docs/installation/database/setup.rst +++ b/docs/installation/database/setup.rst @@ -173,7 +173,7 @@ To add a spatial index for an srid user ``datacube spindex add srid``, e.g. for Note that adding a new spatial index to an existing database does NOT add existing datasets to the spatial index! -After adding a new spatial index to a non-empty database it is necessary to **update** the spatial index afterk +After adding a new spatial index to a non-empty database it is necessary to **update** the spatial index after creation:: datacube spindex update 3577 diff --git a/docs/installation/index.rst b/docs/installation/index.rst index 0d17e478c..59b3a45b9 100644 --- a/docs/installation/index.rst +++ b/docs/installation/index.rst @@ -18,6 +18,11 @@ This section contains information on setting up and managing the Open Data Cube. setup/ubuntu setup/windows +.. toctree:: + :caption: Upgrading an Open Data Cube + :maxdepth: 5 + + MIGRATION-1.8-to-1.9 .. toctree:: :caption: Open Data Cube CLI @@ -25,7 +30,6 @@ This section contains information on setting up and managing the Open Data Cube. cli - .. toctree:: :caption: Configuring the ODC Database :maxdepth: 4 @@ -37,14 +41,12 @@ This section contains information on setting up and managing the Open Data Cube. product-definitions dataset-documents - .. toctree:: :caption: Indexing Data indexing-data/step-guide indexing-data/indexing-from-s3 - .. toctree:: :caption: Advanced Topics