Update and cross-reference migration notes. (#1686)

* Juicing up migration notes, first pass.

* Fix doc links.

* Fix section ref links.

* Still trying to fix section ref links.

* Sphinx is weird.

* Fixed errant copypasta.

* whats_new.rst

* Update docs/installation/MIGRATION-1.8-to-1.9.rst

Co-authored-by: Damien Ayers <[email protected]>

* Update docs/installation/MIGRATION-1.8-to-1.9.rst

Co-authored-by: Damien Ayers <[email protected]>

---------

Co-authored-by: Damien Ayers <[email protected]>

docs/about-core-concepts/datasets.rst

@@ -5,8 +5,8 @@ Datasets are a fundamental part of the Open Data Cube. A dataset is *“The smal
 .. admonition:: Examples of Datasets
   :class: important
 
-  - a Landsat Scene​
-  - an Albers Equal Area tile portion of a Landsat Scene​
+  - a Landsat Scene
+  - an Albers Equal Area tile portion of a Landsat Scene
 
 
 Dataset metadata format

docs/about/index.rst

@@ -9,3 +9,4 @@ About
    glossary
    license
    release_process
+   MIGRATION-1.8-to-1.9

docs/about/whats_new.rst

@@ -8,6 +8,8 @@ What's New
 v1.9.next
 =========
 
+- Update and cross-reference 1.8 to 1.9 migration notes (:pull:`1686`)
+
 v1.9.0-rc12 (9th December 2024)
 ===============================
 

docs/index.rst

@@ -4,7 +4,7 @@ The Open Data Cube
 
 The Open Data Cube is an open-source software ecosystem
 for managing and processing gridded Earth observation data.
-It has been designed to manage decades of analysis ready 
+It has been designed to manage decades of analysis ready
 Earth observation satellite data and related data from multiple sources.
 
 For more information see the following sections:
@@ -16,11 +16,11 @@ For more information see the following sections:
 
 :doc:`installation/index`
 
-* This section contains information on how to install the Open Data Cube.
+* This section contains information on how to install, maintain and upgrade the Open Data Cube.
 
 :doc:`data-access-analysis/index`
 
-* This section contains information on how to access and analyse data that 
+* This section contains information on how to access and analyse data that
   is managed by an Open Data Cube.
 
 :doc:`api/index`

docs/MIGRATION-1.8-to-1.9.rst → docs/installation/MIGRATION-1.8-to-1.9.rst

@@ -15,63 +15,79 @@ approaches becoming the standard, but some minor backwards incompatible changes
 This document describes the changes between 1.8.x and 1.9.x, with a particular focus on backwards incompatible
 changes and new features.
 
-After  the release of 1.9.0, focus will shift to updating secondary ODC libraries to work with ODC-1.9.  (Explorer
-and OWS in particular will require major changes.)   We will continue to support and release 1.8.x versions after
+1.9 compatible releases of `eo-datasets`_, `explorer`_ and `OWS`_ will be released at the same time as datacube-1.9.0.
+After  the release of 1.9.0, focus will shift to updating secondary ODC libraries to work with ODC-1.9.  (e.g.
+`Statistician`_ and `Alchemist`_).   We will continue to support and release 1.8.x versions after
 the release of 1.9.0, until the 1.9.x releases have stabilised and all secondary libraries are up to date.
 
 Smaller ODC installations will probably prefer to stick with the 1.8.x releases for the time being, but if you can
 spare the resources we encourage you to set up a 1.9.x installation to test your existing code and systems
 against the new release, and open issues on github for any problems that you come up against, especially any that are
 not documented here.
 
+.. _`eo-datasets`: https://github.com/opendatacube/eo-datasets
+.. _`explorer`: https://github.com/opendatacube/datacube-explorer
+.. _`OWS`: https://github.com/opendatacube/datacube-ows
+.. _`Statistician`: https://github.com/opendatacube/odc-stats
+.. _`Alchemist`: https://github.com/opendatacube/datacube-alchemist
+
 Major Changes between 1.8.x and 1.9.x
 -------------------------------------
 
-1. Integration with ``odc-geo``.
+1. Integration with `odc-geo`_.
 
-   The old ``datacube.utils.geometry`` library has been replaced by ``odc-geo``.
+   The old ``datacube.utils.geometry`` library has been replaced by `odc-geo`_.
 
-   If you have already used ``odc-geo`` you will appreciate the additional power and flexibility that this brings to
+   If you have already used `odc-geo`_ you will appreciate the additional power and flexibility that this brings to
    core.  If you have not, please take the time to have a read through the
    `odc-geo documentation <https://odc-geo.readthedocs.io/en/latest/>`_  and especially the
    `migration notes <https://odc-geo.readthedocs.io/en/latest/migration.html>`_.  In particular, you should familiarise
-   yourself with ``.odc`` accessor which ``odc-geo`` dynamically adds to all xarray ``DataArray`` and ``Dataset``
+   yourself with ``.odc`` accessor which `odc-geo`_ dynamically adds to all xarray ``DataArray`` and ``Dataset``
    objects.
 
-   Note that ``dc.load()`` now preferentially accepts ``odc-geo`` data types for passing ``GeoBox`` via the ``like``
+   Note that ``dc.load()`` now preferentially accepts `odc-geo`_ data types for passing ``GeoBox`` via the ``like``
    parameter, as well as ``resolution`` and ``align`` values, although backwards compatible behaviour with the old
    types is available with a deprecation warning.
 
    The classes and methods in ``datacube.utils.geometry`` are still available, but raise a deprecation warning when
-   used.  Please migrate all code to use the equivalent methods and classes in ``odc-geo``.
+   used.  Please migrate all code to use the equivalent methods and classes in `odc-geo`_.
 
 2. A new configuration engine has replaced the configuration engine used previously.
 
    There are some backwards-incompatible changes as noted below, but most existing configuration files should
    continue to work as previously with minimal changes.
 
    The behaviour of the new configuration engine (and the reasoning behind the changes) is fully documented in
+   :doc:`database/passing-configuration`.
+
+   The reasoning behind the changes is described in
    `ODC Enhancement Proposal 10 <https://github.com/opendatacube/datacube-core/wiki/ODC-EP-010---Replace-Configuration-Layer>`_
 
    a. Previously multiple config files could be read and merged to generate the final effective configuration file.
-      From 1.9.0 only a single config file is ever read at a time.  Managed instances which have previously allowed
+      From 1.9.0 only a single config file is ever read at a time
+      (see :ref:`Merging-multiple-config-files`).
+
+      Managed instances which have previously allowed
       user customisation by the user creating a minimal config file which was loaded merged on top of a default system
       configuration will have to migrate to a system whereby users take a copy of the default system configuration file
       and edit that copy for their needs.
 
-   b. The "user" section no longer has a special meaning, as the old special meaning is irrelevant now that config
+   b. The "user" section no longer has a special meaning, as the old special meaning is redundant now that config
       files are not merged.
 
    c. Previously only the INI file format was supported for configuration files. The JSON and YAML formats are now also
-      supported.
+      supported (see :ref:`format`).
 
-   d. Previously configuration by Environment Variables was implemented in an inconsistent and ad hoc way that resulted
+   d. Previously configuration by Environment Variables was implemented in an inconsistent ad hoc way that resulted
       in complex interactions that were impossible to predict without intimate knowledge of the source code that
-      implemented it.  There is now a consistent and systematic approach taken to the interaction between the
-      active configuration file and environment variables.  Partial backwards compatibility is attempted, but
+      implemented it.
+
+      There is now a consistent and systematic approach taken to the interaction between the
+      active configuration file and environment variables
+      (see :ref:`4. Generic Environment Variable Overrides`).  Partial backwards compatibility is attempted, but
       full backwards compatibility is not possible due to the ad hoc nature of the previous implementation.
 
-      The new (preferred) environment variable names are of the form ``$ODC_<env_name>_<item_name>``
+      The new (preferred) environment variable names are of the form ``$ODC_<env_name>_<item_name>``.
 
    e. Tighter restrictions are applied to environment names.  This is required to ensure consistent interaction
       between config files and environment variables.  Environment names can now only contain alphanumeric characters.
@@ -94,19 +110,17 @@ Major Changes between 1.8.x and 1.9.x
    metadata, such tools may not be possible in all cases.  (The legacy postgres driver will continue to support
    non-EO3 metadata types until it is dropped in 2.0)
 
-   The postgis driver will support the creation of PostGIS spatial indexes for arbitrary CRSes.  This will improve
+   The postgis driver supports the creation of PostGIS spatial indexes for arbitrary CRSes.  This will improve
    efficiency and accuracy of database searches, particularly when working with data covering regions where
    conversions to/from EPSG:4326 lat/long coordinates are highly non-linear (e.g. the Pacific around the
    anti-meridian and the north and south polar regions).
 
    The postgis driver uses Alembic for managing schema migrations, so future changes to the postgis database
    schema will be much easier to roll out than in the past.
 
-   See below for more information about migrating to the Postgis index driver.
+   See below for more `information about migrating to the Postgis index driver<#the-new-postgis-index-driver>`_.
 
-   Note that many other libraries in the ODC ecosystem may not work well with the Postgis driver at first. As noted
-   above, Explorer and Datacube-OWS in particular will need extensive changes before they can be used with the new
-   index driver.
+   Note that many other libraries in the ODC ecosystem may not work well with the Postgis driver at first.
 
 5. New Lineage API (Postgis driver only)
 
@@ -123,13 +137,19 @@ Major Changes between 1.8.x and 1.9.x
    The handling of lineage in the legacy postgres index driver has not changed - the postgres driver does NOT support
    the new lineage API.
 
-6. Support for multi-dimensional loading of hyperspectral datasets (Coming Soon)
+6. Support for multi-dimensional loading of hyperspectral datasets
 
-   This is a work in progress and will not be available in 1.9.0. It will appear in a later 1.9.x release.
+   Use of this feature currently requires installing `odc-loader`_.
+
+   If `odc-loader`_ is installed, you may specify `driver='rio'` or `driver='zarr'` when loading to access the new hyperspectral loading and
+   dask-chunking features.
 
 7. The long-deprecated "ingestion" workflow and "executor" API have both been removed.
 
-8. Multiple locations per dataset is now deprecated.
+8. Multiple locations per dataset is now deprecated, and is not supported by the ``postgis`` index driver.
+
+.. _`odc-geo`: https://github.com/opendatacube/odc-geo
+.. _`odc-loader`: https://github.com/opendatacube/odc-loader
 
 The New Postgis Index Driver
 ----------------------------
@@ -171,6 +191,8 @@ for EPSG:3577::
 
     datacube -E new spindex create 3577
 
+For more information, see :ref:`Create Spatial Indexes (Postgis Driver Only)`.
+
 Migrating (Cloning) Data From a Postgres Index
 ++++++++++++++++++++++++++++++++++++++++++++++
 
@@ -207,5 +229,3 @@ geopolygon.  In the postgis driver, the polygon is passed directly to Postgis fo
 * Only datasets whose extents overlap the geopolygon will be loaded.
 * Geopolygons whose CRS does NOT have a native spatial index available will be projected to EPSG:4326 for search
   purposes.
-* Datasets whose projected extents are not contained within a given CRSes "valid area" will NOT be included in that
-  CRSes spatial index.

docs/installation/database/setup.rst

@@ -173,7 +173,7 @@ To add a spatial index for an srid user ``datacube spindex add srid``, e.g. for
 
 Note that adding a new spatial index to an existing database does NOT add existing datasets to the spatial index!
 
-After adding a new spatial index to a non-empty database it is necessary to **update** the spatial index afterk
+After adding a new spatial index to a non-empty database it is necessary to **update** the spatial index after
 creation::
 
    datacube spindex update 3577

docs/installation/index.rst

@@ -18,14 +18,18 @@ This section contains information on setting up and managing the Open Data Cube.
     setup/ubuntu
     setup/windows
 
+.. toctree::
+    :caption: Upgrading an Open Data Cube
+    :maxdepth: 5
+
+    MIGRATION-1.8-to-1.9
 
 .. toctree::
     :caption: Open Data Cube CLI
     :maxdepth: 4
 
     cli
 
-
 .. toctree::
     :caption: Configuring the ODC Database
     :maxdepth: 4
@@ -37,14 +41,12 @@ This section contains information on setting up and managing the Open Data Cube.
     product-definitions
     dataset-documents
 
-
 .. toctree::
     :caption: Indexing Data
 
     indexing-data/step-guide
     indexing-data/indexing-from-s3
 
-
 .. toctree::
     :caption: Advanced Topics