From 6625cb7f7d846ea25bbd9f809f588f4d5ec1bac1 Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Thu, 12 Dec 2024 14:46:39 +0100 Subject: [PATCH 1/6] add rfc-6 --- rfc/6/index.md | 378 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 378 insertions(+) create mode 100644 rfc/6/index.md diff --git a/rfc/6/index.md b/rfc/6/index.md new file mode 100644 index 00000000..93b68528 --- /dev/null +++ b/rfc/6/index.md @@ -0,0 +1,378 @@ +# RFC-6: Multiscale + +Turn the `multiscales` array into a single `multiscale` obejct. + + +## Status + +This proposal is very early. Status: D1 + +| Name | GitHub Handle | Institution | Date | Status | +| --------- | ------------- | ----------- | ---------- | ------------------------------------- | +| Norman Rzepka | [normanrz](https://github.com/normanrz) | scalable minds | 2024-12-03 | Author | +| David Stansby | [dstansby](https://github.com/dstansby) | University College London | 2024-12-03 | Author | + +## Overview + +This RFC proposes to change the `multiscales` array into a single `multiscale` object. + +## Background + +The current spec of OME-Zarr (version 0.5) defines that the metadata for a multiscale is stored in a `multiscales` array. + +However, there seem to only very few OME-Zarr images with mutltiple multiscales in the wild. There is one example from IDR: [4995115.zarr](https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0050A/4995115.zarr) + +Additionally, most visualization and processing tools today simply process the first multiscale object in the `multiscales` array, ignoring any subsequent entries. Here is a selection of tools that only load the first multiscale object: + +- [neuroglancer](https://github.com/google/neuroglancer/blob/master/src/datasource/zarr/ome.ts#L265-L310) +- [vizarr](https://github.com/hms-dbmi/vizarr/blob/main/src/utils.ts#L88) +- [itk-vtk](https://github.com/Kitware/itk-vtk-viewer/blob/master/src/IO/ZarrMultiscaleSpatialImage.js#L173) +- [OMERO](https://github.com/ome/ZarrReader/issues/44) + +The current spec seems to acknowledge that this is to be expected to some degree in the following excerpt: + +> If only one multiscale is provided, use it. Otherwise, the user can choose by name, using the first multiscale as a fallback: +> +> ```python +> datasets = [] +> for named in multiscales: +> if named["name"] == "3D": +> datasets = [x["path"] for x in named["datasets"]] +> break +> if not datasets: +> # Use the first by default. Or perhaps choose based on chunk size. +> datasets = [x["path"] for x in multiscales[0]["datasets"]] +> ``` + + +This RFC aims to codify what already seems to be common practice by moving from a multiscales array to a single multisclaes object. This will reduce complexity for implementations. + + + +## Proposal + +The OME-Zarr metadata in a `zarr.json` file of a multiscale MUST contain a single `multiscale` object. This replaces the current `multiscales` array. + +Adapted example from the current spec: +```json +{ + "zarr_format": 3, + "node_type": "group", + "attributes": { + "ome": { + "version": "0.5", + "multiscale": { + "name": "example", + "axes": [ + { "name": "t", "type": "time", "unit": "millisecond" }, + { "name": "c", "type": "channel" }, + { "name": "z", "type": "space", "unit": "micrometer" }, + { "name": "y", "type": "space", "unit": "micrometer" }, + { "name": "x", "type": "space", "unit": "micrometer" } + ], + "datasets": [ + { + "path": "0", + "coordinateTransformations": [ + { + // the voxel size for the first scale level (0.5 micrometer) + "type": "scale", + "scale": [1.0, 1.0, 0.5, 0.5, 0.5] + } + ] + }, + { + "path": "1", + "coordinateTransformations": [ + { + // the voxel size for the second scale level (downscaled by a factor of 2 -> 1 micrometer) + "type": "scale", + "scale": [1.0, 1.0, 1.0, 1.0, 1.0] + } + ] + } + ], + "coordinateTransformations": [ + { + // the time unit (0.1 milliseconds), which is the same for each scale level + "type": "scale", + "scale": [0.1, 1.0, 1.0, 1.0, 1.0] + } + ] + } + } + } +} +``` + +For data that needs to have multiple multiscales, it is encouraged to store them in separate OME-Zarr datasets with their own OME-Zarr metadata. + + + +## Requirements + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [IETF RFC 2119](https://tools.ietf.org/html/rfc2119) + + + + +## Stakeholders + + + +The main stakeholders for this RFC are OME-Zarr tool developers and exsiting OME-Zarr image providers. Developers will have to update their implementations to account for the breaking change. Because this change is not backwards compatible, it will require a change to existing OME-Zarr images to make them compatible with this RFC. + +### Socialization + +* OME-NGFF hackathon Zurich 2024 +* [Github issue](https://github.com/ome/ngff/issues/205) + +## Implementation + +Many visualization and processing tools already expect only a single multiscale. +This proposal will reduce complexity for implementations. + +Examples of implementations that only work with a single multiscale: +- [neuroglancer](https://github.com/google/neuroglancer/blob/master/src/datasource/zarr/ome.ts#L265-L310) +- [vizarr](https://github.com/hms-dbmi/vizarr/blob/main/src/utils.ts#L88) +- [itk-vtk](https://github.com/Kitware/itk-vtk-viewer/blob/master/src/IO/ZarrMultiscaleSpatialImage.js#L173) +- [OMERO](https://github.com/ome/ZarrReader/issues/44) + + + + +## Drawbacks, risks, alternatives, and unknowns + +This is a breaking change with the typical drawbacks of breaking changes. + + + + + + + + + + + +## Compatibility + +This proposal is not backwards compatible and should be released in a new version of OME-Zarr. + + + + + + + + \ No newline at end of file From b572389bb559aea375dc8689769b3e2c0aaebe26 Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Thu, 12 Dec 2024 14:48:24 +0100 Subject: [PATCH 2/6] rm comments --- rfc/6/index.md | 240 +------------------------------------------------ 1 file changed, 1 insertion(+), 239 deletions(-) diff --git a/rfc/6/index.md b/rfc/6/index.md index 93b68528..a729da02 100644 --- a/rfc/6/index.md +++ b/rfc/6/index.md @@ -47,21 +47,6 @@ The current spec seems to acknowledge that this is to be expected to some degree This RFC aims to codify what already seems to be common practice by moving from a multiscales array to a single multisclaes object. This will reduce complexity for implementations. - - ## Proposal The OME-Zarr metadata in a `zarr.json` file of a multiscale MUST contain a single `multiscale` object. This replaces the current `multiscales` array. @@ -120,43 +105,14 @@ Adapted example from the current spec: For data that needs to have multiple multiscales, it is encouraged to store them in separate OME-Zarr datasets with their own OME-Zarr metadata. - ## Requirements The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [IETF RFC 2119](https://tools.ietf.org/html/rfc2119) - - ## Stakeholders - - The main stakeholders for this RFC are OME-Zarr tool developers and exsiting OME-Zarr image providers. Developers will have to update their implementations to account for the breaking change. Because this change is not backwards compatible, it will require a change to existing OME-Zarr images to make them compatible with this RFC. ### Socialization @@ -175,204 +131,10 @@ Examples of implementations that only work with a single multiscale: - [itk-vtk](https://github.com/Kitware/itk-vtk-viewer/blob/master/src/IO/ZarrMultiscaleSpatialImage.js#L173) - [OMERO](https://github.com/ome/ZarrReader/issues/44) - - - ## Drawbacks, risks, alternatives, and unknowns This is a breaking change with the typical drawbacks of breaking changes. - - - - - - - - - - ## Compatibility -This proposal is not backwards compatible and should be released in a new version of OME-Zarr. - - - - - - - - \ No newline at end of file +This proposal is not backwards compatible and should be released in a new version of OME-Zarr. \ No newline at end of file From cf8bdfa3fcd633a389b63bbc949082df6c07941c Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Thu, 12 Dec 2024 15:22:18 +0100 Subject: [PATCH 3/6] endorsers --- rfc/6/index.md | 44 ++++++++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/rfc/6/index.md b/rfc/6/index.md index a729da02..7f222d4f 100644 --- a/rfc/6/index.md +++ b/rfc/6/index.md @@ -1,16 +1,48 @@ # RFC-6: Multiscale -Turn the `multiscales` array into a single `multiscale` obejct. +Turn the `multiscales` array into a single `multiscale` object. ## Status This proposal is very early. Status: D1 -| Name | GitHub Handle | Institution | Date | Status | -| --------- | ------------- | ----------- | ---------- | ------------------------------------- | -| Norman Rzepka | [normanrz](https://github.com/normanrz) | scalable minds | 2024-12-03 | Author | -| David Stansby | [dstansby](https://github.com/dstansby) | University College London | 2024-12-03 | Author | +```{list-table} Record +:widths: 8, 20, 20, 20, 15, 10 +:header-rows: 1 +:stub-columns: 1 + +* - Role + - Name + - GitHub Handle + - Institution + - Date + - Status +* - Author + - Norman Rzepka + - @normanrz + - scalable minds + - 2024-12-03 + - +* - Author + - David Stansby + - @dstansby + - University College London + - 2024-12-03 + - +* - Endorser + - Davis Bennett + - @d-v-b + - + - 2024-12-12 + - +* - Endorser + - Will Moore + - @will-moore + - OME Dundee + - 2024-12-12 + - +``` ## Overview @@ -113,7 +145,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S ## Stakeholders -The main stakeholders for this RFC are OME-Zarr tool developers and exsiting OME-Zarr image providers. Developers will have to update their implementations to account for the breaking change. Because this change is not backwards compatible, it will require a change to existing OME-Zarr images to make them compatible with this RFC. +The main stakeholders for this RFC are OME-Zarr tool developers and existing OME-Zarr image providers. Developers will have to update their implementations to account for the breaking change. Because this change is not backwards compatible, it will require a change to existing OME-Zarr images to make them compatible with this RFC. ### Socialization From 24f3988fe4ec818fe5bcaca4c55fd8540920ccdc Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Tue, 17 Dec 2024 14:12:39 +0100 Subject: [PATCH 4/6] Update endorsers --- rfc/6/index.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rfc/6/index.md b/rfc/6/index.md index 7f222d4f..1589fedf 100644 --- a/rfc/6/index.md +++ b/rfc/6/index.md @@ -39,9 +39,15 @@ This proposal is very early. Status: D1 * - Endorser - Will Moore - @will-moore - - OME Dundee + - OME, University of Dundee - 2024-12-12 - +* - Endorser + - Lachlan Deakin + - LDeakin + - Australian National University + - 2024-12-17 + - ``` ## Overview @@ -169,4 +175,4 @@ This is a breaking change with the typical drawbacks of breaking changes. ## Compatibility -This proposal is not backwards compatible and should be released in a new version of OME-Zarr. \ No newline at end of file +This proposal is not backwards compatible and should be released in a new version of OME-Zarr. From cfc0fba8b58f14178185c28d418f7ce4ae4df68b Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Wed, 18 Dec 2024 18:05:22 +0100 Subject: [PATCH 5/6] Add endorser --- rfc/6/index.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/rfc/6/index.md b/rfc/6/index.md index 1589fedf..eee71605 100644 --- a/rfc/6/index.md +++ b/rfc/6/index.md @@ -48,6 +48,12 @@ This proposal is very early. Status: D1 - Australian National University - 2024-12-17 - +* - Endorser + - Joel Lüthi + - jluethi + - BioVisionCenter, University of Zurich + - 2024-12-18 + - ``` ## Overview From de91a65ce38fed607efadbe00a21eaea7a2f6a42 Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Wed, 8 Jan 2025 16:28:59 +0100 Subject: [PATCH 6/6] Update index.md --- rfc/6/index.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rfc/6/index.md b/rfc/6/index.md index eee71605..94274d00 100644 --- a/rfc/6/index.md +++ b/rfc/6/index.md @@ -44,16 +44,22 @@ This proposal is very early. Status: D1 - * - Endorser - Lachlan Deakin - - LDeakin + - @LDeakin - Australian National University - 2024-12-17 - * - Endorser - Joel Lüthi - - jluethi + - @jluethi - BioVisionCenter, University of Zurich - 2024-12-18 - +* - Endorser + - Eric Perlman + - @perlman + - + - 2024-12-18 + - ``` ## Overview