Zarr v3 #404

will-moore · 2024-11-07T13:58:05Z

This updates ome-zarr-py to use zarr-python v3 but doesn't include support for writing Zarr v3 (OME-Zarr v0.5).
However, it does add support for reading OME-Zarr v0.5 (e.g. for use by napari-ome-zarr).

There were 4 tests that I have disabled as I couldn't understand exactly why they were failing.

 = 4 failed, 384 passed, 3 skipped, 6 xfailed, 1832 warnings in 86.38s (0:01:26) =

4 failing tests are:

TestWriter.test_writer[3D-scale-True-from_array-V01]
TestWriter.test_writer[3D-scale-True-from_array-V02]
TestWriter.test_writer[3D-scale-True-from_array-V03]
TestWriter.test_writer[3D-scale-True-from_array-V04]

          # check memory is contiguous, if so flatten
          if arr.flags.c_contiguous or arr.flags.f_contiguous:
              if flatten:
                  # can flatten without copy
                  arr = arr.reshape(-1, order="A")
          else:
  >           raise ValueError("an array with contiguous memory is required")
  E           ValueError: an array with contiguous memory is required
  
  .tox/py311/lib/python3.11/site-packages/numcodecs/compat.py:113: ValueError

These tests were passing before
* 109f71f6 (HEAD) Refactors v2 codec handling (#2425) (zarr-developers/zarr-python#2425)

In several places I've hard-coded zarr_version=2. This works because you can do zarr.open_group(store=self.__store, path="/", zarr_version=2) in parse_url() to create a group to write into, or find a group to read from without specifying whether you expect the group to already exist.

Without the zarr_version=2, zarr will create zarr.json if the mode of the store is w.

NB: I have another PR (in progress) to add support for writing OME-Zarr v0.5 at #413

Fixes #404

Fixes TypeError: Unsupported type for store_like: 'LocalPath'

…de.py v2

for more information, see https://pre-commit.ci

will-moore · 2024-11-07T14:38:37Z

Zarr v3 is not supported on python 3.9 or 3.10. I'll remove them from the build...

Build on python 3.12 is failing tests with:

  = 106 failed, 282 passed, 3 skipped, 6 xfailed, 2270 warnings in 75.86s (0:01:15) =

imagesc-bot · 2024-11-07T14:41:20Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ome-zarr-py-development-status/104671/4

joshmoore · 2024-11-26T23:05:07Z

@will-moore: what's the next step here?

will-moore · 2024-11-27T11:46:25Z

I have 4 failing tests which I don't really understand:

FAILED tests/test_writer.py::TestWriter::test_writer[3D-scale-True-from_array-V01]
  FAILED tests/test_writer.py::TestWriter::test_writer[3D-scale-True-from_array-V02]
  FAILED tests/test_writer.py::TestWriter::test_writer[3D-scale-True-from_array-V03]
  FAILED tests/test_writer.py::TestWriter::test_writer[3D-scale-True-from_array-V04]
  = 4 failed, 384 passed, 3 skipped, 6 xfailed, 1832 warnings in 86.38s (0:01:26) =

All are from .tox/py311/lib/python3.11/site-packages/numcodecs/compat.py:151: in ensure_contiguous_ndarray ensure_contiguous_ndarray_like(buf, max_buffer_size=max_buffer_size, flatten=flatten)

raise ValueError("an array with contiguous memory is required")

But this PR was really just an investigation into what were the API changes of zarr-python v3 and what is needed for us to consume it.
With this PR, we can now read and write zarr v2 data, but this is hard-coded in a couple of places so we might want a different approach.
It may even "fix" the issue at #343 where we enforce store uses "/" dimension separator for v0.2-v0.4 data.

This PR has no support for OME-Zarr v0.5 and I'm not sure what that would look like yet. It may be that there's no value in this PR on it's own, without v0.5 support.
I could(should) continue to look at what would be needed for OME-Zarr v0.5 read/write?

Any and all reviews and feedback on this PR as it stands would be great to hear, especially an idea about the underlying cause of those failing tests.

We also have to wait for zarr-python v3 release!

will-moore · 2024-12-09T12:13:45Z

With that previous commit (even without a FormatV05) reading works, including via napari-ome-zarr:

$ napari --plugin napari-ome-zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr

will-moore · 2024-12-09T12:47:49Z

Plates also looking good:

$ napari --plugin napari-ome-zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0010/76-45.ome.zarr

cc @joshmoore

…3.0.0-beta.3

codecov · 2024-12-09T12:57:05Z

Codecov Report

Attention: Patch coverage is 86.36364% with 6 lines in your changes missing coverage. Please review.

Project coverage is 85.14%. Comparing base (c99f0b3) to head (e021c13).

Files with missing lines	Patch %	Lines
ome_zarr/io.py	79.31%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #404      +/-   ##
==========================================
- Coverage   85.45%   85.14%   -0.32%     
==========================================
  Files          13       13              
  Lines        1540     1548       +8     
==========================================
+ Hits         1316     1318       +2     
- Misses        224      230       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…_array

will-moore · 2024-12-17T14:20:15Z

@joshmoore can you have a look at this now? See updated description. Thx.

joshmoore

Few general comments and one or two more pointed ones (especially around dimension separator). In general, though, I think this is looking good. Thanks, @will-moore.

joshmoore · 2024-12-17T19:57:00Z

ome_zarr/format.py

@@ -60,7 +60,7 @@ def matches(self, metadata: dict) -> bool:  # pragma: no cover
        raise NotImplementedError()

    @abstractmethod
-    def init_store(self, path: str, mode: str = "r") -> FSStore:
+    def init_store(self, path: str, mode: str = "r") -> RemoteStore:


should this be | LocalStore as well?

joshmoore · 2024-12-17T19:58:38Z

ome_zarr/io.py

+        self.__store: RemoteStore = (
+            path
+            if isinstance(path, RemoteStore)
+            else loader.init_store(self.__path, mode)


would also checking for LocalStore here work?

joshmoore · 2024-12-17T20:00:15Z

ome_zarr/io.py

@@ -104,7 +132,7 @@ def path(self) -> str:
        return self.__path

    @property
-    def store(self) -> FSStore:
+    def store(self) -> RemoteStore:


Unsure why this isn't also | LocalStore

joshmoore · 2024-12-17T20:01:12Z

ome_zarr/io.py

@@ -67,16 +70,41 @@ def __init_metadata(self) -> None:
        """
        Load the Zarr metadata files for the given location.
        """
-        self.zarray: JSONDict = self.get_json(".zarray")


can get_json() now be deprecated?

joshmoore · 2024-12-17T20:03:01Z

ome_zarr/writer.py

-                compressor=options.get("compressor", zarr.storage.default_compressor),
-                dimension_separator=group._store._dimension_separator,
+                compressor=options.pop(
+                    "compressor", Blosc(cname="zstd", clevel=5, shuffle=Blosc.SHUFFLE)


This bit has been duplicated several times. Might be worth putting it in a single (overridable location) so that the TODO can be more easily fixed here but also be end users who don't want to wait for a release.

joshmoore · 2024-12-17T20:03:36Z

tests/data/v2/0/.zarray

@@ -13,6 +13,7 @@
        "id": "blosc",
        "shuffle": 1
    },
+    "dimension_separator": "/",


This surprises me. We really should be able to pass with and without this metadata. (Two tests?)

joshmoore · 2024-12-17T20:04:36Z

tests/test_writer.py

+            and storage_options_list
+            and array_constructor == da.array
+        ):
+            return


use pytest's skip functionality here so this is clearer in the output

Otherwise, I don't immediately have an idea what's going on.

joshmoore · 2024-12-18T08:10:31Z

Also just tried out the branch to see if it fixes #231 and 👍

(Previously pasted error was from an older version of dask)

joshmoore · 2024-12-18T10:51:16Z

Updated with #346.

will-moore and others added 11 commits October 26, 2024 23:03

Basic read example working (no labels)

b0d6b3d

cli_tests passing

da8c32f

Passing all 6 test_io.py

19b89a8

Passing tests/test_io.py and test_node.py

a954161

Include dtype in group.create_array()

80f6e01

Uncomment labels spec. Fixes test_ome_zarr.py download

e568911

Fix test_scaler

b49ecc8

Fixes TypeError: Unsupported type for store_like: 'LocalPath'

Add dimension_separator to existing v2 data .zarray to fix test_upgra…

18abe02

…de.py v2

Fixed test_write_image_dask

86142c3

Pin zarr==v3.0.0-beta.1

31584bf

[pre-commit.ci] auto fixes from pre-commit.com hooks

daa3546

for more information, see https://pre-commit.ci

Remove python 3.9 and 3.10 from build.yml

fa29ccc

will-moore added 7 commits November 7, 2024 14:54

Remove unused imports

8fc02b4

remove fsspec from .isort.cfg

29890b8

mypy fix

35bc979

Use Blosc compression by default

75ba690

Black formatting fixes

52aceb0

Use group.array_values() for iterating arrays

55d4ba9

Use zarr_format=2 for zarr.open() in test_writer.py

0ea21bc

This was referenced Nov 11, 2024

Tracking 3.0 Release Blockers zarr-developers/zarr-python#2412

Open

V2 group read file not found error zarr-developers/zarr-python#2480

Closed

will-moore added 6 commits November 12, 2024 10:21

Fix return type RemoteStore | LocalStore

7fc113b

Support reading of Zarr v3 data

94f7ace

Hard-code zarr_version=2 in parse_url()

d140c6d

Use read_only instead of mode when creating Stores

f7b5f98

Pin zarr-python to specific commit on main branch

c527c77

Fix test_write_image_compressed

d8d5378

Support READING of zarr v3 data

2138160

Merge remote-tracking branch 'origin/master' into zarr_v3. Use zarr v…

af2648d

…3.0.0-beta.3

will-moore added 3 commits December 9, 2024 13:10

Check that PR is green IF we skip test_writer with 3D-scale-True-from…

1ea9e1a

…_array

Bump dependencies including zarr==v3.0.0-beta.3 in docs/requirements.txt

7754774

Specify python 3.12 in .readthedocs.yml

499531f

will-moore mentioned this pull request Dec 13, 2024

Comparison of OME-Zarr libs #407

Open

will-moore force-pushed the zarr_v3 branch from 7134cb4 to 499531f Compare December 17, 2024 14:02

Merge remote-tracking branch 'origin/master' into zarr_v3

c0fe50d

will-moore requested a review from joshmoore December 17, 2024 14:19

will-moore mentioned this pull request Dec 17, 2024

Ome zarr v0.5 writing #413

Open

joshmoore reviewed Dec 17, 2024

View reviewed changes

joshmoore mentioned this pull request Dec 18, 2024

ome_zarr download uses wrong dimension_separator #231

Open

Merge remote-tracking branch 'origin/master' into zarr_v3

e021c13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zarr v3 #404

Zarr v3 #404

will-moore commented Nov 7, 2024 •

edited by joshmoore

Loading

will-moore commented Nov 7, 2024

imagesc-bot commented Nov 7, 2024

joshmoore commented Nov 26, 2024

will-moore commented Nov 27, 2024

will-moore commented Dec 9, 2024 •

edited

Loading

will-moore commented Dec 9, 2024

codecov bot commented Dec 9, 2024 •

edited

Loading

will-moore commented Dec 17, 2024

joshmoore left a comment

joshmoore Dec 17, 2024

joshmoore Dec 17, 2024

joshmoore Dec 17, 2024

joshmoore Dec 17, 2024

joshmoore Dec 17, 2024

joshmoore Dec 17, 2024

joshmoore Dec 17, 2024

joshmoore commented Dec 18, 2024 •

edited

Loading

joshmoore commented Dec 18, 2024

Zarr v3 #404

Are you sure you want to change the base?

Zarr v3 #404

Conversation

will-moore commented Nov 7, 2024 • edited by joshmoore Loading

will-moore commented Nov 7, 2024

imagesc-bot commented Nov 7, 2024

joshmoore commented Nov 26, 2024

will-moore commented Nov 27, 2024

will-moore commented Dec 9, 2024 • edited Loading

will-moore commented Dec 9, 2024

codecov bot commented Dec 9, 2024 • edited Loading

Codecov Report

will-moore commented Dec 17, 2024

joshmoore left a comment

Choose a reason for hiding this comment

joshmoore Dec 17, 2024

Choose a reason for hiding this comment

joshmoore Dec 17, 2024

Choose a reason for hiding this comment

joshmoore Dec 17, 2024

Choose a reason for hiding this comment

joshmoore Dec 17, 2024

Choose a reason for hiding this comment

joshmoore Dec 17, 2024

Choose a reason for hiding this comment

joshmoore Dec 17, 2024

Choose a reason for hiding this comment

joshmoore Dec 17, 2024

Choose a reason for hiding this comment

joshmoore commented Dec 18, 2024 • edited Loading

joshmoore commented Dec 18, 2024

will-moore commented Nov 7, 2024 •

edited by joshmoore

Loading

will-moore commented Dec 9, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

joshmoore commented Dec 18, 2024 •

edited

Loading