Tracking 3.0 Release Blockers

[jhamman]

We are now in the last mile of the 3.0 refactor. This issue is tracking what we are considering release blockers: There is also a milestone for After 3.0.0 where we are putting tickets that can come later.

Docs

• [v3] Documentation: Update developer docs #1765
• [v3] Documentation: Update API docs #1766
• [v3] Documentation: update tutorial docs #1767
• [v3] Documentation: How to extend Zarr #1770
• [V3] v2 -> v3 data migration #1798
• Add documentation for zarr.array #2245
• Add documentation for zarr.group #2246
• Add documentation for zarr.api.asynchronous #2247
• Add documentation for zarr.api.synchronous #2248
• Add documentation for zarr.asyncgroup #2252
• Add documentation for AsyncArray #2253

Array

• [v3] Support numcodecs via entrypoints  #1750
• [v3] Support zero-sized arrays #1977
• [v3] array written by tensorstore returns all nulls #2029
• [v3] default compressor / codec pipeline #2267
• [v3] V2 Codec pipeline is not consistent with legacy usage of filters #2325
• [v3] AttributeError: 'Array' object has no attribute 'append' #2404
• [v3] support write_empty_chunks #2409

Group

• [v3] reimplement / develop Group.info and Array.info properties #2135
• [v3] Group.{zeros, ones, etc.} should accept positional name arguments #2402
• [v3] Implement Group.tree #2406
• Rename exists_ok to overwrite #2527

Top level API

• [v3] array creation: codecs #1943
• [v3] zarr.save - AttributeError: 'str' object has no attribute 'shape' #2403
• [v3] zarr.array silently ignores order kwarg #2408
• [v3] It is possible to create v3 data with features not currently written into the spec (dtypes, codecs, etc.) #2411

Store API

• [v3] Finalize store API #1755
• [v3] deleting a group does not delete subgroups #2191
• RemoteStore doesn't strip protocol correctly, breaking list operations #2342
• [v3] Re-opening an existing RemoteStore V3 group causes the first child to be deleted #2359

Testing / CI

• [v3] CI ENH: Add testing with the development version of dependencies #1496
• [v3] Test environments and CI #1648
• [v3] port zarr-python 2 test suite to v3 branch #1759
• [v3] Windows support #2438

This is non exhaustive but these are the major known issues. Feel free to post additional tickets and we can evaluate if they are really blockers.

[DimitriPapadopoulos]

I would suggest:

• Inconsistent overridden method #2377
• Cyclic imports #2380
• Review TODOs before v3? #2419

[will-moore]

Hi,
Thanks for all the work towards v3.
I wanted to check if this is a blocker / existing issue etc or if I should create one...?
This is using main branch 680142f :

import zarr
from zarr.storage import LocalStore

path = "root.zarr"
fmt = 2
store = LocalStore(path, mode="w")
root = zarr.group(store=store, zarr_format=fmt)

root_group = zarr.open(f"{path}", zarr_format=fmt)

Gives me:

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 48, in <module>
    root_group = zarr.open(f"{path}", zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

[d-v-b]

Hi, Thanks for all the work towards v3. I wanted to check if this is a blocker / existing issue etc or if I should create one...? This is using main branch 680142f :

import zarr
from zarr.storage import LocalStore

path = "root.zarr"
fmt = 2
store = LocalStore(path, mode="w")
root = zarr.group(store=store, zarr_format=fmt)

root_group = zarr.open(f"{path}", zarr_format=fmt)

Gives me:

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 48, in <module>
    root_group = zarr.open(f"{path}", zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

It looks like you are not passing the store instance to zarr.open. Does
root_group = zarr.open(store, path=path, zarr_format=fmt) work?

[will-moore]

If I pass in both the store and the path, it fails with FileNotFoundError: file://root.zarr/root.zarr, so it looks like I don't need both? It also seems I want to create a store with read mode. But this is still failing (but works with fmt = 3)...

read_store = LocalStore(path)
root_group = zarr.open(read_store, zarr_format=fmt)

FileNotFoundError: file://root.zarr

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 49, in <module>
    root_group = zarr.open(read_store, zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

[d-v-b]

ugh, this flow should definitely be super easy, sorry that it's not working. If I have some time tonight I will dig into this.

[will-moore]

@d-v-b with this change, that test passes for me and fixes about 100 failing tests at ome/ome-zarr-py#404

diff --git a/src/zarr/api/asynchronous.py b/src/zarr/api/asynchronous.py
index 40f9b8d5..8dc69250 100644
--- a/src/zarr/api/asynchronous.py
+++ b/src/zarr/api/asynchronous.py
@@ -306,7 +306,7 @@ async def open(
 
     try:
         return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
-    except (KeyError, NodeTypeValidationError):
+    except (KeyError, NodeTypeValidationError, FileNotFoundError):
         # KeyError for a missing key
         # NodeTypeValidationError for failing to parse node metadata as an array when it's
         # actually a group

[maxrjones]

Thanks for all your work on Zarr Python 3.0!

I know this is truly the last quarter mile (or km) before v3.0 and apologize for being so last minute, but since this is the last shot for potential user facing API changes I wanted to promote #2437 as something to settle before the release. In particular, I think it's important to make an explicit decision on whether the tuple(int, int) representation will remain (start, length) vs (start, stop) and whether suffix requests should be denoted by negative integers vs. a typed dict. If people are open to following the same approach as Kyle proposes in #2437 (comment), I would be glad to try implementing this suggestion over the holidays to ensure there are no severe memory/time consequences. I'm +1 on Kyle's idea based on the number of people who have misunderstood the current semantics and the apparent negligible performance impact of more intuitive typing.