Skip to content

Use zarr-python default open mode when opening arrays #10430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions xarray/backends/zarr.py
Original file line number Diff line number Diff line change
@@ -639,7 +639,7 @@ def open_store(
def open_group(
cls,
store,
mode: ZarrWriteModes = "r",
mode: ZarrWriteModes | None = None,
synchronizer=None,
group=None,
consolidated=False,
@@ -1561,7 +1561,7 @@ def open_dataset(
use_cftime=None,
decode_timedelta=None,
group=None,
mode="r",
mode=None,
synchronizer=None,
consolidated=None,
chunk_store=None,
@@ -1746,6 +1746,9 @@ def _get_open_params(
synchronizer=synchronizer,
path=group,
)
if mode is None:
# Let zarr-python use its default open mode
open_kwargs.pop("mode")
open_kwargs["storage_options"] = storage_options

zarr_format = _handle_zarr_version_or_format(

Unchanged files with check annotations Beta

expected = np_arr + 7
actual = xp_arr + 7
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 35 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 35 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 35 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 35 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 35 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 35 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
def test_aggregation(arrays: tuple[xr.DataArray, xr.DataArray]) -> None:
expected = np_arr.sum()
actual = xp_arr.sum()
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 43 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_aggregation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 43 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_aggregation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 43 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_aggregation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 43 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_aggregation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 43 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_aggregation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 43 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_aggregation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
def test_aggregation_skipna(arrays) -> None:
expected = np_arr.sum(skipna=False)
actual = xp_arr.sum(skipna=False)
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 51 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_aggregation_skipna TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 51 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_aggregation_skipna TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 51 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_aggregation_skipna TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 51 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_aggregation_skipna TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 51 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_aggregation_skipna TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 51 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_aggregation_skipna TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
# casting nan warns
actual = xp_arr.astype(xp.int64)
assert actual.dtype == xp.int64
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 62 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_astype TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 62 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_astype TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 62 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_astype TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 62 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_astype TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 62 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_astype TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 62 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_astype TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
def test_broadcast(arrays: tuple[xr.DataArray, xr.DataArray]) -> None:
assert len(actual) == len(expected)
for a, e in zip(actual, expected, strict=True):
assert isinstance(a.data, Array)
assert_equal(a, e)

Check failure on line 75 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_broadcast TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 75 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_broadcast TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 75 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_broadcast TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 75 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_broadcast TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 75 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_broadcast TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 75 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_broadcast TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
def test_broadcast_during_arithmetic(arrays: tuple[xr.DataArray, xr.DataArray]) -> None:
expected = np_arr * np_arr2
actual = xp_arr * xp_arr2
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 86 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_broadcast_during_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 86 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_broadcast_during_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 86 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_broadcast_during_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 86 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_broadcast_during_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 86 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_broadcast_during_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 86 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_broadcast_during_arithmetic TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
expected = np_arr2 * np_arr
actual = xp_arr2 * xp_arr
expected = xr.concat((np_arr, np_arr), dim="x")
actual = xr.concat((xp_arr, xp_arr), dim="x")
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 99 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_concat TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 99 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_concat TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 99 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_concat TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 99 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_concat TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 99 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_concat TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 99 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_concat TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
def test_indexing(arrays: tuple[xr.DataArray, xr.DataArray]) -> None:
expected = np_arr[:, 0]
actual = xp_arr[:, 0]
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 107 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_indexing TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 107 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_indexing TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 107 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_indexing TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 107 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_indexing TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 107 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_indexing TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 107 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_indexing TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
def test_properties(arrays: tuple[xr.DataArray, xr.DataArray]) -> None:
expected = np_arr.transpose()
actual = xp_arr.transpose()
assert isinstance(actual.data, Array)
assert_equal(actual, expected)

Check failure on line 123 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

test_reorganizing_operation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 123 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

test_reorganizing_operation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 123 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.10

test_reorganizing_operation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 123 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.10

test_reorganizing_operation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 123 in xarray/tests/test_array_api.py

GitHub Actions / ubuntu-latest py3.13

test_reorganizing_operation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>

Check failure on line 123 in xarray/tests/test_array_api.py

GitHub Actions / macos-latest py3.13

test_reorganizing_operation TypeError: Expected Array or Python scalar; got <class 'numpy.ndarray'>
def test_stack(arrays: tuple[xr.DataArray, xr.DataArray]) -> None:
assert_identical(ds, expected)
def test_non_existent_store(self) -> None:
with pytest.raises(

Check failure on line 2381 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.12 all-but-dask

TestZarrDictStore.test_non_existent_store[3] Failed: DID NOT RAISE <class 'FileNotFoundError'>

Check failure on line 2381 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

TestZarrWriteEmpty.test_non_existent_store[2] Failed: DID NOT RAISE <class 'FileNotFoundError'>

Check failure on line 2381 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

TestZarrDirectoryStore.test_non_existent_store[2] Failed: DID NOT RAISE <class 'FileNotFoundError'>

Check failure on line 2381 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

TestZarrDictStore.test_non_existent_store[2] Failed: DID NOT RAISE <class 'FileNotFoundError'>

Check failure on line 2381 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.13 all-but-numba

TestZarrWriteEmpty.test_non_existent_store[2] Failed: DID NOT RAISE <class 'FileNotFoundError'>

Check failure on line 2381 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.13

TestZarrWriteEmpty.test_non_existent_store[2] Failed: DID NOT RAISE <class 'FileNotFoundError'>

Check failure on line 2381 in xarray/tests/test_backends.py

GitHub Actions / macos-latest py3.13

TestZarrWriteEmpty.test_non_existent_store[2] Failed: DID NOT RAISE <class 'FileNotFoundError'>
FileNotFoundError, match="(No such file or directory|Unable to find group)"
):
xr.open_zarr(f"{uuid.uuid4()}")

Check failure on line 2384 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.10

TestZarrWriteEmpty.test_non_existent_store[2] RuntimeWarning: Failed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider: 1. Consolidating metadata in this existing store with zarr.consolidate_metadata(). 2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or 3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.

Check failure on line 2384 in xarray/tests/test_backends.py

GitHub Actions / macos-latest py3.10

TestZarrWriteEmpty.test_non_existent_store[2] RuntimeWarning: Failed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider: 1. Consolidating metadata in this existing store with zarr.consolidate_metadata(). 2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or 3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.
@pytest.mark.skipif(has_zarr_v3, reason="chunk_store not implemented in zarr v3")
def test_with_chunkstore(self) -> None:
def test_open_nczarr(self) -> None:
with create_tmp_file(suffix=".zarr") as tmp:
expected = self._create_nczarr(tmp)
actual = xr.open_zarr(tmp, consolidated=False)

Check failure on line 6320 in xarray/tests/test_backends.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

TestNCZarr.test_open_nczarr KeyError: 'Zarr object is missing the attribute `_ARRAY_DIMENSIONS`, which is required for xarray to determine variable dimensions.'
assert_identical(expected, actual)
def test_overwriting_nczarr(self) -> None:
# otherwise numpy unsigned ints will silently cast to the signed counterpart
fill_value = fill_value.item()
# passes if provided fill value fits in encoded on-disk type
new_fill = encoded_dtype.type(fill_value)

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).

Check warning on line 237 in xarray/coding/variables.py

GitHub Actions / ubuntu-latest py3.10 min-all-deps

NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future. For the old behavior, usually: np.array(value).astype(dtype)` will give the desired result (the cast overflows).
except OverflowError:
encoded_kind_str = "signed" if encoded_dtype.kind == "i" else "unsigned"
warnings.warn(
if xp == np:
# numpy currently doesn't have a astype:
return data.astype(dtype, **kwargs)

Check warning on line 253 in xarray/core/duck_array_ops.py

GitHub Actions / macos-latest py3.10

invalid value encountered in cast

Check warning on line 253 in xarray/core/duck_array_ops.py

GitHub Actions / macos-latest py3.10

invalid value encountered in cast
return xp.astype(data, dtype, **kwargs)