docs: add migration page to user guide #2596

jhamman · 2024-12-29T23:07:30Z

This PR adds a page describing the migration process for users upgrading to zarr 3.0.

Notes for reviewers:

This was split out from Doc updates for 3.0 release #2568 and builds on docs: split tutorial into multiple user guide sections #2589. Initially started in Doc/v3 migration guide #2102.

dstansby

This is a good start, but I think needs some major improvements and additions. In particular:

Removing the "getting ready" section, since there won't be a decent period of time between these docs existing and the final release for folks to prepare for v3.
Being more explicit about what's change and how to adapt existing code (I left inline comments for this)
Listing where keyword only arguments have been introduced (e.g., I noticed these in zarr.create), as this is a major breaking change where users will have to update code

I also compared the v2 and v3 API, and there's some major additions that need making to this guide so users can migrate:

Explaining how to adapt code that previously used zarr.hierarchy
Explaining how to migrate code that uses functions in zarr.storage that no longer exist in v3
Explaining what the alternative to zarr.n5 is
Explaining how to update code that imports from zarr.codecs
Explaining how to update code that uses zarr.attrs
Explaining how to update code that uses zarr.sync

docs/user-guide/v3_migration.rst

dstansby · 2024-12-30T17:37:53Z

docs/user-guide/v3_migration.rst

+  * :func:`zarr.copy`
+  * :func:`zarr.copy_all`


According to #2407 (comment), copy and copy_all aren't going to be ported at all?

Right, we haven't taken any action there yet though.

Is the plan to eventually implement copy and copy_all, or not? Currently this PR implies yes, but #2407 (comment) implies no.

d-v-b · 2025-01-01T17:00:23Z

In addition to a migration guide, what do people think about a "zarr v2 vs zarr v3" section that just describes how the semantics of v2 and v3 arrays are different? Not sure if that's in scope for this PR, or if it should be a separate effort, but it seems like the content is highly overlapping. I think the migration guide could link to the "v2 vs v3 concepts" section as needed when people have questions like "what's the logic behind with dimension separator handling zarr-python 3.xx"

jhamman · 2025-01-02T04:14:01Z

I'm now reasonably happy with this as a first cut and suggest we do what we can to get this online ASAP. There are gaps that can be filled in as we go but this is much better than the current alternative (no migration guide).

to @d-v-b's #2596 (comment), let's add a zarr v2 vs v3 page elsewhere. I will not try to squeeze that into this PR.

to @dstansby's comment about specific migration paths for stuff in zarr.hierarchy, zarr.attrs, etc. I believe the statement in the docs reflects our position:

The following internal modules are being removed or significantly changed. If your application relies on imports from any of the below modules, you will need to either a) modify your application to no longer rely on these imports or b) vendor the parts of the specific modules that you need.

I'm open to specific suggestions for how we could improve this wording.

docs/user-guide/v3_migration.rst

dstansby · 2025-01-02T14:13:11Z

docs/user-guide/v3_migration.rst

+Configuration
+~~~~~~~~~~~~~
+
+There is a new configuration system based on `donfig <https://github.com/pytroll/donfig>`_,


Ah I didn't realise - in that case, this should be expanded to explain how users migrate from the old zarr.config system to the new system.

dstansby · 2025-01-02T14:15:27Z

docs/user-guide/v3_migration.rst

+  * :func:`zarr.copy` (:issue:`2407`)
+  * :func:`zarr.copy_all` (:issue:`2407`)
+  * :func:`zarr.copy_store` (:issue:`2407`)
+  * :func:`zarr.Group.move` (:issue:`2108`)


It's a bit confusing that these say they're not ported, but are linked to functions in the API docs (which would imply they are implemented). Do they need removing from the API docs?

All of these exist as stubs in the library but currently raise a NotImplementedError

Should we remove them from the API docs then, so the docs don't give a misleading impression they exist?

No opinion on whether to keep them in the API docs.

docs/user-guide/v3_migration.rst

dstansby · 2025-01-02T14:23:13Z

docs/user-guide/v3_migration.rst

+  * :func:`zarr.copy`
+  * :func:`zarr.copy_all`


Is the plan to eventually implement copy and copy_all, or not? Currently this PR implies yes, but #2407 (comment) implies no.

Co-authored-by: David Stansby <[email protected]>

dstansby

🙌 this is looking great! I left another round of review comments, and a couple of more general things:

Near the top, language should be changed from (e.g.) "will be" to "has been", since these are changes that have now happened.
Since zarr.sync has gone with no replacement, there should be a short section on how to do process and thread syncrhonization in v3 (if it is possible - if not, just a short bit explaining it's not possible should be added)

docs/user-guide/v3_migration.rst

dstansby · 2025-01-02T14:54:47Z

docs/user-guide/v3_migration.rst

+Configuration
+~~~~~~~~~~~~~
+
+There is a new configuration system based on `donfig <https://github.com/pytroll/donfig>`_,


Is there an example of code that someone would write in v2, which they now have to use the configuration system for instead? I think should be the measure of if something should go in the migration guide or not.

dstansby · 2025-01-02T14:55:34Z

docs/user-guide/v3_migration.rst

+  * :func:`zarr.copy` (:issue:`2407`)
+  * :func:`zarr.copy_all` (:issue:`2407`)
+  * :func:`zarr.copy_store` (:issue:`2407`)
+  * :func:`zarr.Group.move` (:issue:`2108`)


Should we remove them from the API docs then, so the docs don't give a misleading impression they exist?

docs/user-guide/v3_migration.rst

jhamman · 2025-01-03T15:45:49Z

I'm going to wrap this up today. Fixing conflicts and resolving the final comments.

…into docs/3.0-migration

…rs/zarr-python into docs/3.0-migration

dstansby

These previous comments still need resolving somehow

dstansby · 2025-01-03T19:51:32Z

docs/user-guide/v3_migration.rst

+The goals described above necessitated some breaking changes to the API (hence the
+major version update), but where possible we have maintained backwards compatibility
+in the most widely used parts of the API. This in the :class:`zarr.Array` and
+:class:`zarr.Group` classes and the "top-level API" (e.g. :func:`zarr.open_array` and
+:func:`zarr.open_group`).


Suggested change

The goals described above necessitated some breaking changes to the API (hence the

major version update), but where possible we have maintained backwards compatibility

in the most widely used parts of the API. This in the :class:`zarr.Array` and

:class:`zarr.Group` classes and the "top-level API" (e.g. :func:`zarr.open_array` and

:func:`zarr.open_group`).

The goals described above necessitated some breaking changes to the API (hence the

major version update), but where possible we have maintained backwards compatibility

with functions in the top-level ``zarr`` namespace (e.g. :func:`zarr.open_array` and :func:`zarr.open_group`).

We have also attempted to maintain backward compatibility for the Group/Array classes. Neither the top level API or the Array/Group classes hit 100% backward compatibility.

dcherian

LGTM. Didn't verify the correctness, just reviewed the text.

responded to all outstanding comments

jhamman added 2 commits December 29, 2024 13:56

docs: split tutorial into multiple user guide sections

01bc352

docs: add migration page to user guide

639eb54

jhamman added the documentation Improvements to the documentation label Dec 29, 2024

jhamman added this to the 3.0.0 milestone Dec 30, 2024

dstansby requested changes Dec 30, 2024

View reviewed changes

This was referenced Dec 30, 2024

The future of chunk_store in Zarr #2495

Open

[v3] zarr.array from from an existing zarr.Array #2410

Open

dstansby reviewed Dec 30, 2024

View reviewed changes

rev migration guide again

28829f8

fixup

9427b10

dstansby reviewed Jan 2, 2025

View reviewed changes

Apply suggestions from code review

175495b

Co-authored-by: David Stansby <[email protected]>

dstansby requested changes Jan 2, 2025

View reviewed changes

dstansby mentioned this pull request Jan 2, 2025

Document deprecated or non-working function arguments #2620

Open

Base automatically changed from docs/3.0-user-guide to main January 3, 2025 14:31

jhamman added 3 commits January 3, 2025 09:12

Merge branch 'main' of https://github.com/zarr-developers/zarr-python …

8f2d18f

…into docs/3.0-migration

Merge branch 'docs/3.0-migration' of https://github.com/zarr-develope…

455f853

…rs/zarr-python into docs/3.0-migration

update migration guide

b85fa9b

jhamman requested a review from dstansby January 3, 2025 17:26

update migration guide

edb65da

dstansby previously requested changes Jan 3, 2025

View reviewed changes

fixup

dff0799

dstansby reviewed Jan 3, 2025

View reviewed changes

removed config section

b31bc7a

jhamman requested a review from dstansby January 3, 2025 20:42

dcherian approved these changes Jan 3, 2025

View reviewed changes

Merge branch 'main' into docs/3.0-migration

c1f95fa

jhamman enabled auto-merge (squash) January 3, 2025 21:28

jhamman merged commit eef18f3 into main Jan 3, 2025
33 checks passed

jhamman mentioned this pull request Jan 3, 2025

[V3] v2 -> v3 data migration #1798

Open

jhamman deleted the docs/3.0-migration branch January 4, 2025 01:19

This was referenced Jan 4, 2025

Guide to converting code from zarr-python v2 to zarr-python v3 #2564

Closed

Stores are no longer MutableMappings #2023

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add migration page to user guide #2596

docs: add migration page to user guide #2596

jhamman commented Dec 29, 2024 •

edited

Loading

dstansby left a comment

dstansby Dec 30, 2024

jhamman Jan 2, 2025

dstansby Jan 2, 2025

d-v-b commented Jan 1, 2025

jhamman commented Jan 2, 2025

dstansby Jan 2, 2025

dstansby Jan 2, 2025

jhamman Jan 2, 2025

dstansby Jan 2, 2025

jhamman Jan 3, 2025

dstansby Jan 2, 2025

dstansby left a comment

dstansby Jan 2, 2025

dstansby Jan 2, 2025

jhamman commented Jan 3, 2025

dstansby left a comment

dstansby Jan 3, 2025

jhamman Jan 3, 2025

dcherian left a comment

docs: add migration page to user guide #2596

docs: add migration page to user guide #2596

Conversation

jhamman commented Dec 29, 2024 • edited Loading

dstansby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d-v-b commented Jan 1, 2025

jhamman commented Jan 2, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dstansby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhamman commented Jan 3, 2025

dstansby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcherian left a comment

Choose a reason for hiding this comment

jhamman commented Dec 29, 2024 •

edited

Loading