Skip to content

Conversation

@mkitti
Copy link
Contributor

@mkitti mkitti commented Oct 16, 2025

This document outlines the suffix chunk key encoding for Zarr v3, which allows appending user-defined file extensions to chunk keys, enhancing interoperability with traditional file-based tools.

This document outlines the `suffix` chunk key encoding for Zarr v3, which allows appending user-defined file extensions to chunk keys, enhancing interoperability with traditional file-based tools.
The configuration for this encoding is a JSON object with one required and one optional member.

* `"suffix"`: **(Required)** A string that will be appended to the encoded chunk key.
* `"base-encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used.
Copy link
Member

@normanrz normanrz Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* `"base-encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used.
* `"base_encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used.

Zarr uses snake_case in the JSON metadata.

I would make this required, because it is also required in the top-level array metadata. It helps implementations to make this explicit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, should be required.

@LDeakin
Copy link
Member

LDeakin commented Oct 16, 2025

Cool, I wrote something similar here https://docs.rs/zarrs/latest/zarrs/array/chunk_key_encoding/default_suffix/struct.DefaultSuffixChunkKeyEncoding.html, but this is much better because it can compose over other chunk grids.

If omitted, the store's default chunk key encoding is used.

Stores don't have a default chunk key encoding?

Combining the two comments:

* Change to snake case.
* Refer to the default chunk key encoding.
@d-v-b
Copy link
Contributor

d-v-b commented Oct 16, 2025

i love this! do we care at all about suffixes that ascend the directory tree on local file systems, e.g. `"../foo.tif"?

@mkitti
Copy link
Contributor Author

mkitti commented Oct 16, 2025

i love this! do we care at all about suffixes that ascend the directory tree on local file systems, e.g. `"../foo.tif"?

I think the base key would need to be a directory first for this to work on a local file system. I'm not sure how .. works on S3.

I'm thinking that we do not care about this. If something needs to be contained within a directory, it would not be Zarr's responsibilty to maintain that containment. Someone would probably want to use a chroot jail.

@mkitti
Copy link
Contributor Author

mkitti commented Oct 16, 2025

A complimentary chunk key encoding might be something to ascend directories to assist in mapping from a high dimensionality arrays to a squeezed lower dimensionality array.

Imagine an array with dimensions (256, 1024, 1024, 1, 1) in ZYXCT order. We might want a way to map a 3D XYZ array to chunks with a 5D chunk layout or a 5D array mapped to a 3D chunk layout. I suspect this is probably a separate chunk key encoding and pull request.

@LDeakin
Copy link
Member

LDeakin commented Oct 16, 2025

Implemented here: zarrs/zarrs#286

I question the base-encoding being optional given that chunk_key_encoding is itself mandatory. Despite the name, the default chunk key encoding is not default and must be explicitly listed.

@mkitti
Copy link
Contributor Author

mkitti commented Oct 16, 2025

base-encoding is now a required configuration parameter as of 08d9470

Copy link
Member

@normanrz normanrz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR meets the requirements for being merged. However, there still seems to be some discussion.
Let me know when you're ready to.

mkitti and others added 4 commits October 20, 2025 14:28
Co-authored-by: Norman Rzepka <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>
@mkitti
Copy link
Contributor Author

mkitti commented Oct 20, 2025

Should I put something in place of "Identifier: (A unique URI to be assigned upon formal adoption)" ?

@mkitti mkitti mentioned this pull request Oct 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants