-
Notifications
You must be signed in to change notification settings - Fork 10
feat: Add suffix chunk key encoding
#28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat: Add suffix chunk key encoding
#28
Conversation
This document outlines the `suffix` chunk key encoding for Zarr v3, which allows appending user-defined file extensions to chunk keys, enhancing interoperability with traditional file-based tools.
chunk-key-encodings/suffix/README.md
Outdated
| The configuration for this encoding is a JSON object with one required and one optional member. | ||
|
|
||
| * `"suffix"`: **(Required)** A string that will be appended to the encoded chunk key. | ||
| * `"base-encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| * `"base-encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used. | |
| * `"base_encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used. |
Zarr uses snake_case in the JSON metadata.
I would make this required, because it is also required in the top-level array metadata. It helps implementations to make this explicit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, should be required.
|
Cool, I wrote something similar here https://docs.rs/zarrs/latest/zarrs/array/chunk_key_encoding/default_suffix/struct.DefaultSuffixChunkKeyEncoding.html, but this is much better because it can compose over other chunk grids.
Stores don't have a default chunk key encoding? |
Combining the two comments: * Change to snake case. * Refer to the default chunk key encoding.
|
i love this! do we care at all about suffixes that ascend the directory tree on local file systems, e.g. `"../foo.tif"? |
I think the base key would need to be a directory first for this to work on a local file system. I'm not sure how I'm thinking that we do not care about this. If something needs to be contained within a directory, it would not be Zarr's responsibilty to maintain that containment. Someone would probably want to use a chroot jail. |
|
A complimentary chunk key encoding might be something to ascend directories to assist in mapping from a high dimensionality arrays to a squeezed lower dimensionality array. Imagine an array with dimensions |
|
Implemented here: zarrs/zarrs#286 I question the |
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR meets the requirements for being merged. However, there still seems to be some discussion.
Let me know when you're ready to.
Co-authored-by: Norman Rzepka <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>
|
Should I put something in place of "Identifier: (A unique URI to be assigned upon formal adoption)" ? |
This document outlines the
suffixchunk key encoding for Zarr v3, which allows appending user-defined file extensions to chunk keys, enhancing interoperability with traditional file-based tools.