Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "zarrista"
version = "0.1.0-rc.1"
version = "0.1.0"
authors = ["Kyle Barron <kyle@developmentseed.org>"]
edition = "2024"
description = "A small, prototypical zarrita-like Python Zarr implementation on top of zarrs."
Expand Down
31 changes: 27 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

A fast, low-level [Zarr] API for Python, powered from Rust by [Zarrs].

[Very early benchmarking](https://github.com/developmentseed/zarrista/pull/151) suggests potential 17-35x faster read throughput than [Zarr-Python].
[Initial benchmarks](https://github.com/zarrs/zarr_benchmarks/pull/12) suggest Zarrista is **1.9x to 2.7x faster** than [Zarr-Python] for reading compressed or compressed+sharded data, respectively, from a local file system. We expect future async-focused benchmarks to be even faster.

While Zarrista will exist as a standalone Python library and can be used directly, the goal is to integrate Zarrista directly into [Zarr-Python] so that existing users can get improved performance out of the box and to avoid fracturing the ecosystem.

Expand Down Expand Up @@ -46,6 +46,29 @@ This library is beta-quality. The underlying [Zarrs] library is reliable and bro
[`Tensor`]: https://developmentseed.org/zarrista/latest/api/tensor/#zarrista.Tensor
[DLPack]: https://dmlc.github.io/dlpack/latest/
[`ml_dtypes`]: https://github.com/jax-ml/ml_dtypes
[Tensorstore]: https://github.com/google/tensorstore

## Benchmarks

In [our PR](https://github.com/zarrs/zarr_benchmarks/pull/12) to [`zarr_benchmarks`](https://github.com/zarrs/zarr_benchmarks), Zarrista is the fastest Python chunked array library, in line with Google's [Tensorstore]. It's surpassed only by the Rust [Zarrs] library, which Zarrista uses internally.

### Read All

The minimum time and peak memory usage to read an entire dataset into memory.

![](assets/benchmark_read_all.svg)

### Read Chunk-By-Chunk

The minimum time and peak memory usage to read a dataset chunk-by-chunk into memory.

![](assets/benchmark_read_chunks.svg)

### Read Subchunk-By-Subchunk

The minimum time and peak memory usage to read a dataset subchunk-by-subchunk into memory.

![](assets/benchmark_read_subchunks.svg)

## Example

Expand Down Expand Up @@ -87,11 +110,11 @@ You can also read individual chunks by their grid index:
data = array.retrieve_chunk([0, 0])
```

## AI Policy
## AI Usage

This project is _minimally_ vibe-coded.
Zarrista's source code is _minimally_ vibe-coded.

Most of the library code was written by hand by @kylebarron, sometimes in conversation with Claude.
Most of the library code was written by hand by Kyle, sometimes resulting from a conversation with Claude.

Documentation and type stubs are mixed. Much documentation is written by hand but Python type stubs are partially kept up to date via Claude.

Expand Down
Loading
Loading