obstore

The simplest, highest-throughput ¹ Python interface to S3, GCS, Azure Storage, & other S3-compliant APIs, powered by Rust.

• Sync and async API with full type hinting.
• Streaming downloads with configurable chunking.
• Streaming uploads from files or async or sync iterators.
• Streaming list, with no need to paginate.
• Automatic multipart uploads for large file objects.
• File-like object API and fsspec integration.
• Easy to install with no required Python dependencies.
• Support for conditional put ("put if not exists"), as well as custom tags and attributes.
• Optionally return list results in Apache Arrow format, which is faster and more memory-efficient than materializing Python dicts.
• Zero-copy data exchange between Rust and Python via the buffer protocol.

Installation

To install obstore using pip:

pip install obstore

Obstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:

conda install -c conda-forge obstore

Documentation

Full documentation is available on the website.

Head to Getting Started to dig in.

Footnotes

1. Benchmarking is currently ongoing, but early results indicate 9x higher throughput than aioboto3 and fsspec. ↩