Skip to content

Add Azure Blob Storage backend - #9

Draft
werkamsus wants to merge 1 commit into
tobi:mainfrom
werkamsus:feat/azure-blob-store
Draft

Add Azure Blob Storage backend#9
werkamsus wants to merge 1 commit into
tobi:mainfrom
werkamsus:feat/azure-blob-store

Conversation

@werkamsus

@werkamsus werkamsus commented Aug 25, 2026

Copy link
Copy Markdown

TL;DR: Azure deployments currently need an extra S3/GCS service or a private Walgit fork; this adds an optional Azure Blob backend that implements the existing ObjectStore contract.

  • Problem: Walgit cannot use the object store that is native to Azure.
  • Outcome: store.backend = "azure" supports workload identity in Azure and account-key authentication for Azurite or explicit deployments.
  • Scope: Store implementation, config, contract tests, local Azurite recipe, and the existing D42/contract documentation.
  • Compatibility: S3, GCS, and memory behavior stay unchanged; Azure remains an optional Cargo feature.

PR overview

azure-architecture-trust-v2.png azure-runtime-flow-v2.png

Why

Running Walgit on Azure currently requires a second cloud storage protocol or an out-of-tree patch. That adds another service, credential model, and failure boundary even though Azure Blob can provide the versioned object operations Walgit's store contract needs.

This work ports the Azure backend from #1 by @kzu onto current main, then narrows and hardens it for the current contract. Thank you to Daniel Cazzulino for the original implementation.

What changed

Azure store contract

  • Add AzureStore behind the optional azure feature.
  • Use Blob ETags as opaque version tokens for compare-and-swap writes.
  • Implement conditional and ranged reads, head, listing, deletion, streaming uploads, multipart block uploads, compose, and signed GET URLs.
  • Give each multipart upload unique fixed-width block IDs. Concurrent uploads to one key cannot commit each other's staged blocks.
  • Use the control or bulk HTTP client from the same key/range classifier as GCS.

Authentication and trust boundaries

  • Require an explicit credential mode: workload_identity or account_key.
  • Keep account keys outside TOML. Config names the environment variable that contains the key.
  • Reject non-HTTPS workload-identity Blob endpoints.
  • Allow account-key HTTP only for loopback endpoints such as Azurite.
  • Cache Azure user-delegation keys with an expiry safety margin instead of requesting one per signed URL.

Local development and documentation

  • Add an Azurite Compose service and just dev-azurite / just test-azure recipes.
  • Extend the shared store contract suite with Azure and concurrent multipart coverage.
  • Document Azure as a first-class backend in D42 and docs/CONTRACT.md.
  • Keep the existing round-trip budget unchanged.

Security

Workload identity is the production default and does not require an account key. Account-key mode reads one explicitly configured environment variable and never accepts the key as a TOML value. Workload tokens cannot be sent to plaintext custom endpoints. Signed URLs use service SAS with an account key or a cached user-delegation key with workload identity.

How this was tested

  1. cargo fmt --all -- --check
    • Proves the Rust changes match repository formatting.
  2. cargo check -p walgit-cli --locked
    • Proves the full CLI compiles with the Azure feature enabled by walgit-cli.
  3. cargo test -p walgit-config
    • 18 tests passed, including explicit credential modes and unknown-field rejection.
  4. cargo test -p walgit-store --features azure
    • 56 tests passed across unit and contract support code.
  5. just test-azure against local Azurite
    • Proved ETag CAS, conditional GET, range/head/list/delete, streaming and multipart writes, compose fallback, signed GET, and concurrent multipart isolation.
  6. Adversarial review found and this branch fixed six issues:
    1. cross-upload block ID collisions;
    2. bulk-client use for all GETs;
    3. one delegation-key request per signed URL;
    4. workload tokens allowed on HTTP custom endpoints;
    5. a healthy-path HEAD before conditional delete;
    6. control-client use for bulk-key writes.

After merge

No existing deployment changes behavior. Azure users enable the Cargo feature, select store.backend = "azure", and configure one credential mode. A live Azure deployment should still verify workload-identity token exchange, user-delegation SAS, and server-side Put Block From URL; Azurite covers the account-key and local compose paths.

Add Azure Blob Storage as an optional ObjectStore backend with explicit workload-identity or account-key credentials.

Implement the existing store contract with ETag CAS, conditional and ranged reads, streaming uploads, isolated multipart block IDs, server-side compose, and signed GET URLs. Add Azurite contract coverage and keep large-object traffic on the bulk client.

This ports and hardens the Azure backend from upstream PR tobi#1 by @kzu onto current main. It removes unrelated platform changes and narrows authentication to explicit modes.
@werkamsus

Copy link
Copy Markdown
Author

@tobi saw you closed #1, is there any appetite to add support for this? keeping this in drafts to not clutter reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant