feat(datagen): add DatagenStore read/write service API across all six layers - #212
Merged
jackye1995 merged 7 commits intoJul 27, 2026
Merged
Conversation
Bring the datagen checkpoint delta-log in line with the authoritative
spec. This changes the event vocabulary and folded model only; the
storage mechanism (single append-only log.lance, MemWAL sharding,
deterministic event_id, blob offload) is unchanged.
- Event types 6 -> 7: add STEP_STARTED.
- Replace the `terminal` column with a `status` column
(running / completed / filtered / failed).
- Replace step_instance_id + iteration provenance with structured
step_kind / enclosing_step / selector_step.
- Structured DatagenItemId with materialized path `root/step:idx/...`.
- New DatagenStepKind {Root, Leaf, Sequence, Loop, MapReduce, Branch,
SubPipeline, Conditional, Router}; only drivers emit STEP_STARTED.
- fold_datagen_events returns Option (None when no ITEM_CREATED);
FIELD_SET last-writer-wins, FIELD_APPEND accumulates; two read
lenses (lifecycle vs failure). DATAGEN_SCHEMA_VERSION = 2.
- Rewrite specs/datagen-checkpoint-schema.md for schema v2: 7 events, status column, structured item id + step provenance, read lenses. - Add docs/design/using-datagen-store.md: a client-facing walkthrough of open/write/checkpoint/resume/read with runnable snippets. - Add 7 fold + store tests: step_kind/status parse round-trips, filtered terminal, selector_step on the chosen child, fan-out sub-item lineage, set/append mixing rejection, resume open-frame (started minus completed), and a fan-out tree read + root classification through the store. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lta-log-reshape # Conflicts: # crates/lance-context-core/src/datagen_store.rs # docs/src/design/using-datagen-store.md
- lance-context umbrella re-exported the pre-reshape name DatagenTrajectoryPoint; rename to DatagenTrajectory so the crate (and the python wheel + tests that depend on it) compiles again. - allow(large_enum_variant) on DatagenItemLookup (Found is the hot path) and drop a redundant #[must_use] on iter(). - cargo fmt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… layers The schema-v2 reshape (lance-format#205) landed the data model but not the API layer on top of it. This wires the datagen store through the full stack so clients can append, fold, and read blobs both embedded and over the server: - api: `DatagenStoreApi` trait (RPITIT) + wire DTOs mirroring the Python dicts - core: `impl DatagenStoreApi for DatagenStore` with core<->DTO converters - client: `RemoteDatagenStore` HTTP client - unified: `enum DatagenStore {Local, Remote}` with dispatch - server: `/api/v1/datagen` routes (create/list/get/delete, events, fold, failures, root-item-statuses, blob fetch) - python: PyO3 `DatagenStore` binding + `open`/`connect`/`connect_or_create` Verified end-to-end locally: identical checkpoint round-trips through both the embedded (local-file) and remote (server) paths fold to the same item state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
beinan
approved these changes
Jul 27, 2026
…-api # Conflicts: # crates/lance-context/src/lib.rs
The lint job runs ruff + pyright without building the wheel, so re-exporting datagen_event_id via a redundant alias left it unresolved (_internal has no stub). Wrap it in a thin function like generate_id, and fix import ordering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jackye1995
pushed a commit
to lance-format/lance
that referenced
this pull request
Jul 27, 2026
## Summary `lance-context` has a `docs/src` tree in the same shape as the other subprojects, but nothing pulls it into lance.org. This wires it in alongside spark/ray/trino so its docs publish the same way they do. Context: I originally set up a standalone GitHub Pages site on the lance-context repo, and @jackye1995 pointed out that the org deliberately doesn't enable per-subproject Pages — public docs should build into lance.org instead. This is that change; the matching lance-context-side PR is lance-format/lance-context#212. ## Changes Follows the existing pattern exactly, in both scripts: - `LANCE_CONTEXT_REPO` (default `$HOME/oss/lance-context`), documented in the usage block like the others - `docs/src` → `docs/src/integrations/context`, adding the nav entry only when the checkout is present, so a partial local build still works and just warns - `CONTRIBUTING.md` → `community/project-specific/context.md` - `clean-full-website.sh` removes both, keeping make/clean symmetric ## Verification Ran the real thing rather than eyeballing the diff: - `make-full-website.sh` against an actual lance-context checkout — all 12 pages copied, `integrations/.pages` nav entry generated, no warning for context - `mkdocs build` succeeds; pages render under `site/integrations/context/` with correct nav nesting (`context/guide/quickstart/` etc. are linked from the integrations index) - `clean-full-website.sh` removes everything it added ## Note on placement I put it under **Integrations** since that's where the other subprojects live and it needs no new top-level section. It's arguably not an "integration" in the same sense as Spark or Trino — it's a storage engine built on Lance — so if you'd prefer a different home, say the word and I'll move it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The schema-v2 reshape (#205) landed the datagen data model — events, all schema columns, ids, the fold, status, provenance — but not the API layer on top of it. This PR wires
DatagenStorethrough the full six-layer stack so clients can append, fold, and read blobs both embedded (local file) and over the server.Layers
lance-context-api) —DatagenStoreApitrait (RPITIT) + wire DTOs mirroring the Python dicts. The api can't depend on core (cycle), so the trait returns DTOs.lance-context-core) —impl DatagenStoreApi for DatagenStore, with core↔DTO converters and DTO→enum validation.lance-context-client) —RemoteDatagenStoreHTTP client.lance-context) —enum DatagenStore { Local, Remote }with dispatch.lance-context-server) —/api/v1/datagenroutes: create/list/get/delete, append (?checkpoint=), fold item, item failures, root-item-statuses, blob fetch.DatagenStorebinding withopen/connect/connect_or_create.Verification
Verified end-to-end locally: the same checkpoint round-trip driven through both the embedded (local-file) and remote (server-backed) paths folds to identical item state (fields, status,
last_item_seq, trajectory, root-item statuses). Remote datasets materialize on the server's local-SSD data dir.cargo fmt,cargo clippy --workspace --all-targets, andcargo test -p lance-context-core datagen(20 passed) are clean.Relation to the gap audit
This closes most of the write-path, blob-read, and Python gaps catalogued against
mainin the downstream ckpt implementation-gaps doc.