Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ops catalog: thread through collection "version id" as part of inferred schemas #1969

Open
psFried opened this issue Feb 25, 2025 · 1 comment
Assignees
Labels
control-plane data-plane enhance New feature or enhancement with UX impact

Comments

@psFried
Copy link
Member

psFried commented Feb 25, 2025

Inferred schemas are materialized into the inferred_schemas table by our ops catalog. The table is keyed on the collection_name, which presents a problem when a collection is deleted and re-created. We can't tell whether the schema is from the prior version of the collection or the new one, and inferred schema updates from both versions will be merged into the same row.

The solution we'd like to implement is to thread through the "version id" of the collection as part of the inferred schemas, and make it part of the tables primary key. The version id is the build id at which the collection was created. It's a part of the collection journal names, like in the example acmeCo/foo/11223344aabbccdd/pivot=00. The desired outcome is for the inferred_schemas table to have collection_name = 'acmeCo/foo', version_id = '11223344aabbccdd'.

Publications will then only use the inferred schema matching the version_id for each collection in the draft.

@psFried psFried added control-plane data-plane enhance New feature or enhancement with UX impact labels Feb 25, 2025
@psFried psFried self-assigned this Feb 25, 2025
@jgraettinger
Copy link
Member

One nit: elsewhere when we use version, especially within connector protocols, it means the current build ID of the task / spec.

What you're describing is the lifecycle ID or evolution ID (name TBD) of the collection.
I guess we should pick a name and run with it, huh 😆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
control-plane data-plane enhance New feature or enhancement with UX impact
Projects
None yet
Development

No branches or pull requests

2 participants