Versioning of Data Products

[MCaviezel]

Hi all,
We are currently implementing Data Mesh in our company.
One discussion, which we are having right now is, if a Data Product should have a Version.
Currently in the https://dataproduct-specification.com 0.0.1. this is not implemented.
Our Use-Case for versioning it is that we want to know which Data Product State is deployed in which environment at what point in time.
Like this we can also create a Rollback Strategy to a specific Data Product version, or identify a root cause of an error.
Any thoughts you can share about this would be interesting to hear!
Cheers,
Marco

[jochenchrist]

Hi @MCaviezel

We have had the discussion with most of our customers.

Based on the experience, we now recommend defining the version on an output port level, as the output port is the physical link, e.g. to a database schema or bucket of the provided data.
The output port defines: the data model, version, technology (s3, databricks, ...), and environment (prod, test, dev).

Take a look at https://demo.datamesh-manager.com how the versions and environments are managed there on the output port.

What do you think?

[MCaviezel]

Hi @jochenchrist

We would like to track which subset of Data transformation Code, Output Port and Data Contract of a Data Product was deployed at what point in time.
If this information is stored in a "Parent" Document (which would be the Data Product Specification), we have technical Auditability, which helps to trace errors in the Mesh. (e.g. if you haven't changed anything of the before stated things, then either the Data or a Producing Data Product must have changed)

Our thought so far is to add Gitlab Tags to Data Product Specification.
This way we can have a "frozen" version of deployed IaC, Data Transformation, Data Product Specifications and Data Contracts.

Or do you have other ideas or experience to solve this requirement?

[jochenchrist]

Of course, you can add a tag for the version or commit-ID to the data product.

Also, think about: can you have multiple versions in parallel? What about different environments?

Also, consider a data product not necessarily as the technical artifact (the produced data set in the data lakehouse), but more as an organizational unit with ownership (!), data contracts, description, consumers, maturity, ... The data product then can have technical artifacts / assets attached to it, which then, of course, have a specific version.

But there is no right or wrong, it must fit to your organization, data platform and data culture.

[MCaviezel]

I believe environments are managed through different output ports, while the Data Product operates not inside a specific environment, but rather at a higher level that spans across them.
In our case, we can deploy only one Data Product at a time, with no parallel deployments possible.
I agree that there are there should be even more organizational artifacts on the Data Product Level, like the guys from https://opendataproducts.org have. But they are missing Output Ports.
Do you plan to add more organizational Artifacts in "your" Specification (https://dataproduct-specification.com)?

[jochenchrist]

Again, look at Data Mesh Manager for more examples:

Here "Assets" are linked (either on a data product level or output port level)

dataProductSpecification: 0.0.1
id: orders
info:
  title: Orders
  description: Successful customer orders in the webshop. All orders since 2020-01-01.
  owner: sales-team
  status: active
  archetype: source-aligned
  icon: snowflake
outputPorts:
- id: snowflake_orders_pii_v1
  name: Orders PII
  description: "All order-created events, with PII."
  type: snowflake
  environment: prod
  version: 1
  server:
    schema: ORDERS_PII_V1
    account: lmtabcd-xn12345
    database: ORDER_DB
  status: retired
  autoApprove: false
  containsPii: true
  links:
    Snowflake WebUI: https://app.snowflake.com/xxx/yyy/#/data/databases/ORDER_DB/schemas/ORDERS_PII_V1
  custom:
    platformRole: op_orders_snowflake_orders_pii_v1_role
- id: snowflake_orders_pii_v1_dev
  name: Orders PII
  description: "All order-created events, with PII."
  type: snowflake
  environment: dev
  version: 1
  server:
    schema: ORDERS_PII_V1
    account: lmtabcd-xn12345-dev
    database: ORDER_DB
  status: retired
  autoApprove: false
  containsPii: true
  custom:
    platformRole: op_orders_snowflake_orders_pii_v1_role
- id: snowflake_orders_pii_v2
  name: Orders PII
  description: "All order-created events, with PII."
  type: snowflake
  environment: prod
  version: 2
  server:
    schema: ORDERS_PII_V2
    account: lmtrwxs-xn14859
    database: ORDER_DB
  status: active
  autoApprove: false
  containsPii: true
  dataContractId: snowflake_orders_pii_v2
  tags:
  - location:EU
  - datacontracttests:enabled
  links:
    Snowflake WebUI: https://app.snowflake.com/lmtrwxs/xn14859/#/data/databases/ORDER_DB/schemas/ORDERS_PII_V2/tables
  custom:
    platformRole: op_orders_snowflake_orders_pii_v2_role
- id: snowflake_orders_pii_v2_dev
  name: Orders PII
  description: "All order-created events, with PII."
  type: snowflake
  environment: dev
  version: 2
  server:
    schema: ORDERS_PII_V2
    account: lmtabcd-xn12345-dev
    database: ORDER_DB
  status: active
  autoApprove: false
  containsPii: true
  custom:
    platformRole: op_orders_snowflake_orders_pii_v2_role
- id: snowflake_orders_npii_v3
  name: Orders NPII
  description: "All order-created events, PII removed."
  type: snowflake
  environment: prod
  version: 3
  server:
    schema: ORDERS_NPII_V3
    account: lmtabcd-xn12345
    database: ORDER_DB
  status: in development
  autoApprove: false
  containsPii: false
- id: snowflake_orders_npii_v2
  name: Orders NPII
  description: "All order-created events, PII removed."
  type: snowflake
  environment: prod
  version: 2
  server:
    schema: ORDERS_NPII_V2
    account: lmtabcd-xn12345
    database: ORDER_DB
  status: deprecated
  autoApprove: false
  containsPii: false
  dataContractId: snowflake_orders_npii_v2
  custom:
    platformRole: op_orders_snowflake_orders_npii_v2_role
- id: snowflake_orders_npii_v1
  name: Orders NPII
  description: "All order-created events, PII removed."
  type: snowflake
  environment: prod
  version: 1
  server:
    schema: ORDERS_NPII_V1
    account: lmtabcd-xn12345
    database: ORDER_DB
  status: retired
  autoApprove: false
  containsPii: false
  custom:
    platformRole: op_orders_snowflake_orders_npii_v1_role
tags:
- demo
- featured
custom:
  platformRole: dp_orders_role
assets:
- assetId: tables-order
  tags:
  - table

[MCaviezel]

Yes, I've looked into this.
But my question ist if you are planning to include more major aspects on Data Product Level and not only on Output Port / Data Contract Level like https://opendataproducts.org/v3.1/#specification-aims-and-aspects ?