Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update dependency @dataform/core to v3 #29

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update dependency @dataform/core to v3 #29

wants to merge 1 commit into from

Conversation

renovate-bot
Copy link
Contributor

@renovate-bot renovate-bot commented Jun 21, 2024 •
edited
Loading

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
@dataform/core 2.0.1 -> 3.0.9 age adoption passing confidence

Release Notes

dataform-co/dataform (@​dataform/core)

v3.0.9

Compare Source

What's Changed

New Contributors

Full Changelog: dataform-co/dataform@3.0.8...3.0.9

v3.0.8

Compare Source

What's Changed

Full Changelog: dataform-co/dataform@3.0.7...3.0.8

v3.0.7

Compare Source

What's Changed

Full Changelog: dataform-co/dataform@3.0.4...3.0.7

v3.0.6

Compare Source

v3.0.5

Compare Source

v3.0.4

Compare Source

What's Changed

New Contributors

Full Changelog: dataform-co/dataform@3.0.2...3.0.4

v3.0.3

Compare Source

v3.0.2

Compare Source

What's Changed

Full Changelog: dataform-co/dataform@3.0.1...3.0.2

v3.0.1

Compare Source

Important: Prefer 3.0.2 over this release - this release has issues.

See https://github.com/dataform-co/dataform/issues/1800 for more context.

What's Changed

New Contributors

Full Changelog: dataform-co/dataform@3.0.0...3.0.1

v3.0.0

Compare Source

TL;DR of What's Changed Since 2.9.0

dataform.json -> workflow_settings.yaml

workflow_settings.yaml has been introduced, which will gradually replace dataform.json in a later version; there is no immediate action to be taken, as dataform.json files are still valid in projects with Dataform Core 3.0.0.

dataform.json is being deprecated in favor of workflow_settings.yaml. This means that:

  • Workflow settings are now strictly typed, in Protobuf format.
  • The Dataform Core version can be specified directly in the workflow_settings.yaml file. Note: to have more than just @​dataform/core as a dependency, a package.json must still be used.

Example conversion of workflow_settings.yaml:

defaultProject: dataform-demos
defaultLocation: us
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
dataformCoreVersion: 3.0.0
vars:
    environmentName: "development"

The above is equivalent to the dataform.json file:

{
  "warehouse": "bigquery",
  "defaultDatabase": "dataform-demos",
  "defaultLocation": "us",
  "defaultSchema": "dataform",
  "assertionSchema": "dataform_assertions"
  "vars": {
    "environmentName": "development"
  }
}
Notebooks Actions and actions.yaml

Notebooks as Dataform actions are on their way - but not quite yet! They're part of the compiled graph, and soon they'll be executable.

A new way of configuring action configs through actions.yaml has been implemented to support this.

An example of loading a notebook in Dataform can be seen at https://github.com/dataform-co/dataform/tree/main/examples/extreme_weather_programming.

Stateless Package Installation by @​dataform/cli

Package installation by @​dataform/cli is now stateless! The CLI will install NPM packages during compilation if version is defined in the workflow_settings.yaml file.

This means no node_modules folder has to be seen in the project, and Dataform users no longer need to be familiar with NPM.

Compilation Output is Now Warehouse Agnostic

Previously the output of compilation results from @​dataform/core would insert warehouse specific SQL into the compiled graph. Where possible, this has been removed - transferring the responsibility of inserting warehouse specific SQL into whichever execution engine is running Dataform.

Additionally, support for non-BigQuery warehouses has been dropped. We're in discussions with Datashell for them to provide a warehouse-agnostic CLI execution engine based off of Dataform compiled graphs. In the meantime however, if you need support for a non-BigQuery warehouse, please continue using the latest version starting with 2.x.x!

dependOnDependencyAssertions

An easier ways to add assertions from dependency as dependencies has been introduced.

dependOnDependencyAssertions in config blocks can be used to add assertions from all dependencies of the action as dependencies.

config { 
    type: "view",
    dependOnDependencyAssertions: true,
    dependencies: ["some_table"]
}

select test from ${ref("some_other_table")}

Additionally, the includeDependentAssertions parameter can be used when setting individual dependencies either in config.dependencies or in ref() to add assertions for these dependencies as the dependencies for current action.

config { 
    type: "view",
    dependencies: [{name: "some_table", includeDependentAssertions: true}]
}

select test from ${ref({name: "some_other_table", includeDependentAssertions: true})}

Full Changelog from 2.9.0: dataform-co/dataform@2.9.0...3.0.0

v2.9.0

Compare Source

What's Changed

New Contributors

Full Changelog: dataform-co/dataform@2.8.4...2.9.0

v2.8.4

Compare Source

What's Changed

Full Changelog: dataform-co/dataform@2.8.3...2.8.4

v2.8.3

Compare Source

What's Changed

New Contributors

Full Changelog: dataform-co/dataform@2.8.2...2.8.3

v2.8.2

Compare Source

What's Changed

Full Changelog: dataform-co/dataform@2.8.1...2.8.2

v2.8.1

Compare Source

What's Changed

New Contributors

Full Changelog: dataform-co/dataform@2.8.0...2.8.1

v2.8.0: : Packaging and compilation performance improvements

Compare Source

The highlights of this release are two significant performance improvements for GCP Dataform projects:

  • The @dataform/core NPM package now no longer has any dependencies, with all dependency packages bundled in to existing minified bundle. This reduces the size of installation significantly and should improve package installation performance and reliability on Dataform on GCP. See https://github.com/dataform-co/dataform/pull/1552 for more info.
  • We've rolled out a change that reduces the amount of work required to decode/encode compiled graphs that we expect to improve compilation performance on Dataform on GCP by ~2x. See https://github.com/dataform-co/dataform/pull/1570 for more info.

What's Changed

New Contributors

Full Changelog: dataform-co/dataform@2.7.0...2.8.0

v2.7.0: : Updates for Dataform GCP incremental SQL

Compare Source

From version 2.7.0 onwards, Dataform projects running on Google Cloud Platform will use updated SQL generation logic for incremental insert tables (tables of type incremental without a uniqueKey specified).

Explicit column names

Column names will be explicitly listed in the insert call, which is inline with OSS Dataform behaviour and prevents schema mismatch during insert, for example:

  • if source_table has columns in a different order that the target - it can lead to data corruption (column values can be swapped during insert)
  • if source_table has different number of columns - the insert into fails since target table columns count does not match with the source

For the new script the incremental query has to list all target table columns (can list other extra columns, but at least must contain target table columns) in any order.

Example of new generated SQL:

INSERT INTO $target_table 
    ($target_columns_list) -- listing target columns
    SELECT target_columns_list -- reordering columns so that subquery column order matches the target column order
    FROM (
        $incrementalQuery
    );
Execution within a procedure

In order to facilitate explicit columns, the new code is executed within a procedure, which will be created on the fly. For example:

EXECUTE IMMEDIATE
"""
CREATE OR REPLACE PROCEDURE $procedure_name()
BEGIN
    $preOperations
    $incrementalInsertStatement
    $postOperations
END;
"""
CALL $procedure_name();
DROP PROCEDURE IF EXISTS $procedure_name;

v2.6.8

Compare Source

Fixing bug for escaping target name and canonical target name. PR: https://github.com/dataform-co/dataform/pull/1551

v2.6.7: : Support NO_COLOR environment variable

Compare Source

@dataform/cli now supports turning off coloured output via setting the NO_COLOR environment variable, per the standard detailed at https://no-color.org/.

v2.6.6: : Enforce tighter requirements around "automatic" importing of includes files

Compare Source

We have fixed two bugs which inadvertently loosened our (expected) requirements around usage of "automatic" includes imports.

For context: in general, to reference the contents of a file in includes (specifically a file's module.exports object), the callsite should call require() on that file, e.g. const foo = require("includes/subdirectory/foo.js");.

Dataform simplifies this for "top-level" includes files, i.e. direct children of the includes directory, by automatically making these files available globally. For example, in order to use includes/foo.js, a callsite does not need to require("includes/foo.js"); instead, a foo object is made available to all Dataform code in the project.

Two bugs have been found and fixed:

  1. includes files can no longer implicitly depend on other includes files
  2. only top-level includes files are now automatically available globally

This unfortunately results in a potentially breaking change to some Dataform projects - but this will only happen upon upgrading @dataform/core to >= 2.6.6.

In order to fix any breakages, the calling file must be changed to explicitly require() the relevant includes file.

For example, in SQLX:

js {
  const whatever = require("includes/subdirectory/whatever.js");
}

Or in JavaScript:

const whatever = require("includes/whatever.js");

For more context, see https://issuetracker.google.com/issues/296162656#comment3.

v2.6.5: : Fix `@​dataform/clifor@​dataform/core pre-2.6.5`.

Compare Source

This version fixes behaviour in rare circumstances which was broken by @dataform/cli version 2.6.3. See https://issuetracker.google.com/issues/296162656#comment3 for details.

v2.6.4: : Fix dataform compile on Windows

Compare Source

v2.6.3: : Bump glob dependency version.

Compare Source

v2.6.2: : Fix: triple-quotes strings getting removed after compilation

Compare Source

The lexer was updated to have multiline string tokens (in 2.6.1), which gave us ability to format triple quoted strings (https://github.com/dataform-co/dataform/releases/tag/2.6.1). But in compilation these tokens were not being handled and thus they were getting removed after compilation. We have fixed this issue in this version.

v2.6.1: : Improve formatting for triply-quoted strings

Compare Source

  • improve SQLX formatting for triply-quoted strings
  • various dependency version upgrades

UPDATE: This version introduced a bug: triple-quoted strings are getting removed after compilation. The fix is in 2.6.2 (https://github.com/dataform-co/dataform/releases/tag/2.6.2)

v2.6.0: : Improve dataform format CLI command

Compare Source

The dataform format command is now dramatically improved and has better formatter output for each SQL dialect that Dataform supports. Thanks @​pokutuna for the contribution!

v2.5.0: : Enable use of schema() and database() context functions in SQLX code

Compare Source

v2.4.2: : Throw clearer compilation errors when non-string vars are provided

Compare Source

v2.4.1: : Add a CLI flag to disable analytics tracking

Compare Source

Users of the CLI (who are otherwise opted-in to analytics tracking) may now pass --track=false on an individual CLI command basis to disable analytics tracking.

v2.4.0: : Fix docker image to run dataform help as default command

Compare Source

v2.3.2: : Fix various CLI options

Compare Source

  • CLI errors regarding missing credentials files should now be rarer, and contain more explanatory detail when they are thrown
  • --include-dependents (for dataform run) now properly includes dependents of selection actions

v2.3.0: : Add a database() function in SQL templating

Compare Source

Both SQLX and JavaScript now support a database() function, which returns the (prefixed, as applicable) name of "this" action's database. (Contrived) examples:

publish("table", ctx => `select "${ctx.database()}" as col1`);

config { type: "table" }
select "${database()}" as col1

Assuming dataform.config contains "defaultDatabase": "foo", these both generate a table action whose SQL is select "foo" as col1. This is likely most useful in type: "operations" scripts.

Note: For warehouses without databases such as Redshift, this will add an error to the compilationErrors property.

Bugfix: The similar name() function has been fixed to properly attach any prefix as appropriate.

v2.2.0: : Add schema() function in SQL templating

Compare Source

Both SQLX and JavaScript now support a schema() function, which returns the (suffixed, as applicable) name of "this" action's schema. (Contrived) examples:

publish("table", (ctx) => `select "${ctx.schema()}" as col1`);

config { type: "table" }
select "${schema()}" as col1

Assuming dataform.config contains "defaultSchema": "foo", these both generate a table action whose SQL is select "foo" as col1. This is likely most useful in type: "operations" scripts.

v2.1.0: : Table auto-assertions now share the same tags as their parent table

Compare Source

v2.0.4: : Default to main compilation.

Compare Source

This release introduces no new features, but it does change the default compilation mode to something we call main compilation: a cleaner and more maintainable compilation script.

v2.0.3: : Include core.proto in published packages.

Compare Source

v2.0.2: : Sandbox security upgrades.

Compare Source

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@renovate-bot renovate-bot force-pushed the renovate/dataform-core-3.x branch from f2387d1 to 809ee83 Compare August 28, 2024 06:54
@renovate-bot renovate-bot force-pushed the renovate/dataform-core-3.x branch 2 times, most recently from 146bb38 to a1ee2d9 Compare October 1, 2024 15:51
@renovate-bot renovate-bot force-pushed the renovate/dataform-core-3.x branch 3 times, most recently from fb49fee to 6be0032 Compare October 15, 2024 12:26
@renovate-bot renovate-bot force-pushed the renovate/dataform-core-3.x branch from 6be0032 to fe73e70 Compare November 4, 2024 11:14
@renovate-bot renovate-bot force-pushed the renovate/dataform-core-3.x branch from fe73e70 to 96b88ed Compare January 3, 2025 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@renovate-bot