Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add project config override cli options #1454

Merged
merged 8 commits into from
Dec 4, 2023
Merged

Add project config override cli options #1454

merged 8 commits into from
Dec 4, 2023

Conversation

Ekrekr
Copy link
Contributor

@Ekrekr Ekrekr commented Feb 14, 2023

Fixes #1295

@Ekrekr Ekrekr requested a review from BenBirt February 14, 2023 15:41
BenBirt
BenBirt requested changes Feb 15, 2023
View reviewed changes
cli/index.ts Outdated Show resolved Hide resolved
cli/index.ts Outdated
const defaultDatabaseOption: INamedOption<yargs.Options> = {
name: "default-database",
option: {
describe: "The default database to use. For BigQuery, this is a Google Cloud Project ID.",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should clarify (for all of these) that they override any settings in dataform.json

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thread above - this is for init, so does not override anything, and is a required flag in this case.

cli/index.ts Outdated Show resolved Hide resolved
cli/index.ts Outdated Show resolved Hide resolved
@Ekrekr Ekrekr requested a review from BenBirt February 15, 2023 14:16
@sukhamritboparai
Copy link

hi team, when is this PR merge or feature available?

@Ekrekr
Copy link
Contributor Author

Ekrekr commented May 2, 2023

Examples:

$ bazel run //packages/@dataform/cli:bin init -- --help
...
dataform init <warehouse> [project-dir]

Create a new dataform project.

Positionals:
  warehouse    The project's data warehouse type. If unset, the value from dataform.json is used.  [required] [choices: "bigquery", "presto", "postgres", "redshift", "snowflake", "sqldatawarehouse"]
  project-dir  The Dataform project directory.  [default: "."]

Options:
  --help              Show help  [boolean]
  --version           Show version number  [boolean]
  --default-database  The default database to use. For BigQuery, this is a Google Cloud Project ID.  [string]
  --default-location  The default BigQuery location to use. See https://cloud.google.com/bigquery/docs/locations for supported values.
  --skip-install      Whether to skip installing NPM packages.  [default: false]

$ bazel run //packages/@dataform/cli:bin run -- --help
...
dataform run [project-dir]

Run the dataform project's scripts on the configured data warehouse.

Positionals:
  project-dir  The Dataform project directory.  [default: "."]

Options:
  --help                Show help  [boolean]
  --version             Show version number  [boolean]
  --dry-run             If set, built SQL is not run against the data warehouse and instead is printed to the console.  [boolean]
  --run-tests           If set, the project's unit tests are required to pass before running the project.  [boolean]
  --full-refresh        Forces incremental tables to be rebuilt from scratch.  [boolean] [default: false]
  --actions             A list of action names or patterns to run. Can include '*' wildcards.  [array]
  --tags                A list of tags to filter the actions to run.  [array]
  --include-deps        If set, dependencies for selected actions will also be run.  [boolean]
  --include-dependents  If set, dependents (downstream) for selected actions will also be run.  [boolean]
  --credentials         The location of the credentials JSON file to use.  [default: ".df-credentials.json"]
  --json                Outputs a JSON representation of the compiled project.  [boolean] [default: false]
  --timeout             Duration to allow project compilation to complete. Examples: '1s', '10m', etc.  [string] [default: null]
  --warehouse           The project's data warehouse type. If unset, the value from dataform.json is used.  [choices: "bigquery", "presto", "postgres", "redshift", "snowflake", "sqldatawarehouse"]
  --default-database    The default database to use. For BigQuery, this is a Google Cloud Project ID. If unset, the value from dataform.json is used.  [string]
  --default-schema      Override for the default schema name. If unset, the value from dataform.json is used.
  --default-location    The default BigQuery location to use. See https://cloud.google.com/bigquery/docs/locations for supported values. If unset, the value from dataform.json is used.
  --assertion-schema    Default assertion schema. If unset, the value from dataform.json is used.
  --vars                Override for variables to inject via '--vars=someKey=someValue,a=b', referenced by `dataform.projectConfig.vars.someValue`. If unset, the value from dataform.json is used.  [string] [default: null]
  --database-suffix     Default assertion schema. If unset, the value from dataform.json is used.
  --schema-suffix       A suffix to be appended to output schema names. If unset, the value from dataform.json is used.
  --table-prefix        Adds a prefix for all table names. If unset, the value from dataform.json is used.

BenBirt
BenBirt approved these changes May 2, 2023
View reviewed changes
Copy link
Collaborator

@BenBirt BenBirt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please delete yarn-error.log.

cli/index.ts
@@ -802,3 +735,191 @@ export function runCli() {
yargs.showHelp();
}
}

class ProjectConfigOptions {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still find this structure confusing.

If we were only ever using these flags to construct ProjectConfig, I'd buy it. But we don't; some flags are used for that sometimes, but for other uses other times.

I think this namespacing is more confusing than it's worth, to be honest.

@Ekrekr Ekrekr merged commit da51cfc into main Dec 4, 2023
4 checks passed
@Ekrekr Ekrekr deleted the add-project-config-override-cli-options branch December 4, 2023 15:03
@Ekrekr Ekrekr mentioned this pull request Apr 3, 2024
Closed
moker-spaghetti pushed a commit to moker-spaghetti/dataform that referenced this pull request May 26, 2024
@Ekrekr
Add project config override cli options (dataform-co#1454)
d5b185b 
* Add CLI override options for all project config fields

* Add CLI override options for all project config fields

* Label static methods as public due to dodgy linter on presubmit

* Remove irrelavant if

* de-dupe project config CLI flag definitions

* Tidy method name

* Bump version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BenBirt BenBirt BenBirt approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Override database config is not applied to Auto generated assertions
3 participants
@Ekrekr @sukhamritboparai @BenBirt