Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt clone command #3742

Merged
merged 19 commits into from
Jul 18, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ exports.versions = [
]

exports.versionedPages = [
{
"page": "reference/commands/clone",
"firstVersion": "1.6",
},
{
"page": "docs/collaborate/govern/project-dependencies",
"firstVersion": "1.6",
Expand All @@ -51,7 +55,7 @@ exports.versionedPages = [
"page": "docs/collaborate/govern/model-contracts",
"firstVersion": "1.5",
},
{
{
"page": "reference/commands/show",
"firstVersion": "1.5",
},
Expand Down
33 changes: 33 additions & 0 deletions website/docs/reference/commands/clone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: "About dbt clone command"
sidebar_label: "clone"
id: "clone"
---

The `dbt clone` command clones selected nodes from the specified state to the target schema(s). This command makes use of the `clone` materialization:
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
- If using a data warehouses that support zero-copy cloning of tables, selected models that are tables will be created as clones
- For views and warehouses that do not support zero-copy cloning, selected models will be created as simple `select * from ...` pointer views
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

Note:
- The state to clone from is based on the location of nodes in the manifest provided to `--state`.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

@aranke aranke Jul 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something else to note here is that you cannot use the same target directory for both the source and target due to <placeholder>.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it literally not allow the target and source to be the same? Or would it yield surprising results?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is something like this typically called out in the docs @runleonarun

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graciegoheen I would add this tidbit as a sentence in the very first paragraph. Maybe by rephrasing as what you should do instead of what you shouldn't, for example:

The dbt clone command clones selected nodes from the specified state to the target schema(s). You must always use different directories for source and target otherwise you will get an error message/overwrite the source. This command makes use of the clone materialization:

(hopefully you just get an error @aranke??)

Copy link
Member

@aranke aranke Jul 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You get a very weird error that source and target states are the same, and so it's a no-op.

I ran into this during local development, so think this warning needs to be surfaced loudly.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like we want a dbt-core follow-on issue?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ yes and add callout to the docs? or yes then we don't need this callout?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say "yes, and" — then when the issue is closed, we can remove the callout from the docs

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs are updated - @dbeatty10 or @aranke are one of y'all able to open up a core issue to improve the error message?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graciegoheen I just now opened a minimal issue here: dbt-labs/dbt-core#8160

dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
- By default, `dbt clone` will not recreate pre-existing relations in the current target. To override this, use the `--full-refresh` flag.
- You may want to specify a higher number of [threads](website/docs/docs/running-a-dbt-project/using-threads.md) to decrease execution time since individual clone statements are independent of one another.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

The `clone` command is useful for:
- blue/green deployment
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
- cloning current production state into development schema(s)
- handling incremental models in Slim CI dbt Cloud jobs

```bash
# clone all of my models from specified state to my target schema(s)
dbt clone --state path/to/artifacts

# clone one_specific_model of my models from specified state to my target schema(s)
dbt clone --select one_specific_model --state path/to/artifacts

# clone all of my models from specified state to my target schema(s) and recreate all pre-exisiting relations in the current target
dbt clone --state path/to/artifacts --full-refresh

# clone all of my models from specified state to my target schema(s), running up to 50 clone statements in parallel
dbt clone --state path/to/artifacts --threads 50
```
2 changes: 2 additions & 0 deletions website/docs/reference/dbt-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Select the tabs that are relevant to the your development workflow. For example,
Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`.

- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
- [clone](/reference/commands/clone): clone selected nodes from specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [deps](/reference/commands/deps): downloads dependencies for a project
- [docs](/reference/commands/cmd-docs) : generates documentation for a project
Expand All @@ -40,6 +41,7 @@ Use the following dbt commands in the [CLI](/docs/core/about-the-cli) and use th

- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
- [clean](/reference/commands/clean): deletes artifacts present in the dbt project
- [clone](/reference/commands/clone): clone selected models from specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [debug](/reference/commands/debug): debugs dbt connections and projects
- [deps](/reference/commands/deps): downloads dependencies for a project
Expand Down
3 changes: 2 additions & 1 deletion website/docs/reference/node-selection/syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,9 @@ dbt can leverage artifacts from a prior invocation as long as their file path is
- [The `state:` selector](/reference/node-selection/methods#the-state-method), whereby dbt can identify resources that are new or modified
by comparing code in the current project against the state manifest.
- [Deferring](/reference/node-selection/defer) to another environment, whereby dbt can identify upstream, unselected resources that don't exist in your current environment and instead "defer" their references to the environment provided by the state manifest.
- The [`dbt clone` command](website/docs/reference/commands/clone.md), whereby dbt can clone nodes based on their location in the manifest provided to the `--state` flag.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

Together, these two features enable ["slim CI"](/guides/legacy/best-practices#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag.
Together, the `state:` selector and deferral enable ["slim CI"](/guides/legacy/best-practices#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

### Establishing state

Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,7 @@ const sidebarSettings = {
items: [
"reference/commands/build",
"reference/commands/clean",
"reference/commands/clone",
"reference/commands/cmd-docs",
"reference/commands/compile",
"reference/commands/debug",
Expand Down
Loading