Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt clone command #3742

Merged
merged 19 commits into from
Jul 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ exports.versions = [
]

exports.versionedPages = [
{
"page": "reference/commands/clone",
"firstVersion": "1.6",
},
{
"page": "docs/collaborate/govern/project-dependencies",
"firstVersion": "1.6",
Expand All @@ -55,7 +59,7 @@ exports.versionedPages = [
"page": "docs/collaborate/govern/model-contracts",
"firstVersion": "1.5",
},
{
{
"page": "reference/commands/show",
"firstVersion": "1.5",
},
Expand Down
39 changes: 39 additions & 0 deletions website/docs/reference/commands/clone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
title: "About dbt clone command"
sidebar_label: "clone"
id: "clone"
---

The `dbt clone` command clones selected nodes from the [specified state](/reference/node-selection/syntax#establishing-state) to the target schema(s). This command makes use of the `clone` materialization:
- If your data platform supports zero-copy cloning of tables, and this model exists as a table in the source environment, dbt will create it in your target environment as a clone
- Otherwise, dbt will create a simple pointer view (`select * from` the source object)
- By default, `dbt clone` will not recreate pre-existing relations in the current target. To override this, use the `--full-refresh` flag.
- You may want to specify a higher number of [threads](/docs/running-a-dbt-project/using-threads) to decrease execution time since individual clone statements are independent of one another.

The `clone` command is useful for:
- blue/green continuous deployment (on data warehouses that support zero-copy cloning tables)
- cloning current production state into development schema(s)
- handling incremental models in Slim CI dbt Cloud jobs (on data warehouses that support zero-copy cloning tables)
- testing code changes on downstream dependencies in your BI tool

```bash
# clone all of my models from specified state to my target schema(s)
dbt clone --state path/to/artifacts

# clone one_specific_model of my models from specified state to my target schema(s)
dbt clone --select one_specific_model --state path/to/artifacts

# clone all of my models from specified state to my target schema(s) and recreate all pre-exisiting relations in the current target
dbt clone --state path/to/artifacts --full-refresh

# clone all of my models from specified state to my target schema(s), running up to 50 clone statements in parallel
dbt clone --state path/to/artifacts --threads 50
```

### When to use `dbt clone` instead of [deferral](/reference/node-selection/defer)?

Unlike deferral, `dbt clone` requires some compute and creation of additional objects in your data warehouse. In many cases, deferral is a cheaper and simpler alternative to `dbt clone`. However, `dbt clone` covers additional use cases where deferral may not be possible.

For example, by creating actual data warehouse objects, `dbt clone` allows you to test out your code changes on downstream dependencies _outside of dbt_ (such as a BI tool).

As another example, you could `clone` your modified incremental models as the first step of your dbt Cloud CI job to prevent costly `full-refresh` builds for warehouses that support zero-copy cloning.
Comment on lines +37 to +39
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

2 changes: 2 additions & 0 deletions website/docs/reference/dbt-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Select the tabs that are relevant to the your development workflow. For example,
Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`.

- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
- [clone](/reference/commands/clone): clone selected nodes from specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [deps](/reference/commands/deps): downloads dependencies for a project
- [docs](/reference/commands/cmd-docs) : generates documentation for a project
Expand All @@ -40,6 +41,7 @@ Use the following dbt commands in the [CLI](/docs/core/about-the-cli) and use th

- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
- [clean](/reference/commands/clean): deletes artifacts present in the dbt project
- [clone](/reference/commands/clone): clone selected models from specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [debug](/reference/commands/debug): debugs dbt connections and projects
- [deps](/reference/commands/deps): downloads dependencies for a project
Expand Down
3 changes: 3 additions & 0 deletions website/docs/reference/node-selection/defer.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ Deferral is a powerful, complex feature that enables compelling workflows. As th
Defer is a powerful feature that makes it possible to run a subset of models or tests in a [sandbox environment](/docs/environments-in-dbt) without having to first build their upstream parents. This can save time and computational resources when you want to test a small number of models in a large project.

Defer requires that a manifest from a previous dbt invocation be passed to the `--state` flag or env var. Together with the `state:` selection method, these features enable "Slim CI". Read more about [state](/reference/node-selection/syntax#about-node-selection).

An alternative command that accomplishes similar functionality for different use cases is `dbt clone` - see the docs for [clone](/reference/commands/clone#when-to-use-dbt-clone-instead-of-deferral) for more information.

### Usage

```shell
Expand Down
3 changes: 2 additions & 1 deletion website/docs/reference/node-selection/syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,9 @@ dbt can leverage artifacts from a prior invocation as long as their file path is
- [The `state:` selector](/reference/node-selection/methods#the-state-method), whereby dbt can identify resources that are new or modified
by comparing code in the current project against the state manifest.
- [Deferring](/reference/node-selection/defer) to another environment, whereby dbt can identify upstream, unselected resources that don't exist in your current environment and instead "defer" their references to the environment provided by the state manifest.
- The [`dbt clone` command](/reference/commands/clone), whereby dbt can clone nodes based on their location in the manifest provided to the `--state` flag.

Together, these two features enable ["slim CI"](/guides/legacy/best-practices#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag.
Together, the `state:` selector and deferral enable ["slim CI"](/guides/legacy/best-practices#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

### Establishing state

Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,7 @@ const sidebarSettings = {
items: [
"reference/commands/build",
"reference/commands/clean",
"reference/commands/clone",
"reference/commands/cmd-docs",
"reference/commands/compile",
"reference/commands/debug",
Expand Down
Loading