You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/integrations/dlt.mdx
+21-25Lines changed: 21 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -65,10 +65,10 @@ To follow the steps in this guide, you'll need:
65
65
-**[To install](/getting-started/install) the following libraries**:
66
66
67
67
```bash
68
-
pip install dagster dagster-embedded-elt
68
+
pip install dagster dagster-dlt
69
69
```
70
70
71
-
Installing `dagster-embedded-elt` will also install the `dlt` package.
71
+
Installing `dagster-dlt` will also install the `dlt` package.
72
72
73
73
---
74
74
@@ -165,10 +165,10 @@ Ensure that these variables are defined in your environment, either in your `.en
165
165
166
166
## Step 4: Define a DagsterDltResource
167
167
168
-
Next, we'll define a <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltResource" />, which provides a wrapper of a dlt pipeline runner. Use the following to define the resource, which can be shared across all dlt pipelines:
168
+
Next, we'll define a <PyObject module="dagster_dlt" object="DagsterDltResource" />, which provides a wrapper of a dlt pipeline runner. Use the following to define the resource, which can be shared across all dlt pipelines:
169
169
170
170
```python
171
-
from dagster_embedded_elt.dlt import DagsterDltResource
171
+
from dagster_dlt import DagsterDltResource
172
172
173
173
dlt_resource = DagsterDltResource()
174
174
```
@@ -179,9 +179,9 @@ We'll add the resource to our <PyObject module="dagster" object="Definitions" />
179
179
180
180
## Step 5: Create a dlt_assets definition for GitHub
181
181
182
-
The <PyObject object="dlt_assets" module="dagster_embedded_elt.dlt" decorator /> decorator takes a `dlt_source` and `dlt_pipeline` parameter. In this example, we used the `github_reactions` source and created a `dlt_pipeline` to ingest data from Github to Snowflake.
182
+
The <PyObject object="dlt_assets" module="dagster_dlt" decorator /> decorator takes a `dlt_source` and `dlt_pipeline` parameter. In this example, we used the `github_reactions` source and created a `dlt_pipeline` to ingest data from Github to Snowflake.
183
183
184
-
In the same file containing your Dagster assets, you can create an instance of your <PyObject object="dlt_assets" module="dagster_embedded_elt.dlt" decorator /> by doing something like the following:
184
+
In the same file containing your Dagster assets, you can create an instance of your <PyObject object="dlt_assets" module="dagster_dlt" decorator /> by doing something like the following:
185
185
186
186
<Note>
187
187
If you are using the{" "}
@@ -195,7 +195,7 @@ In the same file containing your Dagster assets, you can create an instance of y
195
195
196
196
```python
197
197
from dagster import AssetExecutionContext, Definitions
198
-
from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
198
+
from dagster_dlt import DagsterDltResource, dlt_assets
199
199
from dlt import pipeline
200
200
from dlt_sources.github import github_reactions
201
201
@@ -242,19 +242,15 @@ And that's it! You should now have two assets that load data to corresponding Sn
242
242
243
243
### Overriding the translator to customize dlt assets
244
244
245
-
The <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" /> object can be used to customize how dlt properties map to Dagster concepts.
245
+
The <PyObject module="dagster_dlt" object="DagsterDltTranslator" /> object can be used to customize how dlt properties map to Dagster concepts.
246
246
247
-
For example, to change how the name of the asset is derived, you can override the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_asset_key" /> method, or if you would like to change the key of the upstream source asset, you can override the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
247
+
For example, to change how the name of the asset is derived, you can override the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_asset_key" /> method, or if you would like to change the key of the upstream source asset, you can override the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
from dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assets
258
254
from dlt.extract.resource import DltResource
259
255
260
256
from dagster import AssetExecutionContext, AssetKey
@@ -298,13 +294,13 @@ In this example, we customized the translator to change how the dlt assets' name
298
294
299
295
A common question is how to define metadata on the external assets upstream of the dlt assets.
300
296
301
-
This can be accomplished by defining a <PyObject object="AssetSpec" /> with a key that matches the one defined in the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
297
+
This can be accomplished by defining a <PyObject object="AssetSpec" /> with a key that matches the one defined in the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
302
298
303
299
For example, let's say we have defined a set of dlt assets named `thinkific_assets`, we can iterate over those assets and derive a <PyObject object="AssetSpec" /> with attributes like `group_name`.
Copy file name to clipboardExpand all lines: docs/content/integrations/embedded-elt.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,4 +30,4 @@ Sling provides an easy-to-use YAML configuration layer for loading data from fil
30
30
31
31
With the ability to leverage pre-made [verified sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) like [Hubspot](https://dlthub.com/docs/dlt-ecosystem/verified-sources/hubspot) and [Notion](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion), and [destinations](https://dlthub.com/docs/dlt-ecosystem/destinations/) like [Databricks](https://dlthub.com/docs/dlt-ecosystem/destinations/databricks) and [Snowflake](https://dlthub.com/docs/dlt-ecosystem/destinations/snowflake), integrating dlt into your Dagster project enables you to load a data in an easy and structured way.
32
32
33
-
[Click here to get started](/integrations/embedded-elt/dlt) with the dlt integration.
33
+
[Click here to get started](/integrations/dlt) with the dlt integration.
0 commit comments