Skip to content

Commit df30e85

Browse files
[docs] Move dlt docs out of embedded-elt docs (#27071)
## Summary & Motivation As title.
1 parent 1aead34 commit df30e85

File tree

22 files changed

+81
-65
lines changed

22 files changed

+81
-65
lines changed

docs/content/_apidocs.mdx

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,6 +394,13 @@ Dagster also provides a growing set of optional add-on libraries to integrate wi
394394
Datahub from within Dagster ops.
395395
</td>
396396
</tr>
397+
<tr>
398+
<td>
399+
<a href="/_apidocs/libraries/dagster-dlt">dlt</a> (
400+
<code>dagster-dlt</code>)
401+
</td>
402+
<td>Provides support for running dlt within Dagster</td>
403+
</tr>
397404
<tr>
398405
<td>
399406
<a href="/_apidocs/libraries/dagster-docker">Docker</a> (

docs/content/_navigation.json

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1118,7 +1118,7 @@
11181118
"children": [
11191119
{
11201120
"title": "dlt & Dagster",
1121-
"path": "/integrations/embedded-elt/dlt"
1121+
"path": "/integrations/dlt"
11221122
},
11231123
{
11241124
"title": "Sling & Dagster",
@@ -1570,6 +1570,10 @@
15701570
"title": "Delta Lake (dagster-deltalake)",
15711571
"path": "/_apidocs/libraries/dagster-deltalake"
15721572
},
1573+
{
1574+
"title": "dlt (dagster-dlt)",
1575+
"path": "/_apidocs/libraries/dagster-dlt"
1576+
},
15731577
{
15741578
"title": "Docker (dagster-docker)",
15751579
"path": "/_apidocs/libraries/dagster-docker"

docs/content/api/modules.json.gz

9.83 KB
Binary file not shown.

docs/content/api/searchindex.json.gz

261 Bytes
Binary file not shown.

docs/content/api/sections.json.gz

2.26 KB
Binary file not shown.

docs/content/integrations.mdx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ Explore guides for integrations with external services.
7070
href="/integrations/dbt-cloud"
7171
></ArticleListItem>
7272
<ArticleListItem title="DuckDB" href="/integrations/duckdb"></ArticleListItem>
73+
<ArticleListItem title="dlt" href="/integrations/dlt"></ArticleListItem>
7374
<ArticleListItem
7475
title="Embedded ELT"
7576
href="/integrations/embedded-elt"
@@ -138,6 +139,10 @@ Explore libraries that are maintained by the Dagster core team.
138139
title="dbt"
139140
href="/_apidocs/libraries/dagster-dbt"
140141
></ArticleListItem>
142+
<ArticleListItem
143+
title="dlt"
144+
href="/_apidocs/libraries/dagster-dlt"
145+
></ArticleListItem>
141146
<ArticleListItem
142147
title="Embedded ELT"
143148
href="/_apidocs/libraries/dagster-embedded-elt"

docs/content/integrations/embedded-elt/dlt.mdx renamed to docs/content/integrations/dlt.mdx

Lines changed: 21 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,10 @@ To follow the steps in this guide, you'll need:
6565
- **[To install](/getting-started/install) the following libraries**:
6666

6767
```bash
68-
pip install dagster dagster-embedded-elt
68+
pip install dagster dagster-dlt
6969
```
7070

71-
Installing `dagster-embedded-elt` will also install the `dlt` package.
71+
Installing `dagster-dlt` will also install the `dlt` package.
7272

7373
---
7474

@@ -165,10 +165,10 @@ Ensure that these variables are defined in your environment, either in your `.en
165165
166166
## Step 4: Define a DagsterDltResource
167167
168-
Next, we'll define a <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltResource" />, which provides a wrapper of a dlt pipeline runner. Use the following to define the resource, which can be shared across all dlt pipelines:
168+
Next, we'll define a <PyObject module="dagster_dlt" object="DagsterDltResource" />, which provides a wrapper of a dlt pipeline runner. Use the following to define the resource, which can be shared across all dlt pipelines:
169169
170170
```python
171-
from dagster_embedded_elt.dlt import DagsterDltResource
171+
from dagster_dlt import DagsterDltResource
172172
173173
dlt_resource = DagsterDltResource()
174174
```
@@ -179,9 +179,9 @@ We'll add the resource to our <PyObject module="dagster" object="Definitions" />
179179
180180
## Step 5: Create a dlt_assets definition for GitHub
181181
182-
The <PyObject object="dlt_assets" module="dagster_embedded_elt.dlt" decorator /> decorator takes a `dlt_source` and `dlt_pipeline` parameter. In this example, we used the `github_reactions` source and created a `dlt_pipeline` to ingest data from Github to Snowflake.
182+
The <PyObject object="dlt_assets" module="dagster_dlt" decorator /> decorator takes a `dlt_source` and `dlt_pipeline` parameter. In this example, we used the `github_reactions` source and created a `dlt_pipeline` to ingest data from Github to Snowflake.
183183
184-
In the same file containing your Dagster assets, you can create an instance of your <PyObject object="dlt_assets" module="dagster_embedded_elt.dlt" decorator /> by doing something like the following:
184+
In the same file containing your Dagster assets, you can create an instance of your <PyObject object="dlt_assets" module="dagster_dlt" decorator /> by doing something like the following:
185185
186186
<Note>
187187
If you are using the{" "}
@@ -195,7 +195,7 @@ In the same file containing your Dagster assets, you can create an instance of y
195195
196196
```python
197197
from dagster import AssetExecutionContext, Definitions
198-
from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
198+
from dagster_dlt import DagsterDltResource, dlt_assets
199199
from dlt import pipeline
200200
from dlt_sources.github import github_reactions
201201
@@ -242,19 +242,15 @@ And that's it! You should now have two assets that load data to corresponding Sn
242242
243243
### Overriding the translator to customize dlt assets
244244
245-
The <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" /> object can be used to customize how dlt properties map to Dagster concepts.
245+
The <PyObject module="dagster_dlt" object="DagsterDltTranslator" /> object can be used to customize how dlt properties map to Dagster concepts.
246246
247-
For example, to change how the name of the asset is derived, you can override the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_asset_key" /> method, or if you would like to change the key of the upstream source asset, you can override the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
247+
For example, to change how the name of the asset is derived, you can override the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_asset_key" /> method, or if you would like to change the key of the upstream source asset, you can override the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
248248
249-
```python file=/integrations/embedded_elt/dlt_dagster_translator.py
249+
```python file=/integrations/dlt/dlt_dagster_translator.py
250250
from collections.abc import Iterable
251251
252252
import dlt
253-
from dagster_embedded_elt.dlt import (
254-
DagsterDltResource,
255-
DagsterDltTranslator,
256-
dlt_assets,
257-
)
253+
from dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assets
258254
from dlt.extract.resource import DltResource
259255
260256
from dagster import AssetExecutionContext, AssetKey
@@ -298,13 +294,13 @@ In this example, we customized the translator to change how the dlt assets' name
298294
299295
A common question is how to define metadata on the external assets upstream of the dlt assets.
300296
301-
This can be accomplished by defining a <PyObject object="AssetSpec" /> with a key that matches the one defined in the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
297+
This can be accomplished by defining a <PyObject object="AssetSpec" /> with a key that matches the one defined in the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
302298
303299
For example, let's say we have defined a set of dlt assets named `thinkific_assets`, we can iterate over those assets and derive a <PyObject object="AssetSpec" /> with attributes like `group_name`.
304300
305-
```python file=/integrations/embedded_elt/dlt_source_assets.py
301+
```python file=/integrations/dlt/dlt_source_assets.py
306302
import dlt
307-
from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
303+
from dagster_dlt import DagsterDltResource, dlt_assets
308304
309305
from dagster import AssetExecutionContext, AssetSpec
310306
@@ -340,11 +336,11 @@ While still an experimental feature, it is possible to use partitions within you
340336
341337
That said, here is an example of using static named partitions from a dlt source.
342338
343-
```python file=/integrations/embedded_elt/dlt_partitions.py
339+
```python file=/integrations/dlt/dlt_partitions.py
344340
from typing import Optional
345341
346342
import dlt
347-
from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
343+
from dagster_dlt import DagsterDltResource, dlt_assets
348344
349345
from dagster import AssetExecutionContext, StaticPartitionsDefinition
350346
@@ -385,11 +381,11 @@ Want to see real-world examples of dlt in production? Check out how we use it in
385381
386382
## APIs in this guide
387383
388-
| Name | Description |
389-
| ---------------------------------------------------------------------------- | -------------------------------------------------------------- |
390-
| <PyObject module="dagster_embedded_elt.dlt" object="dlt_assets" decorator /> | The core dlt asset factory for building ingestion jobs |
391-
| <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltResource" /> | The dlt resource for running ingestions. |
392-
| <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" /> | A translator for specifying how to map between dlt and Dagster |
384+
| Name | Description |
385+
| --------------------------------------------------------------- | -------------------------------------------------------------- |
386+
| <PyObject module="dagster_dlt" object="dlt_assets" decorator /> | The core dlt asset factory for building ingestion jobs |
387+
| <PyObject module="dagster_.dlt" object="DagsterDltResource" /> | The dlt resource for running ingestions. |
388+
| <PyObject module="dagster_dlt" object="DagsterDltTranslator" /> | A translator for specifying how to map between dlt and Dagster |
393389
394390
---
395391

docs/content/integrations/embedded-elt.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,4 @@ Sling provides an easy-to-use YAML configuration layer for loading data from fil
3030

3131
With the ability to leverage pre-made [verified sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) like [Hubspot](https://dlthub.com/docs/dlt-ecosystem/verified-sources/hubspot) and [Notion](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion), and [destinations](https://dlthub.com/docs/dlt-ecosystem/destinations/) like [Databricks](https://dlthub.com/docs/dlt-ecosystem/destinations/databricks) and [Snowflake](https://dlthub.com/docs/dlt-ecosystem/destinations/snowflake), integrating dlt into your Dagster project enables you to load a data in an easy and structured way.
3232

33-
[Click here to get started](/integrations/embedded-elt/dlt) with the dlt integration.
33+
[Click here to get started](/integrations/dlt) with the dlt integration.

docs/content/integrations/embedded-elt/sling.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,6 @@ def my_assets(context, sling: SlingResource):
308308
></ArticleListItem>
309309
<ArticleListItem
310310
title="dlt & Dagster"
311-
href="/integrations/embedded-elt/dlt"
311+
href="/integrations/dlt"
312312
></ArticleListItem>
313313
</ArticleList>

docs/docs-beta/docs/integrations/libraries/dlt.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ title: Dagster & dlt
66
sidebar_label: dlt
77
excerpt: Easily ingest and replicate data between systems with dlt through Dagster.
88
date: 2024-08-30
9-
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-embedded-elt
10-
docslink: https://docs.dagster.io/integrations/embedded-elt/dlt
9+
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-dlt
10+
docslink: https://docs.dagster.io/integrations/dlt
1111
partnerlink: https://www.getdbt.com/
1212
logo: /integrations/dlthub.jpeg
1313
categories:
@@ -24,7 +24,7 @@ This integration allows you to use [dlt](https://dlthub.com/) to easily ingest a
2424
### Installation
2525

2626
```bash
27-
pip install dagster-embedded-elt
27+
pip install dagster-dlt
2828
```
2929

3030
### Example

0 commit comments

Comments
 (0)