Temporarily revert dagster-dlt (#27103)

The addition of the package is fine, but it turns out some recent changes to how we use uv with our release pipeline are causing issues. So reverting until we get that sorted out.
dagster-io · Jan 14, 2025 · 5c4bf8a · 5c4bf8a · github-actions · Jan 14, 2025
1 parent cdbab74
commit 5c4bf8a
Show file tree

Hide file tree

Showing 55 changed files with 175 additions and 495 deletions.
diff --git a/docs/content/_apidocs.mdx b/docs/content/_apidocs.mdx
@@ -394,13 +394,6 @@ Dagster also provides a growing set of optional add-on libraries to integrate wi
         Datahub from within Dagster ops.
       </td>
     </tr>
-    <tr>
-      <td>
-        <a href="/_apidocs/libraries/dagster-dlt">dlt</a> (
-        <code>dagster-dlt</code>)
-      </td>
-      <td>Provides support for running dlt within Dagster</td>
-    </tr>
     <tr>
       <td>
         <a href="/_apidocs/libraries/dagster-docker">Docker</a> (

diff --git a/docs/content/_navigation.json b/docs/content/_navigation.json
@@ -1118,7 +1118,7 @@
         "children": [
           {
             "title": "dlt & Dagster",
-            "path": "/integrations/dlt"
+            "path": "/integrations/embedded-elt/dlt"
           },
           {
             "title": "Sling & Dagster",
@@ -1570,10 +1570,6 @@
             "title": "Delta Lake (dagster-deltalake)",
             "path": "/_apidocs/libraries/dagster-deltalake"
           },
-          {
-            "title": "dlt (dagster-dlt)",
-            "path": "/_apidocs/libraries/dagster-dlt"
-          },
           {
             "title": "Docker (dagster-docker)",
             "path": "/_apidocs/libraries/dagster-docker"

diff --git a/docs/content/api/modules.json.gz b/docs/content/api/modules.json.gz
diff --git a/docs/content/api/searchindex.json.gz b/docs/content/api/searchindex.json.gz
diff --git a/docs/content/api/sections.json.gz b/docs/content/api/sections.json.gz
diff --git a/docs/content/integrations.mdx b/docs/content/integrations.mdx
@@ -70,7 +70,6 @@ Explore guides for integrations with external services.
     href="/integrations/dbt-cloud"
   ></ArticleListItem>
   <ArticleListItem title="DuckDB" href="/integrations/duckdb"></ArticleListItem>
-  <ArticleListItem title="dlt" href="/integrations/dlt"></ArticleListItem>
   <ArticleListItem
     title="Embedded ELT"
     href="/integrations/embedded-elt"
@@ -139,10 +138,6 @@ Explore libraries that are maintained by the Dagster core team.
     title="dbt"
     href="/_apidocs/libraries/dagster-dbt"
   ></ArticleListItem>
-  <ArticleListItem
-    title="dlt"
-    href="/_apidocs/libraries/dagster-dlt"
-  ></ArticleListItem>
   <ArticleListItem
     title="Embedded ELT"
     href="/_apidocs/libraries/dagster-embedded-elt"

diff --git a/docs/content/integrations/embedded-elt.mdx b/docs/content/integrations/embedded-elt.mdx
@@ -30,4 +30,4 @@ Sling provides an easy-to-use YAML configuration layer for loading data from fil
 
 With the ability to leverage pre-made [verified sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) like [Hubspot](https://dlthub.com/docs/dlt-ecosystem/verified-sources/hubspot) and [Notion](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion), and [destinations](https://dlthub.com/docs/dlt-ecosystem/destinations/) like [Databricks](https://dlthub.com/docs/dlt-ecosystem/destinations/databricks) and [Snowflake](https://dlthub.com/docs/dlt-ecosystem/destinations/snowflake), integrating dlt into your Dagster project enables you to load a data in an easy and structured way.
 
-[Click here to get started](/integrations/dlt) with the dlt integration.
+[Click here to get started](/integrations/embedded-elt/dlt) with the dlt integration.
diff --git a/docs/content/integrations/dlt.mdx → ...content/integrations/embedded-elt/dlt.mdx b/docs/content/integrations/dlt.mdx → ...content/integrations/embedded-elt/dlt.mdx
@@ -65,10 +65,10 @@ To follow the steps in this guide, you'll need:
 - **[To install](/getting-started/install) the following libraries**:
 
   ```bash
-  pip install dagster dagster-dlt
+  pip install dagster dagster-embedded-elt
   ```
 
-  Installing `dagster-dlt` will also install the `dlt` package.
+  Installing `dagster-embedded-elt` will also install the `dlt` package.
 
 ---
 
@@ -165,10 +165,10 @@ Ensure that these variables are defined in your environment, either in your `.en
 
 ## Step 4: Define a DagsterDltResource
 
-Next, we'll define a <PyObject module="dagster_dlt" object="DagsterDltResource" />, which provides a wrapper of a dlt pipeline runner. Use the following to define the resource, which can be shared across all dlt pipelines:
+Next, we'll define a <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltResource" />, which provides a wrapper of a dlt pipeline runner. Use the following to define the resource, which can be shared across all dlt pipelines:
 
 ```python
-from dagster_dlt import DagsterDltResource
+from dagster_embedded_elt.dlt import DagsterDltResource
 
 dlt_resource = DagsterDltResource()
 ```
@@ -179,9 +179,9 @@ We'll add the resource to our <PyObject module="dagster" object="Definitions" />
 
 ## Step 5: Create a dlt_assets definition for GitHub
 
-The <PyObject object="dlt_assets" module="dagster_dlt" decorator /> decorator takes a `dlt_source` and `dlt_pipeline` parameter. In this example, we used the `github_reactions` source and created a `dlt_pipeline` to ingest data from Github to Snowflake.
+The <PyObject object="dlt_assets" module="dagster_embedded_elt.dlt" decorator /> decorator takes a `dlt_source` and `dlt_pipeline` parameter. In this example, we used the `github_reactions` source and created a `dlt_pipeline` to ingest data from Github to Snowflake.
 
-In the same file containing your Dagster assets, you can create an instance of your <PyObject object="dlt_assets" module="dagster_dlt" decorator /> by doing something like the following:
+In the same file containing your Dagster assets, you can create an instance of your <PyObject object="dlt_assets" module="dagster_embedded_elt.dlt" decorator /> by doing something like the following:
 
 <Note>
   If you are using the{" "}
@@ -195,7 +195,7 @@ In the same file containing your Dagster assets, you can create an instance of y
 
 ```python
 from dagster import AssetExecutionContext, Definitions
-from dagster_dlt import DagsterDltResource, dlt_assets
+from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
 from dlt import pipeline
 from dlt_sources.github import github_reactions
 
@@ -242,15 +242,19 @@ And that's it! You should now have two assets that load data to corresponding Sn
 
 ### Overriding the translator to customize dlt assets
 
-The <PyObject module="dagster_dlt" object="DagsterDltTranslator" /> object can be used to customize how dlt properties map to Dagster concepts.
+The <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" /> object can be used to customize how dlt properties map to Dagster concepts.
 
-For example, to change how the name of the asset is derived, you can override the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_asset_key" /> method, or if you would like to change the key of the upstream source asset, you can override the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
+For example, to change how the name of the asset is derived, you can override the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_asset_key" /> method, or if you would like to change the key of the upstream source asset, you can override the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
 
-```python file=/integrations/dlt/dlt_dagster_translator.py
+```python file=/integrations/embedded_elt/dlt_dagster_translator.py
 from collections.abc import Iterable
 
 import dlt
-from dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assets
+from dagster_embedded_elt.dlt import (
+    DagsterDltResource,
+    DagsterDltTranslator,
+    dlt_assets,
+)
 from dlt.extract.resource import DltResource
 
 from dagster import AssetExecutionContext, AssetKey
@@ -294,13 +298,13 @@ In this example, we customized the translator to change how the dlt assets' name
 
 A common question is how to define metadata on the external assets upstream of the dlt assets.
 
-This can be accomplished by defining a <PyObject object="AssetSpec" /> with a key that matches the one defined in the <PyObject module="dagster_dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
+This can be accomplished by defining a <PyObject object="AssetSpec" /> with a key that matches the one defined in the <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" method="get_deps_assets_keys" /> method.
 
 For example, let's say we have defined a set of dlt assets named `thinkific_assets`, we can iterate over those assets and derive a <PyObject object="AssetSpec" /> with attributes like `group_name`.
 
-```python file=/integrations/dlt/dlt_source_assets.py
+```python file=/integrations/embedded_elt/dlt_source_assets.py
 import dlt
-from dagster_dlt import DagsterDltResource, dlt_assets
+from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
 
 from dagster import AssetExecutionContext, AssetSpec
 
@@ -336,11 +340,11 @@ While still an experimental feature, it is possible to use partitions within you
 
 That said, here is an example of using static named partitions from a dlt source.
 
-```python file=/integrations/dlt/dlt_partitions.py
+```python file=/integrations/embedded_elt/dlt_partitions.py
 from typing import Optional
 
 import dlt
-from dagster_dlt import DagsterDltResource, dlt_assets
+from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
 
 from dagster import AssetExecutionContext, StaticPartitionsDefinition
 
@@ -381,11 +385,11 @@ Want to see real-world examples of dlt in production? Check out how we use it in
 
 ## APIs in this guide
 
-| Name                                                            | Description                                                    |
-| --------------------------------------------------------------- | -------------------------------------------------------------- |
-| <PyObject module="dagster_dlt" object="dlt_assets" decorator /> | The core dlt asset factory for building ingestion jobs         |
-| <PyObject module="dagster_.dlt" object="DagsterDltResource" />  | The dlt resource for running ingestions.                       |
-| <PyObject module="dagster_dlt" object="DagsterDltTranslator" /> | A translator for specifying how to map between dlt and Dagster |
+| Name                                                                         | Description                                                    |
+| ---------------------------------------------------------------------------- | -------------------------------------------------------------- |
+| <PyObject module="dagster_embedded_elt.dlt" object="dlt_assets" decorator /> | The core dlt asset factory for building ingestion jobs         |
+| <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltResource" />   | The dlt resource for running ingestions.                       |
+| <PyObject module="dagster_embedded_elt.dlt" object="DagsterDltTranslator" /> | A translator for specifying how to map between dlt and Dagster |
 
 ---
 

diff --git a/docs/content/integrations/embedded-elt/sling.mdx b/docs/content/integrations/embedded-elt/sling.mdx
@@ -308,6 +308,6 @@ def my_assets(context, sling: SlingResource):
   ></ArticleListItem>
   <ArticleListItem
     title="dlt & Dagster"
-    href="/integrations/dlt"
+    href="/integrations/embedded-elt/dlt"
   ></ArticleListItem>
 </ArticleList>
diff --git a/docs/docs-beta/docs/integrations/libraries/dlt.md b/docs/docs-beta/docs/integrations/libraries/dlt.md
@@ -6,8 +6,8 @@ title: Dagster & dlt
 sidebar_label: dlt
 excerpt: Easily ingest and replicate data between systems with dlt through Dagster.
 date: 2024-08-30
-apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-dlt
-docslink: https://docs.dagster.io/integrations/dlt
+apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-embedded-elt
+docslink: https://docs.dagster.io/integrations/embedded-elt/dlt
 partnerlink: https://www.getdbt.com/
 logo: /integrations/dlthub.jpeg
 categories:
@@ -24,7 +24,7 @@ This integration allows you to use [dlt](https://dlthub.com/) to easily ingest a
 ### Installation
 
 ```bash
-pip install dagster-dlt
+pip install dagster-embedded-elt
 ```
 
 ### Example

diff --git a/docs/next/public/objects.inv b/docs/next/public/objects.inv
diff --git a/docs/next/util/redirectUrls.json b/docs/next/util/redirectUrls.json
@@ -758,10 +758,5 @@
     "source": "/integrations/databricks",
     "destination": "/concepts/dagster-pipes/databricks",
     "statusCode": 302
-  },
-  {
-    "source": "/integrations/embedded-elt/dlt",
-    "destination": "/integrations/dlt",
-    "statusCode": 302
   }
 ]
diff --git a/docs/sphinx/index.rst b/docs/sphinx/index.rst
@@ -43,7 +43,6 @@
    sections/api/apidocs/libraries/dagster-deltalake
    sections/api/apidocs/libraries/dagster-deltalake-pandas
    sections/api/apidocs/libraries/dagster-deltalake-polars
-   sections/api/apidocs/libraries/dagster-dlt
    sections/api/apidocs/libraries/dagster-duckdb
    sections/api/apidocs/libraries/dagster-duckdb-pandas
    sections/api/apidocs/libraries/dagster-duckdb-pyspark

diff --git a/docs/sphinx/sections/api/apidocs/libraries/dagster-dlt.rst b/docs/sphinx/sections/api/apidocs/libraries/dagster-dlt.rst
diff --git a/docs/sphinx/sections/api/apidocs/libraries/dagster-embedded-elt.rst b/docs/sphinx/sections/api/apidocs/libraries/dagster-embedded-elt.rst
@@ -2,10 +2,12 @@ Embedded ELT (dagster-embedded-elt)
 -----------------------------------
 
 This package provides a framework for building ELT pipelines with Dagster through helpful pre-built
-assets and resources. This package currently includes the following integration:
+assets and resources. This package currently includes the following integrations:
 
 * `Sling <https://slingdata.io>`_, which provides a simple way to sync data between databases and file systems
 
+* `dlt <https://dlthub.com>`_, or data load tool, which provides a way to load data from systems and APIs
+
 For more information on getting started, see the `Embedded ELT <https://docs.dagster.io/integrations/embedded-elt>`_ documentation.
 
 ----
@@ -32,3 +34,26 @@ Resources (Sling)
     :members: replicate
 
 .. autoclass:: SlingConnectionResource
+
+*******************************
+dlt (dagster-embedded-elt.dlt)
+*******************************
+
+Refer to the `dlt guide <https://docs.dagster.io/integrations/embedded-elt/dlt>`_ to get started.
+
+.. currentmodule:: dagster_embedded_elt.dlt
+
+Assets (dlt)
+=================
+
+.. autodecorator:: dlt_assets
+
+.. autofunction:: build_dlt_asset_specs
+
+.. autoclass:: DagsterDltTranslator
+
+Resources (dlt)
+=================
+
+.. autoclass:: DagsterDltResource
+    :members: run
diff --git a/docs/tox.ini b/docs/tox.ini
@@ -38,9 +38,7 @@ deps =
   -e ../python_modules/libraries/dagster-ssh
   -e ../python_modules/libraries/dagster-duckdb
   -e ../python_modules/libraries/dagster-dbt
-  -e ../python_modules/libraries/dagster-dlt
   -e ../python_modules/libraries/dagster-wandb
-  -e ../python_modules/libraries/dagster-dlt
   -e ../python_modules/libraries/dagster-embedded-elt
   -e ../python_modules/libraries/dagster-deltalake
   -e ../python_modules/libraries/dagster-deltalake-pandas
@@ -81,9 +79,7 @@ deps =
   -e ../python_modules/libraries/dagster-ssh
   -e ../python_modules/libraries/dagster-duckdb
   -e ../python_modules/libraries/dagster-dbt
-  -e ../python_modules/libraries/dagster-dlt
   -e ../python_modules/libraries/dagster-wandb
-  -e ../python_modules/libraries/dagster-dlt
   -e ../python_modules/libraries/dagster-embedded-elt
   -e ../python_modules/libraries/dagster-deltalake
   -e ../python_modules/libraries/dagster-deltalake-pandas

diff --git a/examples/docs_beta_snippets/docs_beta_snippets/integrations/dlt.py b/examples/docs_beta_snippets/docs_beta_snippets/integrations/dlt.py
@@ -1,5 +1,5 @@
 import dlt
-from dagster_dlt import DagsterDltResource, dlt_assets
+from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
 from dlt_sources.github import github_reactions
 
 import dagster as dg

diff --git a/examples/docs_beta_snippets/tox.ini b/examples/docs_beta_snippets/tox.ini
@@ -22,7 +22,6 @@ deps =
   -e ../../python_modules/dagster-graphql
   -e ../../python_modules/dagster-webserver
   -e ../../python_modules/libraries/dagster-dbt
-  -e ../../python_modules/libraries/dagster-dlt
   -e ../../python_modules/libraries/dagster-snowflake
   -e ../../python_modules/libraries/dagster-duckdb
   -e ../../python_modules/libraries/dagster-duckdb-pandas

diff --git a/...ntegrations/dlt/dlt_dagster_translator.py → ...ns/embedded_elt/dlt_dagster_translator.py b/...ntegrations/dlt/dlt_dagster_translator.py → ...ns/embedded_elt/dlt_dagster_translator.py
@@ -1,7 +1,11 @@
 from collections.abc import Iterable
 
 import dlt
-from dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assets
+from dagster_embedded_elt.dlt import (
+    DagsterDltResource,
+    DagsterDltTranslator,
+    dlt_assets,
+)
 from dlt.extract.resource import DltResource
 
 from dagster import AssetExecutionContext, AssetKey

diff --git a/...ippets/integrations/dlt/dlt_partitions.py → ...tegrations/embedded_elt/dlt_partitions.py b/...ippets/integrations/dlt/dlt_partitions.py → ...tegrations/embedded_elt/dlt_partitions.py
@@ -1,7 +1,7 @@
 from typing import Optional
 
 import dlt
-from dagster_dlt import DagsterDltResource, dlt_assets
+from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
 
 from dagster import AssetExecutionContext, StaticPartitionsDefinition
 

diff --git a/...ets/integrations/dlt/dlt_source_assets.py → ...rations/embedded_elt/dlt_source_assets.py b/...ets/integrations/dlt/dlt_source_assets.py → ...rations/embedded_elt/dlt_source_assets.py
@@ -1,5 +1,5 @@
 import dlt
-from dagster_dlt import DagsterDltResource, dlt_assets
+from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
 
 from dagster import AssetExecutionContext, AssetSpec
 

diff --git a/examples/docs_snippets/setup.py b/examples/docs_snippets/setup.py
@@ -27,7 +27,6 @@
         "dagster-deltalake",
         "dagster-deltalake-pandas",
         "dagster-deltalake-polars",
-        "dagster-dlt",
         "dagster-duckdb",
         "dagster-duckdb-pandas",
         "dagster-embedded-elt",

diff --git a/examples/docs_snippets/tox.ini b/examples/docs_snippets/tox.ini
@@ -27,7 +27,6 @@ deps =
   -e ../../python_modules/libraries/dagster-deltalake
   -e ../../python_modules/libraries/dagster-deltalake-pandas
   -e ../../python_modules/libraries/dagster-deltalake-polars
-  -e ../../python_modules/libraries/dagster-dlt
   -e ../../python_modules/libraries/dagster-duckdb
   -e ../../python_modules/libraries/dagster-duckdb-pandas
   -e ../../python_modules/libraries/dagster-embedded-elt
Original file line number	Diff line number	Diff line change
Expand Up		@@ -30,4 +30,4 @@ Sling provides an easy-to-use YAML configuration layer for loading data from fil

		With the ability to leverage pre-made [verified sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) like [Hubspot](https://dlthub.com/docs/dlt-ecosystem/verified-sources/hubspot) and [Notion](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion), and [destinations](https://dlthub.com/docs/dlt-ecosystem/destinations/) like [Databricks](https://dlthub.com/docs/dlt-ecosystem/destinations/databricks) and [Snowflake](https://dlthub.com/docs/dlt-ecosystem/destinations/snowflake), integrating dlt into your Dagster project enables you to load a data in an easy and structured way.

		[Click here to get started](/integrations/dlt) with the dlt integration.
		[Click here to get started](/integrations/embedded-elt/dlt) with the dlt integration.