Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Delete embedded-elt references in docs #27074

Merged
merged 3 commits into from
Jan 15, 2025

Conversation

maximearmstrong
Copy link
Contributor

@maximearmstrong maximearmstrong commented Jan 13, 2025

Summary & Motivation

We keep the top page at /integrations/embedded-elt but references across docs and examples are deleted in favor of references to the dlt and Sling pages.

Copy link

github-actions bot commented Jan 13, 2025

@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 080add6 to 2ea45f0 Compare January 13, 2025 21:05
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 2d55b00 to 40d0d84 Compare January 13, 2025 21:05
@maximearmstrong maximearmstrong marked this pull request as ready for review January 13, 2025 21:11
@maximearmstrong maximearmstrong self-assigned this Jan 13, 2025
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 2ea45f0 to f837991 Compare January 13, 2025 22:11
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 40d0d84 to 4a98ea9 Compare January 13, 2025 22:11
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from f837991 to b08529e Compare January 13, 2025 22:18
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 4a98ea9 to 5c0bfd8 Compare January 13, 2025 22:19
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from b08529e to 936ade3 Compare January 14, 2025 14:27
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 5c0bfd8 to dcecfc7 Compare January 14, 2025 14:28
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 936ade3 to 7773bc0 Compare January 14, 2025 14:42
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from dcecfc7 to 8abfbce Compare January 14, 2025 14:43
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 7773bc0 to 5293d9c Compare January 14, 2025 14:46
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 8abfbce to 46383a5 Compare January 14, 2025 14:46
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 5293d9c to 7cc0074 Compare January 14, 2025 14:55
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 46383a5 to 924837d Compare January 14, 2025 14:55
Copy link
Contributor

@cmpadden cmpadden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximearmstrong is there a top level page that we can redirect for external references, like these coming from the website?

master ~/src/dagster-website
$ grep embedded-elt
pages/lp/dagster-plus-trial.tsx
195:          button_link="/blog/dagster-embedded-elt"

pages/events/embedded-elt-dec-2023.tsx
18:        url: `https://dagster.io/images/events/embedded-elt-og-image-min.jpg?v=7`,
33:        content="https://dagster.io/images/events/embedded-elt-og-image-min.jpg?v=7"
48:                  'linear-gradient(rgba(000, 000, 000, 0.1), rgba(000, 000, 000, 0.1)), url("/images/events/embedded-elt-event-page-header-min.jpg")',
107:                  <a href="/blog/dagster-embedded-elt" className="underline">

pages/guides/etl/etl-pipelines-5-key-components-and-5-critical-best-practices.mdx
161:With a built-in toolkit for data ingestion thanks to our [Embedded ELT](https://docs.dagster.io/integrations/embedded-elt) feature and [integrations with leading providers like Fivetran](https://docs.dagster.io/integrations/fivetran), Dagster allows you to orchestrate your ETL workloads on your platform of choice.

pages/guides/etl/etl-tools-key-features-and-10-tools-to-know-in-2025.mdx
474:It also features integrations with leading providers like [Fivetran](https://docs.dagster.io/integrations/fivetran) and [Airbyte,](https://docs.dagster.io/integrations/airbyte) and a [built-in toolkit for data ingestion using the most popular open-source data integration libraries.](https://docs.dagster.io/integrations/embedded-elt)

pages/events/dagster-launch-week-oct-2023.tsx
215:                  <img className="h-10" src="/images/icons/embedded-elt.svg" />
268:                link="/blog/dagster-embedded-elt"

pages/guides/etl.mdx
194:With a built-in toolkit for data ingestion thanks to our [Embedded ELT](https://docs.dagster.io/integrations/embedded-elt) feature and [integrations with leading providers like Fivetran](https://docs.dagster.io/integrations/fivetran), Dagster allows you to orchestrate your ETL workloads on your platform of choice.

pages/blog/elt-options-in-dagster.mdx
43:By selectively replacing your expensive data ingestion steps with Dagster embedded ELT [you can achieve both significant cost savings, and improve the performance of your data processes](https://dagster.io/blog/sling-out-your-etl-provider-with-embedded-elt).
77:Dagster's embedded ELT feature is a cost-effective and flexible solution for implementing ELT processes. It includes [components like Sling and dlt](https://dagster.io/blog/expanding-dagsters-embedded-elt-ecosystem-with-dlthub-for-data-ingestion) within the dagster-embedded-elt package, which facilitate data ingestion and transformation within Dagster pipelines. This feature provides better control over ingestion steps and integrates with Dagster's orchestration capabilities, making it a cost-effective choice.

pages/blog/see-the-forest-and-trees-dagster-plus.mdx
87:Once you’ve pinpointed the areas where an alternative to Fivetran would save you a lot of money, check out our blog on [using embedded elt](https://dagster.io/blog/sling-out-your-etl-provider-with-embedded-elt) and see how our integrations with [Sling](https://docs.dagster.io/integrations/embedded-elt/sling#sling--dagster) or [dlt](https://docs.dagster.io/integrations/embedded-elt/dlt) can help you reduce your spend.

pages/events/index.tsx
99:            link="/blog/dagster-embedded-elt"

components/TrialFeatureCard.tsx
146:                  href="/blog/dagster-embedded-elt"

pages/blog/sling-out-your-etl-provider-with-embedded-elt.mdx
9:coverImage: '/posts/sling-out-your-etl-provider-with-embedded-elt/sling-out-your-etl-provider-min.jpg'
23:The [Dagster Embedded ELT](https://docs.dagster.io/integrations/embedded-elt) library provides a lightweight toolkit for ingesting and loading data with Dagster. In its current form, Embedded ELT provides a series of assets and resources for syncing data using Sling. Sling is a powerful and lightweight ingestion engine that allows us to define how data is synchronized from source to target data system. To learn more about Embedded ELT you can check out [Pedram’s blog post](https://dagster.io/blog/dagster-embedded-elt) introducing the library.
256:      src="/posts/sling-out-your-etl-provider-with-embedded-elt/dagster-data-chart.png"
277:      src="/posts/sling-out-your-etl-provider-with-embedded-elt/dagster-assets.png"

pages/blog/expanding-dagsters-embedded-elt-ecosystem-with-dlthub-for-data-ingestion.mdx
6:coverImage: '/posts/expanding-dagster-embedded-elt-ecosystem-with-dltHub/dagster-dlt-cover-min.jpg'
13:Today, we’re excited to introduce our officially supported dlt integration. Last October we introduced the [first integration in the Embedded ELT family](https://dagster.io/blog/dagster-embedded-elt) of integrations; leveraging Sling to easily ingest and replicate data between systems.
20:      src="/posts/expanding-dagster-embedded-elt-ecosystem-with-dltHub/dagster-dlt-image-one.png"
25:    https://pypistats.org/packages/dagster-embedded-elt
78:      src="/posts/expanding-dagster-embedded-elt-ecosystem-with-dltHub/dagster-dlt-image-two.png"
85:For more advanced scenarios the dlt integration has followed the implementation architecture of the dbt and Sling integrations, in which a translator can be defined to adjust the parameters of the generated assets. For more information, reference [the API documentation](https://docs.dagster.io/_apidocs/libraries/dagster-embedded-elt#dlt-dagster-embedded-elt-dlt).
103:- [Dagster Embedded ELT Documentation](https://docs.dagster.io/integrations/embedded-elt)

pages/blog/data-platform-week-2024.mdx
71:Additionally, we are continuously investing in purpose-built open-source tooling to make it easier than ever for data teams to use the right tool for the job. With our embedded ELT package, we incorporated ingestion tools Sling and dlt so data teams can get data into their platforms cost-effectively. When we switched to using embedded ELT internally, we saved [$40K annually](https://dagster.io/blog/sling-out-your-etl-provider-with-embedded-elt) on managed solution spending.

pages/integrations/dagster-sling.mdx
8:apireflink: 'https://docs.dagster.io/_apidocs/libraries/dagster-embedded-elt'
9:docslink: 'https://docs.dagster.io/integrations/embedded-elt/sling'
27:pip install dagster-embedded-elt

pages/integrations/dagster-dlt.mdx
8:apireflink: 'https://docs.dagster.io/_apidocs/libraries/dagster-embedded-elt'
9:docslink: 'https://docs.dagster.io/integrations/embedded-elt/dlt'
27:pip install dagster-embedded-elt

pages/blog/dagster-embedded-elt.mdx
9:coverImage: '/posts/dagster-embedded-elt/dagster-embedded-elt-cover-min.jpg'
19:      src="/posts/dagster-embedded-elt/dagster-embedded-etl-ingestion-tools-min.jpeg"
41:      src="/posts/dagster-embedded-elt/embedded-elt-embeddable-libraries-min.png"
82:In that spirit, we’re introducing _dagster-embedded-elt_: a framework for building ELT pipelines with Dagster through helpful pre-built assets and resources.
89:  src="/posts/dagster-embedded-elt/embedded-elt-sling-connectors-min.png"
104:To use our `dagster-embedded-elt` to sync data between systems, it's as simple creating a replication configuration, defining your sources and targets, and calling the replicate method.
178:  Learn more about Embedded ELT in <a href="https://docs.dagster.io/integrations/embedded-elt">the Dagster Docs</a>.
182:* [Sling Out Your ETL Provider with Embedded ELT](sling-out-your-etl-provider-with-embedded-elt)
183:* [Expanding the Dagster Embedded ELT Ecosystem with dltHub for Data Ingestion](expanding-dagsters-embedded-elt-ecosystem-with-dlthub-for-data-ingestion)

@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 9917778 to bb05b7c Compare January 14, 2025 16:49
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from a2b689c to e97d5c6 Compare January 14, 2025 16:49
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from bb05b7c to ca44917 Compare January 14, 2025 17:05
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from e97d5c6 to fdfa196 Compare January 14, 2025 17:06
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from ca44917 to 8ff6725 Compare January 14, 2025 17:29
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from fdfa196 to 5cc6e0c Compare January 14, 2025 17:29
Copy link
Contributor

@cmpadden cmpadden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the mdx needs to be synced:

[2025-01-14T17:32:45Z]  python file=/integrations/sling/sling_connection_resources.py
[2025-01-14T17:32:45Z] -from dagster_sling.resources import SlingConnectionResource, SlingResource
[2025-01-14T17:32:45Z] +from dagster_sling import SlingConnectionResource, SlingResource
[2025-01-14T17:32:45Z]
[2025-01-14T17:32:45Z]  from dagster import EnvVar
[2025-01-14T17:32:45Z]
[2025-01-14T17:32:46Z]
[2025-01-14T17:32:46Z] real	0m0.934s
[2025-01-14T17:32:46Z] user	0m0.811s
[2025-01-14T17:32:46Z] sys	0m0.139s
[2025-01-14T17:32:49Z] 🚨 Error: The command exited with status 1

@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 8ff6725 to 87139b3 Compare January 14, 2025 21:28
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 5cc6e0c to 0220534 Compare January 14, 2025 21:28
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from 87139b3 to d4595cf Compare January 14, 2025 21:53
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from 0220534 to f2b0182 Compare January 14, 2025 21:53
@maximearmstrong maximearmstrong force-pushed the maxime/dagster-sling-docs branch from d4595cf to e8e90d5 Compare January 14, 2025 22:56
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from f2b0182 to d0a8534 Compare January 14, 2025 22:56
Base automatically changed from maxime/dagster-sling-docs to master January 14, 2025 23:10
@maximearmstrong maximearmstrong force-pushed the maxime/delete-embedded-elt-docs branch from d0a8534 to 607b193 Compare January 14, 2025 23:11
Copy link
Contributor

@cmpadden cmpadden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Language on /integrations/embedded-elt will need to be updated to reflect that it's two packages.

content/integrations/embedded-elt.mdx
10:The `dagster-embedded-elt` package provides a framework for building ELT pipelines with Dagster through helpful asset decorators and resources.

Copy link
Contributor

@cmpadden cmpadden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending small change to /integrations/embedded-elt, can also be done in follow-up.

@maximearmstrong maximearmstrong merged commit 1750e8f into master Jan 15, 2025
4 checks passed
@maximearmstrong maximearmstrong deleted the maxime/delete-embedded-elt-docs branch January 15, 2025 12:47
maximearmstrong added a commit that referenced this pull request Jan 15, 2025
## Summary & Motivation

Fixes this
[comment](#27074 (review))
in #27074

## How I Tested These Changes

Docs preview
@neverett neverett added docs-to-migrate Docs to migrate to new docs site and removed docs-to-migrate Docs to migrate to new docs site labels Jan 15, 2025
marijncv pushed a commit to marijncv/dagster that referenced this pull request Jan 21, 2025
## Summary & Motivation

We keep the top page at `/integrations/embedded-elt` but references
across docs and examples are deleted in favor of references to the dlt
and Sling pages.
marijncv pushed a commit to marijncv/dagster that referenced this pull request Jan 21, 2025
## Summary & Motivation

Fixes this
[comment](dagster-io#27074 (review))
in dagster-io#27074

## How I Tested These Changes

Docs preview
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants