Move remaining Operate, Test, and Deploy content #27087
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
…ther testing docs Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
dehume
reviewed
Jan 15, 2025
There was a problem hiding this comment.
Overall looks good. Found a couple of nits. And 2 larger questions:
- For some of the older guides like k8s the language is around
ops. Is there a ticket to eventually rework them to be assets? And when talking about the node abstraction in general, doesopsinclude bothopsandassets? - Noticed their is some inconsistency when referring to the OSS platform. Sometimes it is "Dagster Open Source" or"Dagster Open Source (OSS)" or just "Open Source". Didn't know what the preferred standard is.
| ) | ||
| ``` | ||
|
|
||
| It is recommenced to include definitions in a Python module named `definitions.py`. |
There was a problem hiding this comment.
@cmpadden if this is true, should Dagster U get updated and have the Definitions moved out of the __init__.py
There was a problem hiding this comment.
Yeah, that is a more recent (last year) best practice, and I think DU should be updated.
| {/* TODO turn back into <CodeReferenceLink> once that's implemented */} | ||
| :::tip | ||
|
|
||
| You can find the code for this example on [GitHub](https://github.com/dagster-io/dagster/tree/1.9.8/examples/deploy_ecs). |
There was a problem hiding this comment.
Any reason this Github link is pinned to 1.9.8?
There was a problem hiding this comment.
In old docs, this link is generated by a <CodeReferenceLink> component, but that component isn't implemented in new docs. This is basically a placeholder link until we implement that component -- here's the relevant Linear ticket: https://linear.app/dagster-labs/issue/DOC-685/implement-codereferencelink-component
There was a problem hiding this comment.
This should probably just be https://github.com/dagster-io/dagster/tree/master/examples/deploy_ecs in the meantime, though; I can change that.
|
|
||
| In this case, you'll want to ensure you provide the right connection strings for your Cloud SQL instance, and that the node or container hosting the webserver is able to connect to Cloud SQL. | ||
|
|
||
| Be sure that this file is present, and _DAGSTER_HOME_ is set, on the node where the webserver is running. |
There was a problem hiding this comment.
Suggested change
| Be sure that this file is present, and _DAGSTER_HOME_ is set, on the node where the webserver is running. | |
| Be sure that this file is present, and `_DAGSTER_HOME_` is set, on the node where the webserver is running. |
|
|
||
| Refer to the [boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task) for the full set of available arguments to `run_task`. Additionally, note that: | ||
|
|
||
| - Keys are in `camelCase` as they correspond to arguments in boto's API |
There was a problem hiding this comment.
Does camelCase need to be formatted as code?
There was a problem hiding this comment.
No, I'll remove that formatting.
|
|
||
| ## Pipes protocol | ||
|
|
||
| The Pipes protocol is used to integrate with systems that have their own execution environments. It enables running code in these external environments while allowing Dagster to maintain control and visibility. Example implementations of this approach include [AWS Lambda](https://github.com/dagster-io/dagster/tree/d4b4d5beabf6475c7279b7f02f893a506bca0bb0/python_modules/libraries/dagster-aws/dagster_aws/pipes), [Databricks](https://github.com/dagster-io/dagster/blob/d4b4d5beabf6475c7279b7f02f893a506bca0bb0/python_modules/libraries/dagster-databricks/dagster_databricks/pipes.py), and [Kubernetes](https://github.com/dagster-io/dagster/blob/d4b4d5beabf6475c7279b7f02f893a506bca0bb0/python_modules/libraries/dagster-k8s/dagster_k8s/pipes.py). |
There was a problem hiding this comment.
All the links are tied to a specific branch
Signed-off-by: nikki everett <[email protected]>
Signed-off-by: nikki everett <[email protected]>
dehume
approved these changes
Jan 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Move content into Operate, Test, and Deploy sections of new docs.
How I Tested These Changes
Local build.
Changelog